> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources/snowflake-metadata.md).

# Snowflake

Configuration reference for the Snowflake metadata source: supported stages, account-usage permission requirements, source options, and troubleshooting.

[Snowflake](https://docs.snowflake.com/) is the most complete metadata source for Nilus. A `spec.type: metadata` pipeline introspects Snowflake and publishes source context, databases, schemas, tables, columns, profiles, lineage, usage, and classification, into the DataOS metadata catalog **without copying any table rows**.

For batch row movement out of Snowflake, see the [Snowflake](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/snowflake.md). For the field-by-field authoring contract, see [Understanding Metadata Pipeline Config](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/pipeline-config.md).

## Metadata stages

`service_type: snowflake` supports the full DAG. The required `mode` field decides how much of it runs: `shallow` runs `metadata` + `lineage`; `deep` adds `profiler`, `classification`, and `usage`. Source inventory (`metadata`) runs first; once it succeeds, the remaining stages run in parallel.

| Stage            | Runs in           | What it lands in the catalog                                                                    |
| ---------------- | ----------------- | ----------------------------------------------------------------------------------------------- |
| `metadata`       | `shallow`, `deep` | Databases, schemas, tables, views, dynamic tables, columns, and descriptions.                   |
| `lineage`        | `shallow`, `deep` | Asset and column lineage parsed from view definitions and query history.                        |
| `profiler`       | `deep`            | Per-column statistics (row counts, null counts, distinct counts, min/max, basic distributions). |
| `classification` | `deep`            | Auto-classification tags applied to columns (PII heuristics).                                   |
| `usage`          | `deep`            | Query history and popularity from the account-usage query log.                                  |

The first successful run establishes the source inventory and lineage. In `deep`, profiles, classification, and usage then deepen it. Usage typically finishes later than the other stages because Nilus reads and processes query activity over the configured lookback window. A frequent `shallow` pipeline keeps inventory and lineage fresh; pair it with a less frequent `deep` pipeline for the full profile.

## How Nilus resolves each kind of context

| What Nilus brings in            | How Nilus fetches or resolves it from Snowflake                                                                                                                  |
| ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Databases                       | Discovery queries such as `SHOW DATABASES`, plus database comments and account context, build the source hierarchy.                                              |
| Schemas                         | Schema inspection and `information_schema.schemata`; schema tags from `ACCOUNT_USAGE.TAG_REFERENCES` when available.                                             |
| Tables and views                | `information_schema.tables` and account-usage table records, so datasets, views, and lifecycle changes land in the catalog.                                      |
| Columns                         | `information_schema.columns` and constraint metadata for names, types, nullability, defaults, comments, and key signals.                                         |
| Descriptions                    | Comments from database, schema, table, and column metadata map into catalog descriptions.                                                                        |
| Tags                            | `ACCOUNT_USAGE.TAG_REFERENCES`; non-empty tag values attach to schemas, tables, and columns.                                                                     |
| Stored procedures and functions | Account-usage procedure and function views, with `DESCRIBE PROCEDURE` / `DESCRIBE FUNCTION` as a fallback.                                                       |
| Query usage                     | `ACCOUNT_USAGE.QUERY_HISTORY` within the configured `query_log_duration` and `result_limit`.                                                                     |
| Lineage                         | Parses view definitions, eligible query-history statements, procedure child queries, and external-table locations, then resolves them against ingested metadata. |

## Source options

Metadata pipelines accept only the customer-facing `source.options` keys below. Do **not** set `source_table`, Nilus assigns a stage-specific value to each DAG node internally.

| Option               | Required | Used by stages     | Description                                                                                                                                                                                                                |
| -------------------- | -------- | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `service_type`       | Yes      | all                | Must be `snowflake`.                                                                                                                                                                                                       |
| `database_filter`    | No       | see below          | Optional include/exclude regexes. **Ignored for inventory** when the depot names a database. On `classification` / `profiler`, resolved against databases already in the catalog. Snowflake `lineage` / `usage` ignore it. |
| `schema_filter`      | No       | all                | Restrict by schema name. Same shape as `database_filter`.                                                                                                                                                                  |
| `table_filter`       | No       | all                | Restrict by table / view name. Same shape as `database_filter`.                                                                                                                                                            |
| `query_log_duration` | No       | `lineage`, `usage` | Days of query history to ingest per run. Defaults to `1`. Used by `lineage` (both modes) and `usage` (`deep` only).                                                                                                        |
| `result_limit`       | No       | `lineage`, `usage` | Maximum number of query-history rows to fetch per run. Defaults to `10000000`. Used by `lineage` (both modes) and `usage` (`deep` only).                                                                                   |

`mode` (`shallow` or `deep`) is a required `spec` field, not a `source.options` key. See [Metadata pipeline config → Choosing a mode](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/pipeline-config.md#choosing-a-mode).

## Required permissions

A metadata pipeline needs broader read access than a batch read. Source inventory works with object visibility, but tags, query history, stored procedures and functions, lineage, and usage depend on Snowflake's account-usage surface.

The metadata role should cover:

* `USAGE` on the warehouse, database, and schemas in scope;
* `SELECT` on the tables, external tables, views, and dynamic tables in scope;
* access to the account-usage surfaces Nilus reads behind the scenes:
  * `snowflake.account_usage.tables` for incremental extraction;
  * `snowflake.account_usage.tag_references` for tag ingestion;
  * `snowflake.account_usage.query_history` for lineage and usage;
  * `snowflake.account_usage.procedures` and `snowflake.account_usage.functions` for stored procedures and functions.

You can satisfy the account-usage requirement with `ACCOUNTADMIN`, a role with `IMPORTED PRIVILEGES` on the `SNOWFLAKE` database, or a replicated custom account-usage schema if your platform team exposes one instead of direct `SNOWFLAKE.ACCOUNT_USAGE`.

Example grant pattern for a dedicated metadata role:

```sql
-- Dedicated role and user for the metadata workflow.
CREATE ROLE UNDERSTAND_METADATA_ROLE;
CREATE USER UNDERSTAND_METADATA_USER
  DEFAULT_ROLE = UNDERSTAND_METADATA_ROLE
  PASSWORD = '<password>';
GRANT ROLE UNDERSTAND_METADATA_ROLE TO USER UNDERSTAND_METADATA_USER;

-- Compute and source objects in scope.
GRANT USAGE ON WAREHOUSE <warehouse_name> TO ROLE UNDERSTAND_METADATA_ROLE;
GRANT USAGE ON DATABASE <database_name> TO ROLE UNDERSTAND_METADATA_ROLE;
GRANT USAGE ON SCHEMA <database_name>.<schema_name> TO ROLE UNDERSTAND_METADATA_ROLE;

-- Assets you want the catalog to inventory.
GRANT SELECT ON ALL TABLES IN SCHEMA <database_name>.<schema_name> TO ROLE UNDERSTAND_METADATA_ROLE;
GRANT SELECT ON ALL EXTERNAL TABLES IN SCHEMA <database_name>.<schema_name> TO ROLE UNDERSTAND_METADATA_ROLE;
GRANT SELECT ON ALL VIEWS IN SCHEMA <database_name>.<schema_name> TO ROLE UNDERSTAND_METADATA_ROLE;
GRANT SELECT ON ALL DYNAMIC TABLES IN SCHEMA <database_name>.<schema_name> TO ROLE UNDERSTAND_METADATA_ROLE;
```

If your team uses direct `SNOWFLAKE.ACCOUNT_USAGE` access, also grant:

```sql
GRANT IMPORTED PRIVILEGES ON DATABASE SNOWFLAKE TO ROLE UNDERSTAND_METADATA_ROLE;
```

Key-pair authentication does not change the role requirement, the authentication method changes, but the role behind the user still needs the privileges above.

If lineage for a view is incomplete and that view depends on Snowflake tags or policies, the role may also need policy privileges:

```sql
-- Add only when policy-tagged views require them for complete lineage.
GRANT APPLY MASKING POLICY TO ROLE UNDERSTAND_METADATA_ROLE;
GRANT APPLY ROW ACCESS POLICY TO ROLE UNDERSTAND_METADATA_ROLE;
GRANT APPLY AGGREGATION POLICY TO ROLE UNDERSTAND_METADATA_ROLE;
GRANT APPLY PROJECTION POLICY TO ROLE UNDERSTAND_METADATA_ROLE;
GRANT APPLY TAG TO ROLE UNDERSTAND_METADATA_ROLE;
```

## Sample Nilus config

### Depot-backed, deep (recommended)

`mode: deep` runs the full DAG. Switch to `mode: shallow` to run only `metadata` + `lineage`.

```yaml
name: snowflake-metadata
version: v1alpha
type: nilus
tags: [nilus, metadata]
spec:
  type: metadata
  mode: deep
  compute: comet-compute
  schedule:
    crons:
      - "0 */6 * * *"
    concurrencyPolicy: Forbid
  source:
    address: dataos://snowflakemetadatadepot?purpose=rw
    options:
      service_type: snowflake
      database_filter:
        includes: ["PROD_DB"]
      schema_filter:
        includes: ["^MODEL", "^GOLD_"]
        excludes: ["^TMP_"]
      table_filter:
        excludes: ["^_audit"]
      query_log_duration: 3
      result_limit: 10000
      threads: 4          # parallel workers for profiler/usage — raise to cut runtime
```

This single resource produces a five-node DAG: `snowflake-metadata-metadata` (root) → `snowflake-metadata-lineage`, `snowflake-metadata-profiler`, `snowflake-metadata-classification`, `snowflake-metadata-usage` (parallel). All five stages run every 6 hours under a single workflow.

### Depot-backed, shallow

A `shallow` pipeline keeps inventory and lineage current without heavier scans. Use it on a tighter schedule. Pair it with a less frequent `deep` pipeline for full enrichment.

```yaml
name: snowflake-metadata-shallow
version: v1alpha
type: nilus
tags: [nilus, metadata]
spec:
  type: metadata
  mode: shallow
  compute: comet-compute
  schedule:
    crons:
      - "0 * * * *"
    concurrencyPolicy: Forbid
  source:
    address: dataos://snowflakemetadatadepot?purpose=rw
    options:
      service_type: snowflake
      database_filter:
        includes: ["PROD_DB"]
      schema_filter:
        includes: ["^MODEL", "^GOLD_"]
        excludes: ["^TMP_"]
      query_log_duration: 1
      result_limit: 10000
```

This resource produces a two-node DAG: `snowflake-metadata-shallow-metadata` (root) → `snowflake-metadata-shallow-lineage`.

### Direct URI, no depot

Use a direct `snowflake://` URI when Snowflake is not depot-backed. Project credentials through `spec.use.projection`.

```yaml
name: snowflake-metadata-direct
version: v1alpha
type: nilus
tags: [nilus, metadata]
spec:
  type: metadata
  mode: deep
  compute: comet-compute
  schedule:
    crons:
      - "0 */6 * * *"
    concurrencyPolicy: Forbid
  use:
    projection:
      secrets:
        - id: engineering:snowflake-secret
          contextAlias: snowsecret
      projections:
        envVars:
          - key: SF_USER
            template: "{{ secrets['snowsecret'].user | base64_decode }}"
          - key: SF_PASSWORD
            template: "{{ secrets['snowsecret'].password | base64_decode }}"
  source:
    address: snowflake://{SF_USER}:{SF_PASSWORD}@xy12345.snowflakecomputing.com/PROD_DB?warehouse=METADATA_WH&role=METADATA_RO
    options:
      service_type: snowflake
      database_filter:
        includes: ["PROD_DB"]
      schema_filter:
        includes: ["^MODEL"]
      query_log_duration: 3
      result_limit: 10000
```

## Behavior and capabilities

* **Connection**: use a Snowflake depot (`dataos://<depot>?purpose=rw`) or a direct `snowflake://...` URI with projected credentials. Because `spec.type` is `metadata`, Nilus adds the internal `metadata+` routing prefix automatically. A depot bundles the account, database, and warehouse (in the depot spec) with the username, optional role, and credentials (in a referenced secret) in one place, so the workflow references it instead of embedding connection details. Keep the depot name stable: the catalog tracks the source identity that arrives through it, and renaming it can fork the source entry. The depot `database` field is **required** and names exactly one database. A YAML list is rejected, and a comma-separated string is treated as one literal name.
* **`database_filter` does not expand a depot**: inventory always stays on the depot (or URI) database. Listing two databases in `includes` does not make the `metadata` stage scan two. On a **direct URI with no `/DATABASE` path**, and only then, `database_filter` drives inventory. An unfiltered URI in that mode scans every database the role can see.
* **Deep runs need an explicit filter**: with no `database_filter`, `classification` and `profiler` cover every database already in the catalog for this source, not just the depot database. Set `database_filter.includes` to that database on `mode: deep`. Names that are not already catalogued are dropped silently.
* **Lineage and usage are account-wide**: they ignore both the depot database and `database_filter`. Bound inventory with `schema_filter` / `table_filter`; do not expect lineage or usage to stay inside the depot database.
* **Identifier case**: Snowflake stores unquoted identifiers in upper-case. Use upper-case forms in filter patterns to match how Snowflake reports object names.
* **Scope discipline**: `schema_filter` / `table_filter` are the production levers on a depot-backed pipeline. An unfiltered metadata workflow against a large warehouse can take hours per stage.

## Troubleshooting

| Symptom                                                                | Likely cause                                                        | Resolution                                                                                                                               |
| ---------------------------------------------------------------------- | ------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| Inventory lands but tags, lineage, and usage are empty                 | The role lacks account-usage access.                                | Grant `IMPORTED PRIVILEGES ON DATABASE SNOWFLAKE` (or expose a replicated account-usage schema).                                         |
| Lineage incomplete for policy-tagged views                             | The role lacks the relevant policy/tag privileges.                  | Add the `APPLY ... POLICY` / `APPLY TAG` grants shown above.                                                                             |
| A stage runs for hours                                                 | The extraction scope is unbounded.                                  | Tighten `schema_filter` / `table_filter`. On a depot, `database_filter` does not shrink inventory.                                       |
| `classification` / `profiler` tagged databases the depot does not name | `database_filter` was omitted on a `deep` run.                      | Set `database_filter.includes` to the depot database.                                                                                    |
| Extra databases in `includes` never appear in inventory                | Expected: the bound database wins.                                  | Use a direct URI with no `/DATABASE` path if you need `database_filter` to drive inventory.                                              |
| Usage finishes long after inventory                                    | Usage reads and processes query history across the lookback window. | Expected, inventory is visible in the catalog before usage completes. Lower `query_log_duration` / `result_limit` to shorten the window. |

## Related docs

* [Snowflake](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/snowflake.md): batch row movement out of Snowflake.
* [Metadata sources](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources.md): all metadata-capable sources and how to scope extraction.
* [Metadata pipelines](https://v2.dataos.info/references/resources/nilus/metadata-pipelines.md): the conceptual model.
* [Metadata pipeline config](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/pipeline-config.md): the `spec.type: metadata` contract and DAG anatomy.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources/snowflake-metadata.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
