> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/v1/resources/nilus/metadata-pipelines/sample-configs.md). # Metadata sample configs All examples below use the current Nilus pipeline shape with `spec.type: metadata`: ```yaml type: nilus spec: type: metadata ``` A metadata pipeline extracts catalog, lineage, profiler, classification, and usage information from a connected source system and loads it into the DataOS metadata catalog. You author **one** Nilus resource per source system — Nilus renders it into a DAG behind the scenes. The required `mode` field decides how much of that DAG runs: `shallow` renders `metadata` + `lineage`, and `deep` adds `profiler`, `classification`, and `usage` in parallel after the root. For Lakehouse sources, only the `metadata` stage runs in either mode. For the conceptual model, see [Understanding Metadata Pipelines](/references/v1/resources/nilus/metadata-pipelines.md). For the field-by-field reference and the DAG anatomy, see [Understanding Metadata Pipeline Config](/references/v1/resources/nilus/metadata-pipelines/pipeline-config.md). {% hint style="info" %} **Supported sources.** Metadata pipelines are documented here for **Snowflake**, **Databricks**, **PostgreSQL**, **MongoDB**, **Microsoft Fabric Warehouse**, and **DataOS Lakehouse** (Iceberg). Not every source supports every stage: **MongoDB** has no lineage or usage, **Microsoft Fabric** has no usage, and **DataOS Lakehouse** runs `metadata` only. See each connector page for the exact stage matrix. {% endhint %} {% hint style="info" %} `mode` (`shallow` or `deep`) is required on every metadata pipeline. You do **not** set `source_table` on a metadata resource — Nilus hardcodes it per DAG node. The `source.options` block accepts only the seven fields shown in these samples (`service_type`, `database_filter`, `schema_filter`, `table_filter`, `query_log_duration`, `result_limit`, `threads`); anything else is rejected at schema validation. {% endhint %} ### Snowflake Before running either Snowflake sample, make sure the Snowflake role has the metadata-specific privileges described in [Snowflake (Metadata) → Required permissions](/references/v1/resources/nilus/metadata-pipelines/metadata-sources/snowflake-metadata.md#required-permissions). Inventory can work with object visibility, but tags, query history, stored procedures/functions, lineage, and usage depend on the approved account-usage surface.
Depot-backed, deep (recommended) `mode: deep` runs the full DAG. Switch to `mode: shallow` to run only `metadata` + `lineage`. ```yaml name: snowflake-metadata version: v1alpha type: nilus tags: [nilus, metadata] spec: type: metadata mode: deep compute: comet-compute schedule: crons: - "0 */6 * * *" concurrencyPolicy: Forbid source: address: dataos://snowflakemetadatadepot?purpose=rw options: service_type: snowflake database_filter: includes: ["PROD_DB", "ANALYTICS_DB"] schema_filter: includes: ["^MODEL", "^GOLD_"] excludes: ["^TMP_"] table_filter: excludes: ["^_audit"] query_log_duration: 3 result_limit: 10000 threads: 4 # parallel workers for profiler/usage — raise to cut runtime ``` This single resource produces a 5-node DAG: `snowflake-metadata-metadata` (root) → `snowflake-metadata-lineage`, `snowflake-metadata-profiler`, `snowflake-metadata-classification`, `snowflake-metadata-usage` (parallel). All five stages run every 6 hours under a single workflow.
Depot-backed, shallow (inventory + lineage only) A `shallow` pipeline keeps the source inventory and lineage current without the heavier profiling, classification, and usage scans. It is lighter, so it suits a tighter schedule. Pair it with a less frequent `deep` pipeline when you also need the full enrichment. ```yaml name: snowflake-metadata-shallow version: v1alpha type: nilus tags: [nilus, metadata] spec: type: metadata mode: shallow compute: comet-compute schedule: crons: - "0 * * * *" concurrencyPolicy: Forbid source: address: dataos://snowflakemetadatadepot?purpose=rw options: service_type: snowflake database_filter: includes: ["PROD_DB", "ANALYTICS_DB"] schema_filter: includes: ["^MODEL", "^GOLD_"] excludes: ["^TMP_"] query_log_duration: 1 result_limit: 10000 ``` This resource produces a 2-node DAG: `snowflake-metadata-shallow-metadata` (root) → `snowflake-metadata-shallow-lineage`.
Direct URI (no depot) Use the direct `snowflake://` URI when the Snowflake account is not depot-backed. Because `spec.type` is `metadata`, Nilus adds the internal `metadata+` routing prefix automatically. Project credentials through `spec.use.projection`. ```yaml name: snowflake-metadata-direct version: v1alpha type: nilus tags: [nilus, metadata] spec: type: metadata mode: deep compute: comet-compute schedule: crons: - "0 */6 * * *" concurrencyPolicy: Forbid use: projection: secrets: - id: engineering:snowflake-secret contextAlias: snowsecret projections: envVars: - key: SF_USER template: "{{ secrets['snowsecret'].user | base64_decode }}" - key: SF_PASSWORD template: "{{ secrets['snowsecret'].password | base64_decode }}" source: address: snowflake://{SF_USER}:{SF_PASSWORD}@xy12345.snowflakecomputing.com/PROD_DB?warehouse=METADATA_WH&role=METADATA_RO options: service_type: snowflake database_filter: includes: ["PROD_DB"] schema_filter: includes: ["^MODEL"] query_log_duration: 3 result_limit: 10000 ```
### Databricks (Unity Catalog) Databricks metadata connects only through a direct `databricks://` URI with a projected access token — **there is no DataOS depot variant for Databricks**. Because `spec.type` is `metadata`, Nilus adds the internal `metadata+` routing prefix automatically. Use `service_type: databricks` for both classic and Unity Catalog deployments; the access token goes in the URI password position (`token:@`) and `http_path` is a query parameter. ```yaml name: databricks-metadata version: v1alpha type: nilus tags: [nilus, metadata] spec: type: metadata mode: deep compute: comet-compute schedule: crons: - "0 */6 * * *" concurrencyPolicy: Forbid use: projection: secrets: - id: engineering:databricks-secret contextAlias: dbxsecret projections: envVars: - key: DBX_TOKEN template: "{{ secrets['dbxsecret'].token | base64_decode }}" source: address: databricks://token:{DBX_TOKEN}@adb-12345.6.azuredatabricks.net?http_path=/sql/1.0/warehouses/abc123def456&catalog=main&schema=gold options: service_type: databricks database_filter: includes: ["main"] schema_filter: includes: ["^gold_", "^silver_"] excludes: ["^bronze_tmp_"] query_log_duration: 3 result_limit: 10000 ``` ### PostgreSQL PostgreSQL supports the full DAG. Lineage and usage read query statistics from the `pg_stat_statements` extension — enable it and grant `pg_read_all_stats` before expecting lineage/usage output. See [PostgreSQL (Metadata) → Required permissions](/references/v1/resources/nilus/metadata-pipelines/metadata-sources/postgresql.md#required-permissions). ```yaml name: postgresql-metadata version: v1alpha type: nilus tags: [nilus, metadata] spec: type: metadata mode: deep compute: comet-compute schedule: crons: - "0 */6 * * *" concurrencyPolicy: Forbid source: address: dataos://postgresmetadatadepot?purpose=rw options: service_type: postgres database_filter: includes: ["warehouse"] schema_filter: includes: ["^public$", "^analytics_"] excludes: ["^pg_", "^information_schema$"] query_log_duration: 3 result_limit: 10000 ``` ### MongoDB MongoDB supports `metadata`, `profiler`, and `classification` only — there is no lineage or usage. Fields are inferred from sampled documents. ```yaml name: mongodb-metadata version: v1alpha type: nilus tags: [nilus, metadata] spec: type: metadata mode: deep compute: comet-compute schedule: crons: - "0 */6 * * *" concurrencyPolicy: Forbid source: address: dataos://mongometadatadepot?purpose=rw options: service_type: mongodb database_filter: includes: ["appdb", "analytics"] excludes: ["^admin$", "^local$", "^config$"] table_filter: excludes: ["^system\\."] ``` ### Microsoft Fabric Warehouse Microsoft Fabric connects only through a direct `fabric://` URI with a Microsoft Entra service principal — there is no depot variant, and usage is not supported. Because `spec.type` is `metadata`, Nilus adds the internal `metadata+` routing prefix automatically. The client ID goes in the username position, the projected client secret in the password position, a mandatory port (typically `1433`) follows the Fabric hostname, the warehouse goes in the path, and the Microsoft Entra **directory (tenant) ID** — a GUID, **not** your DataOS tenant/workspace name — is passed as the required `tenant_id` query parameter. See [Microsoft Fabric Warehouse (Metadata) → Authentication and prerequisites](/references/v1/resources/nilus/metadata-pipelines/metadata-sources/microsoft-fabric.md#authentication-and-prerequisites). ```yaml name: fabric-metadata version: v1alpha type: nilus tags: [nilus, metadata] spec: type: metadata mode: deep compute: comet-compute schedule: crons: - "0 */6 * * *" concurrencyPolicy: Forbid use: projection: secrets: - id: engineering:fabric-secret contextAlias: fabricsecret projections: envVars: - key: FABRIC_CLIENT_ID template: "{{ secrets['fabricsecret'].client_id | base64_decode }}" - key: FABRIC_CLIENT_SECRET template: "{{ secrets['fabricsecret'].client_secret | base64_decode }}" source: address: fabric://{FABRIC_CLIENT_ID}:{FABRIC_CLIENT_SECRET}@abcd1234.datawarehouse.fabric.microsoft.com:1433/analytics_wh?tenant_id=11111111-2222-3333-4444-555555555555 options: service_type: fabric schema_filter: includes: ["^dbo$", "^gold_"] query_log_duration: 3 result_limit: 10000 ``` ### DataOS Lakehouse (Iceberg)
Metadata only ```yaml name: lakehouse-metadata version: v1alpha type: nilus tags: [nilus, metadata] spec: type: metadata mode: shallow compute: comet-compute schedule: crons: - "0 */6 * * *" concurrencyPolicy: Forbid source: address: dataos://datalakehouse?purpose=rw options: service_type: lakehouse schema_filter: includes: ["^analytics_"] ``` {% hint style="info" %} When `service_type: lakehouse`, the rendered workflow contains **only** the `metadata` DAG node in both modes — lineage, profiler, classification, and usage stages are skipped at template-render time. `mode` is still required by the schema (set `shallow`), but it does not change the rendered DAG. `query_log_duration` and `result_limit` have no effect because the stages that consume them aren't generated. {% endhint %}
## Validation Notes * Use `type: nilus` and `spec.type: metadata` for catalog / lineage / profiler / classification / usage extraction. * `mode` is required and must be `shallow` or `deep`. `shallow` runs `metadata` + `lineage`; `deep` adds `profiler`, `classification`, and `usage`. A metadata resource without `mode` fails schema validation. * `source.options.service_type` is required. This guide documents `snowflake`, `databricks`, `postgres` (alias `postgresql`), `mongodb`, `fabric`, and `lakehouse` (alias `iceberg`). * Keep the customer-facing `source.options` block to `service_type`, `database_filter`, `schema_filter`, `table_filter`, `query_log_duration`, `result_limit`, and `threads` (parallel worker count — raise it to cut runtime on large `profiler` / `usage` workflows). Don't add `source_table`, `stored_procedure_filter`, `classification_filter`, or other lower-level engine knobs until the Nilus domain template exposes them. * One Nilus resource yields one workflow. For full-DAG sources (`snowflake` / `databricks` / `postgres`) that workflow is a 2-stage DAG in `shallow` and a 5-stage DAG in `deep`. `fabric` omits `usage` (4-stage `deep`); `mongodb` omits `lineage` and `usage` (`metadata` only in `shallow`, 3-stage `deep`); `lakehouse` is a 1-stage DAG in both modes. Don't create separate Nilus resources to get separate stages. * Always set `database_filter` / `schema_filter` / `table_filter` on production scopes. Unbounded sweeps over a large warehouse can take hours per stage, and the cost compounds across every scheduled run. * `query_log_duration` and `result_limit` affect `lineage` (both modes) and `usage` (`deep` only). They're ignored by `metadata`, `profiler`, and `classification`. * Metadata pipelines do not declare a `sink`. The catalog target is configured at the DataOS metadata-service level — Nilus auto-attaches the catalog sink to every DAG node during translation. ## Related Docs * [Understanding Metadata Pipelines](/references/v1/resources/nilus/metadata-pipelines.md) * [Metadata Sources](/references/v1/resources/nilus/metadata-pipelines/metadata-sources.md) * [Understanding Metadata Pipeline Config](/references/v1/resources/nilus/metadata-pipelines/pipeline-config.md) * [Batch Sample Configs](/references/v1/resources/nilus/batch/sample-configs.md) * [CDC Sample Configs](/references/v1/resources/nilus/cdc/sample-configs.md) * [Stream Sample Configs](/references/v1/resources/nilus/stream/sample-configs.md) * [Secrets and Projections](/references/v1/resources/nilus/concepts/secrets-and-projections.md) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://v2.dataos.info/references/v1/resources/nilus/metadata-pipelines/sample-configs.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.