> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources.md).

# Metadata sources

Index of Nilus metadata-capable sources, their supported shallow/deep stages, and how to scope extraction with database/schema/table filters.

Nilus metadata sources are upstream systems that Nilus can introspect with a `spec.type: metadata` pipeline. A metadata pipeline does not move table rows into a destination. It publishes source inventory and, where supported, lineage, profiling, classification, and usage context into the DataOS metadata catalog.

Every metadata pipeline sets a required `mode`: `shallow` runs source inventory (`metadata`) and `lineage`; `deep` adds `profiler`, `classification`, and `usage`. Use this page to decide whether a source is metadata-capable. Use [Metadata pipeline config](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/pipeline-config.md) for the field-by-field YAML contract and [Metadata sample configs](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/sample-configs.md) for ready-to-edit examples.

## Supported metadata sources

| Source                                                                                                                                            | `service_type` | `shallow` stages      | `deep` stages                                                | Notes                                                                                                                                                                                                                                                                    |
| ------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | --------------------- | ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| [Snowflake](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources/snowflake-metadata.md)                          | `snowflake`    | `metadata`, `lineage` | `metadata`, `lineage`, `profiler`, `classification`, `usage` | Use a Snowflake depot or a `snowflake://...` URI. Grant access to object metadata, account-usage views, query history, tags, procedures, and functions when lineage and usage are required.                                                                              |
| [Databricks](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources/databricks-metadata.md)                        | `databricks`   | `metadata`, `lineage` | `metadata`, `lineage`, `profiler`, `classification`, `usage` | Connect through a `databricks://...` URI. There is no DataOS depot variant for Databricks. For Unity Catalog, grant the token principal catalog, schema, and table visibility for the extraction scope.                                                                  |
| [PostgreSQL](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources/postgresql.md)                                 | `postgres`     | `metadata`, `lineage` | `metadata`, `lineage`, `profiler`, `classification`, `usage` | Use a PostgreSQL depot or a `postgresql://...` URI. Lineage and usage require the `pg_stat_statements` extension and `pg_read_all_stats`.                                                                                                                                |
| [MongoDB](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources/mongodb.md)                                       | `mongodb`      | `metadata`            | `metadata`, `profiler`, `classification`                     | Use a MongoDB depot or a `mongodb://...` URI. Fields are inferred from sampled documents because MongoDB is schema-less. **Lineage and usage are not supported.**                                                                                                        |
| [Microsoft Fabric Lakehouse](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources/microsoft-fabric-lakehouse.md) | `fabric`       | `metadata`, `lineage` | `metadata`, `lineage`, `profiler`, `classification`          | Use a direct `fabric://...` URI against the Lakehouse **SQL analytics endpoint**. Configure a Microsoft Entra service principal and an explicit port (typically `1433`). There is no DataOS depot variant. Same connector as the Fabric Warehouse. **No `usage` stage.** |
| [Microsoft Fabric Warehouse](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources/microsoft-fabric.md)           | `fabric`       | `metadata`, `lineage` | `metadata`, `lineage`, `profiler`, `classification`          | Use a direct `fabric://...` URI with a Microsoft Entra service principal and an explicit port (typically `1433`). There is no DataOS depot variant. **No `usage` stage.**                                                                                                |
| [DataOS Lakehouse](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources/dataos-lakehouse-metadata.md)            | `lakehouse`    | `metadata` only       | `metadata` only                                              | Use a DataOS Lakehouse depot. The Lakehouse metadata path publishes catalog inventory only in both modes; lineage, profiler, classification, and usage are skipped.                                                                                                      |

## How to choose scope

Metadata extraction should be deliberately bounded in production. Filters do **not** all mean the same thing:

* Use `mode` to choose how much runs: `shallow` for inventory + lineage on a tight schedule, `deep` for the full enrichment less often.
* Use `schema_filter` and `table_filter` to bound schemas/namespaces and tables on every connector. These are the reliable production levers.
* Use `database_filter` only when you understand [Metadata pipeline config](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/pipeline-config.md). A depot or URI that already names a database **wins**: inventory stays on that database, extra `includes` names do not add databases, and classification/profiler resolve the filter against what is already in the catalog.
* On MongoDB, filter Mongo databases with `schema_filter`, not `database_filter`.
* Use `query_log_duration` and `result_limit` only for warehouse sources where lineage and usage are needed.

The same authored Nilus resource becomes one workflow. For full-DAG sources (Snowflake, Databricks, PostgreSQL, Microsoft Fabric), Nilus runs the source inventory stage first, then runs the stages the mode selects (lineage in `shallow`; lineage, profiler, classification, and usage in `deep`) after inventory succeeds. Some sources support only a subset of stages: **Microsoft Fabric** (Warehouse and Lakehouse) runs `metadata`, `lineage`, `profiler`, and `classification` but **not `usage`** (`deep` is a 4-node DAG); **MongoDB** supports only `metadata`, `profiler`, and `classification` (no lineage or usage); and **DataOS Lakehouse** renders only the `metadata` stage in either mode. Stages a source does not support produce no output even in `deep`.

## Before you configure a source

* Confirm that the connection has metadata-read permissions, not just data-read permissions.
* Confirm that query-history access is available before expecting lineage or usage output.
* For Snowflake, confirm that the role can read the approved account-usage surface before expecting tags, query history, stored procedure/function context, lineage, or usage output.
* Keep the source identity stable. Changing the depot name or service identity can create a new source entry in the catalog instead of updating the existing one.
* Do not set `source_table` on a metadata resource. Nilus assigns stage-specific values internally.

## Connector page structure

Each metadata-capable source page should include requirements, metadata-specific permissions, supported metadata stages, sample Nilus config, behavior and capabilities, troubleshooting, and related docs.

## Related docs

* [Metadata pipelines](https://v2.dataos.info/references/resources/nilus/metadata-pipelines.md)
* [Metadata pipeline config](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/pipeline-config.md)
* [Metadata sample configs](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/sample-configs.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
