> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/build/v1/understand/scan-metadata.md).

# Scan metadata

Without metadata in DataOS, a source can't be explored or trusted. A metadata scan fixes that: it collects technical and business context from a connected system and registers it for discovery, without copying or moving the data. Run it when a source is connected but its tables are not showing up in the [Datasets](/build/v1/understand/explore-data/datasets.md) view.

Nilus metadata pipeline is a single workflow that extracts catalog metadata, column profiles, classification tags, query lineage, and query usage from a connected source.

Metadata pipelines support 6 source systems: Snowflake, Databricks, DataOS Lakehouse, PostgreSQL, MongoDB, and Microsoft Fabric. See [Nilus metadata sources](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources) in References for per-source setup.

## Choosing a mode

`mode` is **required** on every metadata pipeline. It lets you group lighter and heavier extraction work and trigger them on different cadences.

| Mode      | Stages it runs (warehouse sources)                               | Use it when                                                                                                                                                                                   |
| --------- | ---------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `shallow` | `metadata` + `lineage`                                           | You want the source inventory and column-level lineage kept current, without the heavier profiling, classification, and usage scans. Lighter and faster, so it can run on a tighter schedule. |
| `deep`    | `metadata` + `lineage` + `profiler` + `classification` + `usage` | You want the full enrichment: column statistics, sensitive-data tags, and query-usage analytics in addition to inventory and lineage. Heavier, so it is usually scheduled less frequently.    |

`deep` is a strict superset of `shallow`: it runs the same `metadata` and `lineage` stages and adds `profiler`, `classification`, and `usage`. A common pattern is a frequent `shallow` pipeline for fresh structure and lineage plus a less frequent `deep` pipeline for the full profile.

## What each stage produces <a href="#what-each-stage-produces" id="what-each-stage-produces"></a>

| Stage            | Runs in           | What it lands in the catalog                                                        |
| ---------------- | ----------------- | ----------------------------------------------------------------------------------- |
| `metadata`       | `shallow`, `deep` | Database, schema, table, and column entities. Foundation for everything else.       |
| `lineage`        | `shallow`, `deep` | Column-level data lineage extracted by parsing query history.                       |
| `profiler`       | `deep`            | Per-column statistics (null counts, distinct counts, min/max, basic distributions). |
| `classification` | `deep`            | Auto-classification tags applied to columns (e.g. PII heuristics).                  |
| `usage`          | `deep`            | Query history and usage frequency from the service's query log.                     |

{% hint style="info" %}
What is available depends on the source system. After a scan, the assets appear in the Datasets, Products, lineage, and profiling views.
{% endhint %}

## Before you begin

Make sure you have:

* Access to the DataOS tenant where the metadata should be registered.
* A source connection through a [Depot](/build/v1/understand/connect-sources.md), or an approved direct connection.
* Permission to read metadata from the source, and query history if you want lineage and usage.
* A compute profile for the pipeline.
* A clear extraction scope: which databases, schemas, or tables.

## The metadata scan flow

```
Choose source -> Define scope -> Configure pipeline -> Run -> Monitor -> Verify in DataOS
```

## Create the Nilus pipeline definition

Create one metadata pipeline definition (.yaml file) for each source service you want to catalog.

### 1. Provide pipeline metadata

| <p><br>Field</p> | Description                                                                                                         |
| ---------------- | ------------------------------------------------------------------------------------------------------------------- |
| `name`           | Unique pipeline identifier. Used as the prefix for every DAG node name (`{name}-metadata`, `{name}-profiler`, ...). |
| `version`        | Use `v1alpha` for the current Nilus resource shape.                                                                 |
| `type`           | Must be `nilus` for Nilus-managed pipelines.                                                                        |
| `tags`           | Optional labels for search, grouping, and operations.                                                               |
| `description`    | Optional human-readable summary.                                                                                    |

```yaml
name: snowflake-metadata
version: v1alpha
type: nilus
tags:
  - nilus
  - metadata
description: Catalog Snowflake metadata, schema, lineage, and query usage
```

### 2. Provide Nilus spec

| <p><br>Field</p> | Description                                                                                                                                                                                                                             |
| ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `spec.type`      | Must be `metadata` for catalog / lineage / profiler / classification / usage extraction.                                                                                                                                                |
| `mode`           | `shallow` or `deep`. Selects how much of the DAG runs: `shallow` is `metadata` + `lineage`; `deep` adds `profiler`, `classification`, and `usage`. Required on every metadata pipeline.                                                 |
| `compute`        | Compute profile used to run each DAG node. The reference template uses `comet-compute`.                                                                                                                                                 |
| `logLevel`       | `INFO` (default), `DEBUG`, `WARNING`, or `ERROR`. Applies to every stage in the DAG.                                                                                                                                                    |
| `runAsUser`      | Optional runtime identity. When omitted, Nilus uses the resource owner.                                                                                                                                                                 |
| `resources`      | Optional CPU and memory requests / limits applied to **every** DAG node. Metadata workflows are typically lightweight: `200m` / `512Mi` is a sensible starting point. `deep` runs more stages, so give it more headroom than `shallow`. |
| `schedule`       | Optional cron schedule for the whole workflow. One cron drives every stage the mode selects.                                                                                                                                            |
| `use`            | Optional secret projection rules. Same projections are applied to every DAG node.                                                                                                                                                       |
| `source`         | The source system Nilus introspects and the per-stage options.                                                                                                                                                                          |

### 3. Choose a refresh schedule

Schedule the scan when metadata must stay current for discovery, lineage, or usage. Otherwise run it manually:

```yaml
spec:
  schedule:
    crons:
      - "0 */6 * * *"
    timezone: UTC
    concurrencyPolicy: Forbid
```

{% hint style="info" %}
If `schedule` is omitted, the workflow behaves like an instance that must be triggered manually.
{% endhint %}

### 4. Choose the source and confirm the connection

The source determines what connection details and scan types are available. Use a Depot when one exists, so credentials stay out of the pipeline definition:

```yaml
spec:
  source:
    address: dataos://snowflakedepot?purpose=rw
```

{% hint style="info" %}
Use a direct connector URI only when the source is not backed by a Depot and your team has approved it.
{% endhint %}

### 5. Define the scope

Do not scan everything unless you mean to. Large unfiltered warehouses take a long time. Filter to the relevant databases, schemas, and tables:

```yaml
source:
  options:
    service_type: snowflake
    database_filter:
      includes: ["SP_TEST_DB", "ANALYTICS_DB"]
    schema_filter:
      includes: ["^MODEL"]
      excludes: ["^TMP_"]
    table_filter:
      excludes: ["^_audit"]
```

| Filter            | Use it to                                          |
| ----------------- | -------------------------------------------------- |
| `database_filter` | Limit to selected databases, projects, or catalogs |
| `schema_filter`   | Limit to selected schemas or namespaces            |
| `table_filter`    | Include or exclude specific tables or views        |

### 6. Decide query-history coverage

For lineage and usage, decide how much query history to collect. Start small, then expand only if you need deeper history:

```yaml
query_log_duration: 3      # days of history per run
result_limit: 10000        # max query-history rows per run
threads: 4                 # Parallel workers for profiler/usage
```

### 7. Create and run the pipeline

Write one pipeline definition per source service, then apply it:

```bash
dataos-ctl resource apply -f ./snowflake-metadata.yaml -w <your-workspace>
```

Monitor the run:

```bash
dataos-ctl resource get -t nilus -n snowflake-metadata -w <your-workspace>
```

<details>

<summary><strong>Full example</strong>: Snowflake via Depot</summary>

```yaml
name: snowflake-metadata
version: v1alpha
type: nilus
tags:
  - nilus
  - metadata
description: Catalog Snowflake metadata, schema, lineage, and query usage
spec:
  type: metadata
  mode: deep
  compute: comet-compute
  logLevel: INFO
  resources:
    requests:
      cpu: "200m"
      memory: "512Mi"
  schedule:
    crons:
      - "0 */6 * * *"
    concurrencyPolicy: Forbid
  source:
    address: dataos://snowflakedepot?purpose=rw
    options:
      service_type: snowflake
      database_filter:
        includes:
          - "PROD_DB"
          - "ANALYTICS_DB"
      schema_filter:
        includes:
          - "^MODEL"
        excludes:
          - "^TMP_"
      table_filter:
        excludes:
          - "^_audit"
      query_log_duration: 3
      result_limit: 10000
```

</details>

{% hint style="info" %}
This example sets `mode: deep`, so single resource scans all five types of scan- (`metadata`, `lineage`, `profiler`, `classification`, `usage`). With `mode: shallow` it would scan only `metadata` and `lineage`.
{% endhint %}

For the full configuration reference, see [Nilus metadata pipelines](https://v2.dataos.info/references/resources/nilus/metadata-pipelines) in References.

## Verify the scan

A scan is done when the assets show up where people look for them. Open **Datasets** in DataOS and confirm:

| Where        | What to confirm                                                 |
| ------------ | --------------------------------------------------------------- |
| Source list  | The connected source appears                                    |
| Dataset list | Tables and views from the scoped source are visible             |
| Overview     | Columns, tags, descriptions, and update info appear             |
| Lineage      | Table or column lineage appears where query history supports it |
| Queries      | Query activity appears when usage extraction is enabled         |

<figure><img src="/files/GT1MTFGNk2xZgnJbFLOC" alt=""><figcaption></figcaption></figure>

See [Explore data → Datasets](/build/v1/understand/explore-data/datasets.md) for more details.

If something is missing, review the scope, connection, service type, and permissions, then re-run. Once the assets are visible, move on to [Explore data](/build/v1/understand/explore-data.md) to inspect them in depth.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://v2.dataos.info/build/v1/understand/scan-metadata.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.