Skip to content
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Scan metadata

Scan a source so its tables, columns, lineage, and usage become discoverable in DataOS, without moving the underlying data.

Without metadata in DataOS, a source can't be explored or trusted. A metadata scan fixes that: it collects technical and business context from a connected system and registers it for discovery, without copying or moving the data. Run it when a source is connected but its tables are not showing up in the Datasets view.

A Nilus metadata pipeline is a single workflow that extracts catalog metadata, column profiles, classification tags, query lineage, and query usage from a connected source.

Metadata pipelines support 6 source systems: Snowflake, Databricks, DataOS Lakehouse, PostgreSQL, MongoDB, and Microsoft Fabric. See Metadata sources in References for per-source setup.

Choosing a mode

mode is required on every metadata pipeline. It lets you group lighter and heavier extraction work and trigger them on different cadences.

Mode
Stages it runs (warehouse sources)
Use it when

shallow

metadata + lineage

You want the source inventory and column-level lineage kept current, without the heavier profiling, classification, and usage scans. Lighter and faster, so it can run on a tighter schedule.

deep

metadata + lineage + profiler + classification + usage

You want the full enrichment: column statistics, sensitive-data tags, and query-usage analytics in addition to inventory and lineage. Heavier, so it is usually scheduled less frequently.

deep is a strict superset of shallow: it runs the same metadata and lineage stages and adds profiler, classification, and usage. A common pattern is a frequent shallow pipeline for fresh structure and lineage plus a less frequent deep pipeline for the full profile.

What each stage produces

Stage
Runs in
What it lands in the catalog

metadata

shallow, deep

Database, schema, table, and column entities. Foundation for everything else.

lineage

shallow, deep

Column-level data lineage extracted by parsing query history.

profiler

deep

Per-column statistics (null counts, distinct counts, min/max, basic distributions).

classification

deep

Auto-classification tags applied to columns (e.g. PII heuristics).

usage

deep

Query history and usage frequency from the service's query log.

What is available depends on the source system. MongoDB has no lineage or usage stage, Microsoft Fabric has no usage stage, and DataOS Lakehouse runs only metadata in either mode. After a scan, the assets appear in the Datasets, Products, lineage, and profiling views.

Before you begin

Make sure you have:

  • Access to the DataOS tenant where the metadata should be registered.

  • A source connection through a Depot, or an approved direct connection. Databricks and Microsoft Fabric always use a direct connection.

  • Permission to read metadata from the source, and query history if you want lineage and usage.

  • A compute profile for the pipeline.

  • A clear extraction scope: which databases, schemas, or tables.

The metadata scan flow

Create the Nilus pipeline definition

Create one metadata pipeline definition (.yaml file) for each source service you want to catalog.

1. Provide pipeline metadata

Field

Description

name

Unique pipeline identifier. Used as the prefix for every DAG node name ({name}-metadata, {name}-profiler, ...).

version

Use v1alpha for the current Nilus resource shape.

type

Must be nilus for Nilus-managed pipelines.

tags

Optional labels for search, grouping, and operations.

description

Optional human-readable summary.

2. Provide Nilus spec

Field

Description

spec.type

Must be metadata for catalog / lineage / profiler / classification / usage extraction.

mode

shallow or deep. Selects how much of the DAG runs: shallow is metadata + lineage; deep adds profiler, classification, and usage. Required on every metadata pipeline.

compute

Compute profile used to run each DAG node. The reference template uses comet-compute. Required.

logLevel

INFO, DEBUG, WARNING, or ERROR. Applies to every stage in the DAG.

runAsUser

Optional runtime identity. When omitted, Nilus uses the resource owner.

resources

Optional CPU and memory requests / limits applied to every DAG node. Metadata workflows are typically lightweight: 200m / 512Mi is a sensible starting point. deep runs more stages, so give it more headroom than shallow.

schedule

Optional cron schedule for the whole workflow. One cron drives every stage the mode selects.

use

Optional secret projection rules. Same projections are applied to every DAG node.

source

The source system Nilus introspects and the per-stage options. Required.

3. Choose a refresh schedule

Schedule the scan when metadata must stay current for discovery, lineage, or usage. Otherwise run it manually:

If schedule is omitted, the workflow behaves like an instance that must be triggered manually.

4. Choose the source and confirm the connection

The source determines what connection details and scan types are available. Use a Depot when one exists, so credentials stay out of the pipeline definition:

Databricks and Microsoft Fabric have no Depot variant, so they always use a direct connector URI (databricks://..., fabric://...) with credentials projected through spec.use.projection. For other sources, use a direct URI only when the source is not backed by a Depot and your team has approved it.

5. Define the scope

Do not scan everything unless you mean to. Large unfiltered warehouses take a long time. Filter to the relevant databases, schemas, and tables:

Filter
Use it to

database_filter

Limit to selected databases, projects, or catalogs. Does not override a database already bound on the depot or URI.

schema_filter

Limit to selected schemas or namespaces

table_filter

Include or exclude specific tables or views

On MongoDB, filter Mongo databases with schema_filter, not database_filter. The catalog models each Mongo database at the schema level.

6. Decide query-history coverage

For lineage and usage, decide how much query history to collect. By default each run reads 1 day of history (query_log_duration) and up to 10000000 rows (result_limit). Set them under spec.source.options, then expand only if you need deeper history:

7. Create and run the pipeline

Write one pipeline definition per source service, then apply it:

Monitor the run:

Full example: Snowflake via Depot

This example sets mode: deep, so a single resource runs all five scan types (metadata, lineage, profiler, classification, usage). With mode: shallow it would scan only metadata and lineage.

For the full configuration reference, see Metadata pipelines and Metadata pipeline config in References.

Verify the scan

A scan is done when the assets show up where people look for them. Open Datasets in DataOS and confirm:

Where
What to confirm

Source list

The connected source appears

Dataset list

Tables and views from the scoped source are visible

Overview

Columns, tags, descriptions, and update info appear

Lineage

Table or column lineage appears where query history supports it

Queries

Query activity appears when usage extraction is enabled

See Explore data → Datasets for more details.

If something is missing, review the scope, connection, service type, and permissions, then re-run. Once the assets are visible, move on to Explore data to inspect them in depth.

Last updated

Was this helpful?