Scan metadata
Scan a source so its tables, columns, lineage, and usage become discoverable in DataOS, without moving the underlying data.
Without metadata in DataOS, a source can't be explored or trusted. A metadata scan fixes that: it collects technical and business context from a connected system and registers it for discovery, without copying or moving the data. Run it when a source is connected but its tables are not showing up in the Datasets view.
A Nilus metadata pipeline is a single workflow that extracts catalog metadata, column profiles, classification tags, query lineage, and query usage from a connected source.
Metadata pipelines support 6 source systems: Snowflake, Databricks, DataOS Lakehouse, PostgreSQL, MongoDB, and Microsoft Fabric. See Metadata sources in References for per-source setup.
Choosing a mode
mode is required on every metadata pipeline. It lets you group lighter and heavier extraction work and trigger them on different cadences.
shallow
metadata + lineage
You want the source inventory and column-level lineage kept current, without the heavier profiling, classification, and usage scans. Lighter and faster, so it can run on a tighter schedule.
deep
metadata + lineage + profiler + classification + usage
You want the full enrichment: column statistics, sensitive-data tags, and query-usage analytics in addition to inventory and lineage. Heavier, so it is usually scheduled less frequently.
deep is a strict superset of shallow: it runs the same metadata and lineage stages and adds profiler, classification, and usage. A common pattern is a frequent shallow pipeline for fresh structure and lineage plus a less frequent deep pipeline for the full profile.
What each stage produces
metadata
shallow, deep
Database, schema, table, and column entities. Foundation for everything else.
lineage
shallow, deep
Column-level data lineage extracted by parsing query history.
profiler
deep
Per-column statistics (null counts, distinct counts, min/max, basic distributions).
classification
deep
Auto-classification tags applied to columns (e.g. PII heuristics).
usage
deep
Query history and usage frequency from the service's query log.
Before you begin
Make sure you have:
Access to the DataOS tenant where the metadata should be registered.
A source connection through a Depot, or an approved direct connection. Databricks and Microsoft Fabric always use a direct connection.
Permission to read metadata from the source, and query history if you want lineage and usage.
A compute profile for the pipeline.
A clear extraction scope: which databases, schemas, or tables.
The metadata scan flow
Create the Nilus pipeline definition
Create one metadata pipeline definition (.yaml file) for each source service you want to catalog.
1. Provide pipeline metadata
Field
Description
name
Unique pipeline identifier. Used as the prefix for every DAG node name ({name}-metadata, {name}-profiler, ...).
version
Use v1alpha for the current Nilus resource shape.
type
Must be nilus for Nilus-managed pipelines.
tags
Optional labels for search, grouping, and operations.
description
Optional human-readable summary.
2. Provide Nilus spec
Field
Description
spec.type
Must be metadata for catalog / lineage / profiler / classification / usage extraction.
mode
shallow or deep. Selects how much of the DAG runs: shallow is metadata + lineage; deep adds profiler, classification, and usage. Required on every metadata pipeline.
compute
Compute profile used to run each DAG node. The reference template uses comet-compute. Required.
logLevel
INFO, DEBUG, WARNING, or ERROR. Applies to every stage in the DAG.
runAsUser
Optional runtime identity. When omitted, Nilus uses the resource owner.
resources
Optional CPU and memory requests / limits applied to every DAG node. Metadata workflows are typically lightweight: 200m / 512Mi is a sensible starting point. deep runs more stages, so give it more headroom than shallow.
schedule
Optional cron schedule for the whole workflow. One cron drives every stage the mode selects.
use
Optional secret projection rules. Same projections are applied to every DAG node.
source
The source system Nilus introspects and the per-stage options. Required.
3. Choose a refresh schedule
Schedule the scan when metadata must stay current for discovery, lineage, or usage. Otherwise run it manually:
4. Choose the source and confirm the connection
The source determines what connection details and scan types are available. Use a Depot when one exists, so credentials stay out of the pipeline definition:
5. Define the scope
Do not scan everything unless you mean to. Large unfiltered warehouses take a long time. Filter to the relevant databases, schemas, and tables:
database_filter
Limit to selected databases, projects, or catalogs. Does not override a database already bound on the depot or URI.
schema_filter
Limit to selected schemas or namespaces
table_filter
Include or exclude specific tables or views
6. Decide query-history coverage
For lineage and usage, decide how much query history to collect. By default each run reads 1 day of history (query_log_duration) and up to 10000000 rows (result_limit). Set them under spec.source.options, then expand only if you need deeper history:
7. Create and run the pipeline
Write one pipeline definition per source service, then apply it:
Monitor the run:
For the full configuration reference, see Metadata pipelines and Metadata pipeline config in References.
Verify the scan
A scan is done when the assets show up where people look for them. Open Datasets in DataOS and confirm:
Source list
The connected source appears
Dataset list
Tables and views from the scoped source are visible
Overview
Columns, tags, descriptions, and update info appear
Lineage
Table or column lineage appears where query history supports it
Queries
Query activity appears when usage extraction is enabled

See Explore data → Datasets for more details.
If something is missing, review the scope, connection, service type, and permissions, then re-run. Once the assets are visible, move on to Explore data to inspect them in depth.
Last updated
Was this helpful?