For the complete documentation index, see llms.txt. Markdown versions of documentation pages are available by appending .md to page URLs.
Skip to content

DataOS Lakehouse

Configuration reference for the DataOS Lakehouse metadata-only source: what inventory it brings in, source options, required depot permissions, and troubleshooting.

DataOS Lakehouse (Iceberg) is supported as a metadata-only source. A spec.type: metadata pipeline brings Lakehouse source inventory into the DataOS metadata catalog so users can find the Lakehouse-backed datasets that exist.

Unlike Snowflake and Databricks, Lakehouse metadata pipelines publish catalog inventory only. Lineage, profiler, classification, and usage stages are skipped at template-render time, the rendered workflow contains a single metadata node. For the field-by-field authoring contract, see Metadata pipeline config.

Metadata stages

Stage
Status

metadata

Runs, catalog, namespace, table, column, and partition inventory.

lineage

Not generated for Lakehouse.

profiler

Not generated for Lakehouse.

classification

Not generated for Lakehouse.

usage

Not generated for Lakehouse.

mode (shallow or deep) is required on every metadata pipeline, but it does not change the rendered DAG for Lakehouse, only the metadata stage runs in either mode. Set mode: shallow. Because lineage, profiler, classification, and usage are never generated, query_log_duration and result_limit also have no effect.

What Nilus brings in from Lakehouse

What Nilus brings in
How Nilus fetches or resolves it from Lakehouse

Catalog context

Reads the Lakehouse catalog name from the connection configuration, or uses the default catalog when one is not provided.

Namespaces

Uses the Lakehouse metadata interface to list namespaces.

Tables

Lists tables inside each namespace and loads each table's metadata before publishing it to the catalog.

Columns

Reads table schema fields so dataset detail pages can show the available columns and types.

Partition details

Reads the table partition specification and maps those fields into partition context.

Table descriptions

Reads description-like values from table metadata and properties when the source provides them.

Owners

Reads the configured ownership property when present, then resolves it to a known user before publishing ownership context.

Nilus does not bring Lakehouse tags, stored procedures, query usage, or lineage through this path.

Source options

Metadata pipelines accept only the customer-facing source.options keys below. Do not set source_table, Nilus assigns the stage value internally.

Option
Required
Description

service_type

Yes

Must be lakehouse (alias iceberg).

database_filter

No

No-op. The connection binds one catalog, which appears as one database. Use schema_filter or table_filter to restrict inventory.

schema_filter

No

Restrict by namespace name. Same shape as database_filter.

table_filter

No

Restrict by table name. Same shape as database_filter.

Required permissions

Use a DataOS Lakehouse depot. The depot secret must include an apikey (a DataOS API token). Nilus passes it to the Iceberg REST catalog as the connection token. If the token cannot read a particular table, that table is skipped. Grant the token read access to every catalog and namespace you expect in the inventory. The metadata path also needs read access to the Lakehouse catalog, namespaces, and table metadata in scope.

Sample Nilus config

This resource renders a single-node metadata workflow. mode is required (set shallow), but it does not change the Lakehouse DAG. See the DataOS Lakehouse (Iceberg) for more.

Behavior and capabilities

  • Pipeline shape: exactly one DAG node (metadata) in both modes; enrichment stages are never generated.

  • Connection: a DataOS Lakehouse depot (dataos://<depot>?purpose=rw). Each catalog is a separate source.

  • database_filter: a no-op. It cannot narrow the bound catalog or inventory another catalog. Use schema_filter or table_filter. See How database_filter is evaluated.

  • Product context: if a Lakehouse-backed dataset is part of a productized flow, the product, semantic, and metric context comes from a separate product layer, not from this metadata path. Read the Lakehouse inventory alongside those product surfaces, not in isolation.

Troubleshooting

Symptom
Likely cause
Resolution

service_type 'lakehouse' only supports source_table ['metadata']

A source_table other than metadata was set, or an enrichment stage was forced.

Remove source_table from the resource. Lakehouse renders only the metadata stage.

No lineage / usage / profiles appear

Expected, Lakehouse is metadata-only today.

Use Snowflake or Databricks if you need the enrichment stages.

Namespaces missing from the catalog

The catalog/namespace is outside the configured filters or not visible to the depot.

Check schema_filter and the depot's catalog scope.

Run succeeds but logs show 403 / Authentication failed / Not Authorized! Invalid token. and fewer records are returned

The depot apikey can reach the catalog but lacks read access to some Iceberg tables. Those tables are skipped and the run still reports success.

This is expected partial-inventory behavior. Grant the token read access to the missing catalogs, namespaces, and tables, then re-run.

Last updated

Was this helpful?