DataOS Lakehouse
Configuration reference for the DataOS Lakehouse metadata-only source: what inventory it brings in, source options, required depot permissions, and troubleshooting.
DataOS Lakehouse (Iceberg) is supported as a metadata-only source. A spec.type: metadata pipeline brings Lakehouse source inventory into the DataOS metadata catalog so users can find the Lakehouse-backed datasets that exist.
Unlike Snowflake and Databricks, Lakehouse metadata pipelines publish catalog inventory only. Lineage, profiler, classification, and usage stages are skipped at template-render time, the rendered workflow contains a single metadata node. For the field-by-field authoring contract, see Metadata pipeline config.
Metadata stages
metadata
Runs, catalog, namespace, table, column, and partition inventory.
lineage
Not generated for Lakehouse.
profiler
Not generated for Lakehouse.
classification
Not generated for Lakehouse.
usage
Not generated for Lakehouse.
What Nilus brings in from Lakehouse
Catalog context
Reads the Lakehouse catalog name from the connection configuration, or uses the default catalog when one is not provided.
Namespaces
Uses the Lakehouse metadata interface to list namespaces.
Tables
Lists tables inside each namespace and loads each table's metadata before publishing it to the catalog.
Columns
Reads table schema fields so dataset detail pages can show the available columns and types.
Partition details
Reads the table partition specification and maps those fields into partition context.
Table descriptions
Reads description-like values from table metadata and properties when the source provides them.
Owners
Reads the configured ownership property when present, then resolves it to a known user before publishing ownership context.
Nilus does not bring Lakehouse tags, stored procedures, query usage, or lineage through this path.
Source options
Metadata pipelines accept only the customer-facing source.options keys below. Do not set source_table, Nilus assigns the stage value internally.
service_type
Yes
Must be lakehouse (alias iceberg).
database_filter
No
No-op. The connection binds one catalog, which appears as one database. Use schema_filter or table_filter to restrict inventory.
schema_filter
No
Restrict by namespace name. Same shape as database_filter.
table_filter
No
Restrict by table name. Same shape as database_filter.
Required permissions
Use a DataOS Lakehouse depot. The depot secret must include an apikey (a DataOS API token). Nilus passes it to the Iceberg REST catalog as the connection token. If the token cannot read a particular table, that table is skipped. Grant the token read access to every catalog and namespace you expect in the inventory. The metadata path also needs read access to the Lakehouse catalog, namespaces, and table metadata in scope.
Sample Nilus config
This resource renders a single-node metadata workflow. mode is required (set shallow), but it does not change the Lakehouse DAG. See the DataOS Lakehouse (Iceberg) for more.
Behavior and capabilities
Pipeline shape: exactly one DAG node (
metadata) in both modes; enrichment stages are never generated.Connection: a DataOS Lakehouse depot (
dataos://<depot>?purpose=rw). Each catalog is a separate source.database_filter: a no-op. It cannot narrow the bound catalog or inventory another catalog. Useschema_filterortable_filter. See How database_filter is evaluated.Product context: if a Lakehouse-backed dataset is part of a productized flow, the product, semantic, and metric context comes from a separate product layer, not from this metadata path. Read the Lakehouse inventory alongside those product surfaces, not in isolation.
Troubleshooting
service_type 'lakehouse' only supports source_table ['metadata']
A source_table other than metadata was set, or an enrichment stage was forced.
Remove source_table from the resource. Lakehouse renders only the metadata stage.
No lineage / usage / profiles appear
Expected, Lakehouse is metadata-only today.
Use Snowflake or Databricks if you need the enrichment stages.
Namespaces missing from the catalog
The catalog/namespace is outside the configured filters or not visible to the depot.
Check schema_filter and the depot's catalog scope.
Run succeeds but logs show 403 / Authentication failed / Not Authorized! Invalid token. and fewer records are returned
The depot apikey can reach the catalog but lacks read access to some Iceberg tables. Those tables are skipped and the run still reports success.
This is expected partial-inventory behavior. Grant the token read access to the missing catalogs, namespaces, and tables, then re-run.
Related docs
Metadata sources: all metadata-capable sources and how to scope extraction.
Metadata pipelines: the conceptual model.
Metadata pipeline config: the
spec.type: metadatacontract and DAG anatomy.Metadata sample configs: ready-to-edit YAML.
Last updated
Was this helpful?