> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources/dataos-lakehouse-metadata.md).

# DataOS Lakehouse

Configuration reference for the DataOS Lakehouse metadata-only source: what inventory it brings in, source options, required depot permissions, and troubleshooting.

DataOS Lakehouse (Iceberg) is supported as a **metadata-only** source. A `spec.type: metadata` pipeline brings Lakehouse source inventory into the DataOS metadata catalog so users can find the Lakehouse-backed datasets that exist.

Unlike Snowflake and Databricks, Lakehouse metadata pipelines publish **catalog inventory only**. Lineage, profiler, classification, and usage stages are skipped at template-render time, the rendered workflow contains a single `metadata` node. For the field-by-field authoring contract, see [Metadata pipeline config](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/pipeline-config.md).

## Metadata stages

| Stage            | Status                                                            |
| ---------------- | ----------------------------------------------------------------- |
| `metadata`       | Runs, catalog, namespace, table, column, and partition inventory. |
| `lineage`        | Not generated for Lakehouse.                                      |
| `profiler`       | Not generated for Lakehouse.                                      |
| `classification` | Not generated for Lakehouse.                                      |
| `usage`          | Not generated for Lakehouse.                                      |

{% hint style="info" %}
`mode` (`shallow` or `deep`) is required on every metadata pipeline, but it does **not** change the rendered DAG for Lakehouse, only the `metadata` stage runs in either mode. Set `mode: shallow`. Because lineage, profiler, classification, and usage are never generated, `query_log_duration` and `result_limit` also have no effect.
{% endhint %}

## What Nilus brings in from Lakehouse

<table data-search="false"><thead><tr><th>What Nilus brings in</th><th>How Nilus fetches or resolves it from Lakehouse</th></tr></thead><tbody><tr><td>Catalog context</td><td>Reads the Lakehouse catalog name from the connection configuration, or uses the default catalog when one is not provided.</td></tr><tr><td>Namespaces</td><td>Uses the Lakehouse metadata interface to list namespaces.</td></tr><tr><td>Tables</td><td>Lists tables inside each namespace and loads each table's metadata before publishing it to the catalog.</td></tr><tr><td>Columns</td><td>Reads table schema fields so dataset detail pages can show the available columns and types.</td></tr><tr><td>Partition details</td><td>Reads the table partition specification and maps those fields into partition context.</td></tr><tr><td>Table descriptions</td><td>Reads description-like values from table metadata and properties when the source provides them.</td></tr><tr><td>Owners</td><td>Reads the configured ownership property when present, then resolves it to a known user before publishing ownership context.</td></tr></tbody></table>

Nilus does not bring Lakehouse tags, stored procedures, query usage, or lineage through this path.

## Source options

Metadata pipelines accept only the customer-facing `source.options` keys below. Do **not** set `source_table`, Nilus assigns the stage value internally.

| Option            | Required | Description                                                                                                                          |
| ----------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| `service_type`    | Yes      | Must be `lakehouse` (alias `iceberg`).                                                                                               |
| `database_filter` | No       | No-op. The connection binds one catalog, which appears as one database. Use `schema_filter` or `table_filter` to restrict inventory. |
| `schema_filter`   | No       | Restrict by namespace name. Same shape as `database_filter`.                                                                         |
| `table_filter`    | No       | Restrict by table name. Same shape as `database_filter`.                                                                             |

## Required permissions

Use a DataOS Lakehouse depot. The depot secret must include an `apikey` (a DataOS API token). Nilus passes it to the Iceberg REST catalog as the connection token. If the token cannot read a particular table, that table is skipped. Grant the token read access to every catalog and namespace you expect in the inventory. The metadata path also needs read access to the Lakehouse catalog, namespaces, and table metadata in scope.

## Sample Nilus config

```yaml
name: lakehouse-metadata
version: v1alpha
type: nilus
tags: [nilus, metadata]
spec:
  type: metadata
  mode: shallow
  compute: comet-compute
  schedule:
    crons:
      - "0 */6 * * *"
    concurrencyPolicy: Forbid
  source:
    address: dataos://datalakehouse?purpose=rw
    options:
      service_type: lakehouse
      schema_filter:
        includes: ["^analytics_"]
```

This resource renders a single-node `metadata` workflow. `mode` is required (set `shallow`), but it does not change the Lakehouse DAG. See the [Metadata sample configs](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/sample-configs.md#dataos-lakehouse-iceberg) for more.

## Behavior and capabilities

* **Pipeline shape**: exactly one DAG node (`metadata`) in both modes; enrichment stages are never generated.
* **Connection**: a DataOS Lakehouse depot (`dataos://<depot>?purpose=rw`). Each catalog is a separate source.
* **`database_filter`**: a no-op. It cannot narrow the bound catalog or inventory another catalog. Use `schema_filter` or `table_filter`. See [Metadata pipeline config](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/pipeline-config.md#how-database_filter-is-evaluated).
* **Product context**: if a Lakehouse-backed dataset is part of a productized flow, the product, semantic, and metric context comes from a separate product layer, not from this metadata path. Read the Lakehouse inventory alongside those product surfaces, not in isolation.

## Troubleshooting

| Symptom                                                                                                                      | Likely cause                                                                                                                                       | Resolution                                                                                                                             |
| ---------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| `service_type 'lakehouse' only supports source_table ['metadata']`                                                           | A `source_table` other than `metadata` was set, or an enrichment stage was forced.                                                                 | Remove `source_table` from the resource. Lakehouse renders only the `metadata` stage.                                                  |
| No lineage / usage / profiles appear                                                                                         | Expected, Lakehouse is metadata-only today.                                                                                                        | Use Snowflake or Databricks if you need the enrichment stages.                                                                         |
| Namespaces missing from the catalog                                                                                          | The catalog/namespace is outside the configured filters or not visible to the depot.                                                               | Check `schema_filter` and the depot's catalog scope.                                                                                   |
| Run succeeds but logs show `403` / `Authentication failed` / `Not Authorized! Invalid token.` and fewer records are returned | The depot `apikey` can reach the catalog but lacks read access to some Iceberg tables. Those tables are skipped and the run still reports success. | This is expected partial-inventory behavior. Grant the token read access to the missing catalogs, namespaces, and tables, then re-run. |

## Related docs

* [Metadata sources](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources.md): all metadata-capable sources and how to scope extraction.
* [Metadata pipelines](https://v2.dataos.info/references/resources/nilus/metadata-pipelines.md): the conceptual model.
* [Metadata pipeline config](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/pipeline-config.md): the `spec.type: metadata` contract and DAG anatomy.
* [Metadata sample configs](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/sample-configs.md): ready-to-edit YAML.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources/dataos-lakehouse-metadata.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
