> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources/microsoft-fabric-lakehouse.md).

# Microsoft Fabric Lakehouse

Configure Nilus metadata pipelines for Microsoft Fabric Lakehouse SQL analytics endpoints.

[Microsoft Fabric](https://learn.microsoft.com/en-us/fabric/) is supported as a metadata source through its **Lakehouse**. Nilus reads the Lakehouse's **SQL analytics endpoint** to catalog its tables and views. A `spec.type: metadata` pipeline introspects a Fabric Lakehouse and publishes source context, including workspaces, lakehouses, schemas, tables, views, columns, lineage, profiles, and classification, into the DataOS metadata catalog **without copying any table rows**. Nilus connects over ODBC using a Microsoft Entra service principal.

This is the same `service_type: fabric` connector used for the Fabric **Warehouse**; the difference is only which SQL analytics endpoint you point it at. For the Warehouse variant, see [Microsoft Fabric Warehouse metadata](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources/microsoft-fabric.md). For batch loading into a Lakehouse, see [Microsoft Fabric Lakehouse destination](https://v2.dataos.info/references/resources/nilus/destinations/microsoft-fabric-lakehouse.md).

For the field-by-field authoring contract, see [Metadata pipeline config](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/pipeline-config.md).

## Metadata stages

`service_type: fabric` supports `metadata`, `lineage`, `profiler`, and `classification`. The required `mode` field decides how much of the DAG runs: `shallow` runs `metadata` + `lineage`; `deep` adds `profiler` and `classification`. The `usage` stage is **not supported** for Microsoft Fabric. Nilus skips it even in `deep` mode.

| Stage            | Runs in           | What it lands in the catalog                                                                                                                              |
| ---------------- | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `metadata`       | `shallow`, `deep` | Workspaces, lakehouses, schemas, tables, views, and columns, including data types, nullability, precision/scale, and object descriptions where available. |
| `lineage`        | `shallow`, `deep` | Asset and column lineage parsed from view definitions and query history.                                                                                  |
| `profiler`       | `deep`            | Per-column statistics (row counts, null counts, distinct counts, min/max, basic distributions).                                                           |
| `classification` | `deep`            | Auto-classification tags applied to columns from sampled data (PII heuristics).                                                                           |

`query_log_duration` and `result_limit` tune the `lineage` stage.

## Asset hierarchy

Microsoft Fabric Lakehouse assets map into the Datasets App as `Workspace → Lakehouse → Schema → Table/View → Column`. The workspace and lakehouse are derived from the SQL analytics endpoint and the lakehouse name in the connection; schemas, tables, views, and columns are read from the lakehouse's SQL analytics catalog.

## Source options

Metadata pipelines accept only the customer-facing `source.options` keys below. Do **not** set `source_table`. Nilus assigns a stage-specific value to each DAG node internally.

| Option               | Required | Used by stages | Description                                                                            |
| -------------------- | -------- | -------------- | -------------------------------------------------------------------------------------- |
| `service_type`       | Yes      | all            | Must be `fabric`.                                                                      |
| `schema_filter`      | No       | all            | Restrict by schema name. Object with `includes` / `excludes` arrays of regex patterns. |
| `table_filter`       | No       | all            | Restrict by table / view name. Same shape as `schema_filter`.                          |
| `query_log_duration` | No       | `lineage`      | Days of query history to ingest per run. Defaults to `1`.                              |
| `result_limit`       | No       | `lineage`      | Maximum number of query-history rows to fetch per run. Defaults to `10000000`.         |
| `threads`            | No       | `profiler`     | Parallel worker count for the profiler. Raise it to cut runtime on large scopes.       |

`mode` (`shallow` or `deep`) is a required `spec` field, not a `source.options` key.

## Authentication and prerequisites

Microsoft Fabric Lakehouse metadata connects through a **Microsoft Entra service principal**. There is no DataOS depot variant for Fabric. Before configuring the pipeline:

{% stepper %}
{% step %}
**Register an Entra application**

Register an Entra application (App registration) in the Fabric tenant. Note its **Application (client) ID**, **Directory (tenant) ID**, and create a **client secret**.
{% endstep %}

{% step %}
**Record the Lakehouse SQL analytics endpoint and port**

The URI must include an explicit port after the endpoint host (typically `1433`). Nilus rejects a Fabric URI without a port.
{% endstep %}

{% step %}
**Enable service principal access in Fabric**

In the Fabric/Power BI admin tenant settings, allow service principals to use Fabric APIs (and, where required, add the app to the security group permitted to do so).
{% endstep %}

{% step %}
**Grant the service principal access to the workspace**

Add the app as a member of the Fabric workspace that contains the Lakehouse (Viewer role is sufficient for read-only metadata).
{% endstep %}

{% step %}
**Grant read on the Lakehouse**

In the Lakehouse's SQL analytics endpoint, create a database user for the service principal and grant it read:

```sql
CREATE USER [<app-registration-name>] FROM EXTERNAL PROVIDER;
GRANT CONNECT TO [<app-registration-name>];
-- Inventory, profiling, and classification (sampling).
GRANT SELECT ON DATABASE::<lakehouse_name> TO [<app-registration-name>];
-- Lineage (parsing view definitions).
GRANT VIEW DEFINITION ON DATABASE::<lakehouse_name> TO [<app-registration-name>];
```

{% endstep %}
{% endstepper %}

{% hint style="info" %}
Store the client secret in a DataOS secret and project it into the pipeline; never inline the literal secret in the manifest.
{% endhint %}

## Sample Nilus config

Fabric metadata connects only through a direct `fabric://` URI. Because `spec.type` is `metadata`, Nilus adds the internal `metadata+` routing prefix automatically. The service principal's **client ID** goes in the username position, the **client secret** in the password position (projected), the Lakehouse's **SQL analytics endpoint** host and its mandatory **port** follow, the lakehouse name goes in the path, and the Microsoft Entra **directory (tenant) ID** is passed as the required `tenant_id` query parameter.

{% hint style="info" %}
The `tenant_id` here is the **Microsoft Entra (Azure AD) directory tenant ID**. It is a GUID, for example `11111111-2222-3333-4444-555555555555`, taken from your Fabric app registration. It is **not** your DataOS tenant or workspace name.
{% endhint %}

You can project the SQL analytics endpoint host itself from a secret (for example `FABRIC_SQL_ENDPOINT`) and reference it in the URI; Nilus resolves the projected value before generating the catalog service ID, so a projected endpoint produces the same service ID as an inline one.

```yaml
name: fabric-lakehouse-metadata
version: v1alpha
type: nilus
tags: [nilus, metadata]
description: Catalog Microsoft Fabric Lakehouse metadata, schema, lineage, and classification
spec:
  type: metadata
  mode: deep
  compute: comet-compute
  schedule:
    crons:
      - "0 */6 * * *"
    concurrencyPolicy: Forbid
  use:
    projection:
      secrets:
        - id: engineering:fabric-lakehouse-secret
          contextAlias: fabriclhsecret
      projections:
        envVars:
          - key: FABRIC_CLIENT_ID
            template: "{{ secrets['fabriclhsecret'].client_id | base64_decode }}"
          - key: FABRIC_CLIENT_SECRET
            template: "{{ secrets['fabriclhsecret'].client_secret | base64_decode }}"
          - key: FABRIC_SQL_ENDPOINT
            template: "{{ secrets['fabriclhsecret'].sql_endpoint | base64_decode }}"
  source:
    address: fabric://{FABRIC_CLIENT_ID}:{FABRIC_CLIENT_SECRET}@{FABRIC_SQL_ENDPOINT}:1433/analytics_lh?tenant_id=11111111-2222-3333-4444-555555555555
    options:
      service_type: fabric
      schema_filter:
        includes: ["^dbo$", "^gold_"]
        excludes: ["^staging_"]
      table_filter:
        excludes: ["^_tmp"]
      query_log_duration: 3
      result_limit: 10000
      threads: 4
```

The `hostPort` is the Lakehouse's **SQL analytics endpoint plus its mandatory port** (for example `<id>.datawarehouse.fabric.microsoft.com:1433`), and the path segment is the **lakehouse** name. Nilus connects with the `ODBC Driver 18 for SQL Server` by default; override it with a `driver` query parameter only if your runtime ships a different ODBC driver.

With `mode: deep`, this resource produces a four-node DAG (`metadata` root → `lineage`, `profiler`, `classification`). Switch to `mode: shallow` for a 2-node `metadata` + `lineage` DAG. For more ready-to-edit examples, see [Metadata sample configs](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/sample-configs.md).

## Behavior and capabilities

* **Connection:** direct `fabric://...` URI only. There is **no DataOS depot variant for Fabric**. Nilus adds the internal `metadata+` routing prefix from `spec.type: metadata`. The URI requires an explicit port, a lakehouse name in the path, and `tenant_id` as a query parameter. Omitting any of them fails fast at parse time.
* **Bound Lakehouse wins:** The Lakehouse named in the URI path defines inventory scope. A `database_filter` targeting another workspace object is ignored. See [database\_filter evaluation guidance](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/pipeline-config.md).
* **Authentication:** Microsoft Entra service principal (client ID + client secret + tenant ID) over ODBC. Rotate the client secret before it expires. An expired secret fails authentication.
* **Supported capabilities:** inventory (workspaces, lakehouses, schemas, tables, views, columns, data types, nullability, precision/scale, descriptions), lineage, profiling, and classification. **Usage is not supported** for Microsoft Fabric.
* **Scope discipline:** set `schema_filter` / `table_filter` in production. An unfiltered sweep over a large lakehouse can take hours per stage.

## Troubleshooting

| Symptom                                                           | Likely cause                                                                            | Resolution                                                                                         |
| ----------------------------------------------------------------- | --------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| `Microsoft Fabric URI requires a port.`                           | The endpoint host has no `:<port>` segment.                                             | Add the SQL analytics endpoint port after the host, typically `:1433`.                             |
| `Microsoft Fabric URI requires 'tenant_id' as a query parameter.` | `tenant_id` missing from the URI.                                                       | Add `?tenant_id=<directory-tenant-id>` to the address.                                             |
| `Microsoft Fabric URI requires warehouse name in the path.`       | No lakehouse segment in the URI path.                                                   | Append `/<lakehouse_name>` to the host in the address.                                             |
| `Login failed` / `AADSTS7000215: Invalid client secret`           | Wrong or expired client secret, or the service principal is not enabled in Fabric.      | Rotate the client secret; confirm the tenant setting allows service principals to use Fabric APIs. |
| Inventory is empty                                                | The service principal is not a member of the workspace, or lacks read on the lakehouse. | Add the SP to the workspace and grant `SELECT` / `VIEW DEFINITION` on the lakehouse.               |
| `Data source name not found` / ODBC driver error                  | The runtime is missing the expected ODBC driver.                                        | Ensure `ODBC Driver 18 for SQL Server` is available, or pass a matching `driver` query parameter.  |
| Lineage incomplete                                                | The SP can read tables but not view definitions.                                        | Grant `VIEW DEFINITION` on the lakehouse.                                                          |

## Related Docs

* [Microsoft Fabric Warehouse](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources/microsoft-fabric.md): the Warehouse variant of the same connector.
* [Microsoft Fabric Lakehouse](https://v2.dataos.info/references/resources/nilus/destinations/microsoft-fabric-lakehouse.md): batch loading into a Fabric Lakehouse.
* [Metadata sources](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources.md): all metadata-capable sources and how to scope extraction.
* [Metadata pipelines](https://v2.dataos.info/references/resources/nilus/metadata-pipelines.md): the conceptual model.
* [Metadata pipeline config](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/pipeline-config.md): the `spec.type: metadata` contract and DAG anatomy.
* [Metadata sample configs](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/sample-configs.md): ready-to-edit YAML.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources/microsoft-fabric-lakehouse.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
