> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/v1/resources/nilus/metadata-pipelines/metadata-sources/microsoft-fabric.md). # Microsoft Fabric [Microsoft Fabric](https://learn.microsoft.com/en-us/fabric/) is supported as a metadata source through its **Warehouse** (the T-SQL data warehouse and the SQL analytics endpoint). A `spec.type: metadata` pipeline introspects a Fabric Warehouse and publishes source context — workspaces, warehouses, schemas, tables, views, columns, lineage, profiles, and classification — into the DataOS metadata catalog **without copying any table rows**. Nilus connects over ODBC using a Microsoft Entra service principal. For the field-by-field authoring contract, see [Understanding Metadata Pipeline Config](/references/v1/resources/nilus/metadata-pipelines/pipeline-config.md). ## Metadata stages `service_type: fabric` supports `metadata`, `lineage`, `profiler`, and `classification`. **Usage is not supported** for Microsoft Fabric in the current implementation. The required `mode` field decides how much of the DAG runs: `shallow` runs `metadata` + `lineage`; `deep` adds `profiler` and `classification`. | Stage | Runs in | What it lands in the catalog | | ---------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `metadata` | `shallow`, `deep` | Workspaces, warehouses, schemas, tables, views, and columns — including data types, nullability, precision/scale, and object descriptions where available. | | `lineage` | `shallow`, `deep` | Asset and column lineage parsed from view definitions and query history. | | `profiler` | `deep` | Per-column statistics (row counts, null counts, distinct counts, min/max, basic distributions). | | `classification` | `deep` | Auto-classification tags applied to columns from sampled data (PII heuristics). | | `usage` | — | Not supported for Microsoft Fabric. | Because usage is not applicable, a `deep` Fabric pipeline runs `metadata` + `lineage` + `profiler` + `classification`. `query_log_duration` and `result_limit` still tune the `lineage` stage. ## Asset hierarchy Microsoft Fabric assets map into the Datasets App as `Workspace → Warehouse → Schema → Table/View → Column`. The workspace and warehouse are derived from the SQL analytics endpoint and the warehouse name in the connection; schemas, tables, views, and columns are read from the warehouse catalog. ## Source options Metadata pipelines accept only the customer-facing `source.options` keys below. Do **not** set `source_table` — Nilus assigns a stage-specific value to each DAG node internally. | Option | Required | Used by stages | Description | | -------------------- | -------- | -------------- | ----------------------------------------------------------------------------------------- | | `service_type` | Yes | all | Must be `fabric`. | | `database_filter` | No | all | Restrict by warehouse name. Object with `includes` / `excludes` arrays of regex patterns. | | `schema_filter` | No | all | Restrict by schema name. Same shape as `database_filter`. | | `table_filter` | No | all | Restrict by table / view name. Same shape as `database_filter`. | | `query_log_duration` | No | `lineage` | Days of query history to ingest per run. Defaults to `1`. | | `result_limit` | No | `lineage` | Maximum number of query-history rows to fetch per run. Defaults to `10000000`. | | `threads` | No | `profiler` | Parallel worker count for the profiler. Raise it to cut runtime on large scopes. | `mode` (`shallow` or `deep`) is a required `spec` field, not a `source.options` key. ## Authentication and prerequisites Microsoft Fabric metadata connects through a **Microsoft Entra service principal** — there is no DataOS depot variant for Fabric. Before configuring the pipeline: {% stepper %} {% step %} ### Register an Entra application Register an Entra application (App registration) in the Fabric tenant. Note its **Application (client) ID**, **Directory (tenant) ID**, and create a **client secret**. {% endstep %} {% step %} ### Enable service principal access in Fabric In the Fabric/Power BI admin tenant settings, allow service principals to use Fabric APIs and, where required, add the app to the security group permitted to do so. {% endstep %} {% step %} ### Grant the service principal access to the workspace Add the app as a member of the Fabric workspace that contains the warehouse. The Viewer role is sufficient for read-only metadata. {% endstep %} {% step %} ### Grant read on the warehouse In the warehouse, create a database user for the service principal and grant it read: ```sql CREATE USER [] FROM EXTERNAL PROVIDER; GRANT CONNECT TO []; -- Inventory, profiling, and classification (sampling). GRANT SELECT ON DATABASE:: TO []; -- Lineage (parsing view definitions). GRANT VIEW DEFINITION ON DATABASE:: TO []; ``` {% endstep %} {% step %} ### Record the SQL analytics endpoint and port Record the SQL analytics endpoint and port. The URI must include an explicit port after the Fabric hostname, typically `1433`. Nilus rejects a Fabric URI without a port. {% endstep %} {% endstepper %} {% hint style="warning" %} Store the client secret in a DataOS secret and project it into the pipeline; never inline the literal secret in the manifest. {% endhint %} ## Sample Nilus config Fabric metadata connects only through a direct `fabric://` URI. Because `spec.type` is `metadata`, Nilus adds the internal `metadata+` routing prefix automatically. The service principal's **client ID** goes in the username position, the **client secret** in the password position (projected), the required **port** follows the Fabric hostname, the warehouse name goes in the path, and the Microsoft Entra **directory (tenant) ID** is passed as the required `tenant_id` query parameter. > The `tenant_id` here is the **Microsoft Entra (Azure AD) directory tenant ID** — a GUID (for example `11111111-2222-3333-4444-555555555555`) taken from your Fabric app registration. It is **not** your DataOS tenant or workspace name. ```yaml name: fabric-metadata version: v1alpha type: nilus tags: [nilus, metadata] description: Catalog Microsoft Fabric Warehouse metadata, schema, and lineage spec: type: metadata mode: deep compute: comet-compute schedule: crons: - "0 */6 * * *" concurrencyPolicy: Forbid use: projection: secrets: - id: engineering:fabric-secret contextAlias: fabricsecret projections: envVars: - key: FABRIC_CLIENT_ID template: "{{ secrets['fabricsecret'].client_id | base64_decode }}" - key: FABRIC_CLIENT_SECRET template: "{{ secrets['fabricsecret'].client_secret | base64_decode }}" source: address: fabric://{FABRIC_CLIENT_ID}:{FABRIC_CLIENT_SECRET}@abcd1234.datawarehouse.fabric.microsoft.com:1433/analytics_wh?tenant_id=11111111-2222-3333-4444-555555555555 options: service_type: fabric schema_filter: includes: ["^dbo$", "^gold_"] excludes: ["^staging_"] table_filter: excludes: ["^_tmp"] query_log_duration: 3 result_limit: 10000 threads: 4 ``` The `hostPort` is the workspace's **SQL analytics endpoint plus its mandatory port** (for example `.datawarehouse.fabric.microsoft.com:1433`), and the path segment is the **warehouse** name. Nilus connects with the `ODBC Driver 18 for SQL Server` by default; override it with a `driver` query parameter only if your runtime ships a different ODBC driver. With `mode: deep`, this resource produces a four-node DAG (`metadata` root → `lineage`, `profiler`, `classification`). Switch to `mode: shallow` for a 2-node `metadata` + `lineage` DAG. For more ready-to-edit examples, see [Metadata Sample Configs](/references/v1/resources/nilus/metadata-pipelines/sample-configs.md). ## Behavior and capabilities * **Connection** — direct `fabric://...` URI only; there is **no DataOS depot variant for Fabric**. Nilus adds the internal `metadata+` routing prefix from `spec.type: metadata`. The URI requires an explicit port, a warehouse name in the path, and `tenant_id` as a query parameter — omitting any of them fails fast at parse time. * **Authentication** — Microsoft Entra service principal (client ID + client secret + tenant ID) over ODBC. Rotate the client secret before it expires; an expired secret fails authentication. * **Supported capabilities** — inventory (workspaces, warehouses, schemas, tables, views, columns, data types, nullability, precision/scale, descriptions), lineage, profiling, and classification. * **Known limitations** — **usage is not supported**; query-popularity metrics are not produced for Fabric. Only Fabric **Warehouse** (T-SQL endpoint) is in scope for this connector. * **Scope discipline** — set `database_filter` / `schema_filter` / `table_filter` in production; an unfiltered sweep over a large warehouse can take hours per stage. ## Troubleshooting | Symptom | Likely cause | Resolution | | ----------------------------------------------------------------- | --------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | | `Microsoft Fabric URI requires a port.` | The Fabric hostname has no `:` segment. | Add the SQL endpoint port after the hostname, typically `:1433`. | | `Microsoft Fabric URI requires 'tenant_id' as a query parameter.` | `tenant_id` missing from the URI. | Add `?tenant_id=` to the address. | | `Microsoft Fabric URI requires warehouse name in the path.` | No warehouse segment in the URI path. | Append `/` to the host in the address. | | `Login failed` / `AADSTS7000215: Invalid client secret` | Wrong or expired client secret, or the service principal is not enabled in Fabric. | Rotate the client secret; confirm the tenant setting allows service principals to use Fabric APIs. | | Inventory is empty | The service principal is not a member of the workspace, or lacks read on the warehouse. | Add the SP to the workspace and grant `SELECT` / `VIEW DEFINITION` on the warehouse. | | `Data source name not found` / ODBC driver error | The runtime is missing the expected ODBC driver. | Ensure `ODBC Driver 18 for SQL Server` is available, or pass a matching `driver` query parameter. | | Lineage incomplete | The SP can read tables but not view definitions. | Grant `VIEW DEFINITION` on the warehouse. | ## Related Docs * [Metadata Sources](/references/v1/resources/nilus/metadata-pipelines/metadata-sources.md) — all metadata-capable sources and how to scope extraction. * [Understanding Metadata Pipelines](/references/v1/resources/nilus/metadata-pipelines.md) — the conceptual model. * [Understanding Metadata Pipeline Config](/references/v1/resources/nilus/metadata-pipelines/pipeline-config.md) — the `spec.type: metadata` contract and DAG anatomy. * [Metadata Sample Configs](/references/v1/resources/nilus/metadata-pipelines/sample-configs.md) — ready-to-edit YAML. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://v2.dataos.info/references/v1/resources/nilus/metadata-pipelines/metadata-sources/microsoft-fabric.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.