> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/resources/nilus/destinations/microsoft-fabric-lakehouse.md).

# Microsoft Fabric Lakehouse

Write Delta tables from Nilus batch pipelines to Microsoft Fabric Lakehouses in OneLake.

Nilus can write batch datasets directly into a [Microsoft Fabric](https://learn.microsoft.com/en-us/fabric/) Lakehouse. Data lands as [Delta](https://delta.io/) tables in OneLake and is immediately visible under the Lakehouse's **Tables** view in the Fabric UI. Nilus authenticates with a Microsoft Entra service principal and writes over the OneLake ABFSS endpoint. No Spark cluster or external ingestion tool is required.

## Requirements

### Connectivity

* The Nilus runtime must reach the OneLake endpoint (`onelake.dfs.fabric.microsoft.com`) and, for lineage, the Lakehouse's SQL analytics endpoint.
* The target **Lakehouse** must already exist in a Fabric workspace. Nilus creates tables inside it but does not create the Lakehouse itself.

### Authentication and prerequisites

Fabric Lakehouse connects through a **Microsoft Entra service principal**. There is no DataOS depot variant. Before configuring the pipeline:

{% stepper %}
{% step %}
**Register an Entra application**

Register an Entra application (App registration) in the Fabric tenant. Note its **Application (client) ID**, **Directory (tenant) ID**, and create a **client secret**.
{% endstep %}

{% step %}
**Enable service principal access in Fabric**

In the Fabric/Power BI admin tenant settings, allow service principals to use Fabric APIs and, where required, add the app to the security group permitted to do so.
{% endstep %}

{% step %}
**Grant workspace access**

Add the service principal to the Fabric workspace that contains the Lakehouse with a role that permits writing data (Contributor or higher).
{% endstep %}

{% step %}
**Collect workspace and Lakehouse GUIDs**

Both the `workspace_id` and `lakehouse_id` are GUIDs taken from the Fabric portal; they are used to build the OneLake ABFSS path.
{% endstep %}
{% endstepper %}

{% hint style="info" %}
Store the client secret in a DataOS secret and project it into the pipeline; never inline the literal secret in the manifest.
{% endhint %}

### Sink address

Fabric Lakehouse uses a direct `fabric+lakehouse://` URI (there is no `dataos://` depot form):

```
fabric+lakehouse://<lakehouse_name>?tenant_id=<tenant_id>&client_id=<client_id>&client_secret=<client_secret>&workspace_id=<workspace_guid>&lakehouse_id=<lakehouse_guid>&sql_endpoint=<sql_analytics_endpoint_host>
```

| URI part                  | Required | Description                                                                                                                                                                              |
| ------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `<lakehouse_name>` (host) | Yes      | The Lakehouse display name. Used only for catalog/database naming, not for path construction.                                                                                            |
| `tenant_id`               | Yes      | Microsoft Entra **directory (tenant) ID** (GUID) of the Fabric app registration.                                                                                                         |
| `client_id`               | Yes      | Entra service principal **application (client) ID**.                                                                                                                                     |
| `client_secret`           | Yes      | Entra service principal **client secret** (project from a DataOS secret).                                                                                                                |
| `workspace_id`            | Yes      | Fabric **workspace GUID**. Used to build the OneLake ABFSS URL.                                                                                                                          |
| `lakehouse_id`            | Yes      | Fabric **Lakehouse GUID**. Used to build the OneLake ABFSS URL.                                                                                                                          |
| `sql_endpoint`            | No       | The Lakehouse **SQL analytics endpoint** host. Optional for data movement, but **required for lineage**. If it is absent and lineage is enabled, Nilus logs a warning and skips lineage. |

Nilus rejects the URI at parse time if any required parameter is missing (`Invalid Fabric Lakehouse URI: missing required parameters: …`).

## Sink options

| Option                 | Required    | Description                                                                                            |
| ---------------------- | ----------- | ------------------------------------------------------------------------------------------------------ |
| `dest_table`           | Yes         | Target table in `<schema>.<table>` form (exactly two dot-separated parts).                             |
| `incremental_strategy` | Yes         | One of `append`, `replace`, `merge`.                                                                   |
| `primary_key`          | Conditional | Required for `merge`: the key (or composite key) used to reconcile rows. Set it on the source options. |

## Sample Nilus config

The service principal's **client secret** is projected from a DataOS secret and referenced in the URI. `workspace_id` and `lakehouse_id` are the workspace and Lakehouse GUIDs from the Fabric portal.

```yaml
name: pg-to-fabric-lakehouse
version: v1alpha
type: nilus
tags: [nilus, batch]
description: Load PostgreSQL tables into a Microsoft Fabric Lakehouse
spec:
  type: batch
  compute: universe-compute
  use:
    projection:
      secrets:
        - id: engineering:fabric-lakehouse-secret
          contextAlias: fabriclhsecret
      projections:
        envVars:
          - key: FABRIC_CLIENT_ID
            template: "{{ secrets['fabriclhsecret'].client_id | base64_decode }}"
          - key: FABRIC_CLIENT_SECRET
            template: "{{ secrets['fabriclhsecret'].client_secret | base64_decode }}"
          - key: FABRIC_TENANT_ID
            template: "{{ secrets['fabriclhsecret'].tenant_id | base64_decode }}"
  source:
    address: dataos://niluspgdepot?purpose=rw
    options:
      source_table: nilus_testing.orders
      primary_key: id
      incremental_key: last_updated_ts
  sink:
    address: fabric+lakehouse://analytics_lh?tenant_id={FABRIC_TENANT_ID}&client_id={FABRIC_CLIENT_ID}&client_secret={FABRIC_CLIENT_SECRET}&workspace_id=11111111-2222-3333-4444-555555555555&lakehouse_id=66666666-7777-8888-9999-000000000000&sql_endpoint=abcd1234.datawarehouse.fabric.microsoft.com
    options:
      dest_table: sales.orders
      incremental_strategy: merge
```

Data tables land in OneLake under `/Tables` (Delta format, visible in the Fabric UI); Nilus keeps its internal load-state tables under `/Files`, where they stay hidden from the Fabric **Tables** view.

## Behavior and capabilities

* **Table format:** Delta tables written to OneLake `/Tables`, queryable directly from Fabric.
* **Authentication:** Microsoft Entra service principal (client ID + client secret + tenant ID). Nilus fetches an Azure AD token per connection, so long-running loads refresh automatically. Rotate the client secret before it expires; an expired secret fails authentication.
* **Supported pipeline mode:** `batch` only.
* **Supported incremental strategies:** `append`, `replace`, and `merge`. `merge` reconciles by the configured `primary_key`; composite primary keys are supported.
* **Table creation:** destination tables are created automatically when they do not already exist.
* **Schema contract:** the connector enforces a **strict** schema contract. When the source presents an unsupported change (a new column, or an incompatible data-type change), the pipeline fails **before** writing rather than silently evolving the table. Automatic schema evolution is not performed; schema changes require manual intervention on the destination.

{% hint style="warning" %}
**Known limitation: `TIMESTAMP_NTZ` and the Fabric Table View**

Microsoft Fabric Lakehouse Table View may fail to open Delta tables containing TIMESTAMP\_NTZ columns, particularly when the table also includes non-nullable columns such as a primary key. Fabric may display a protocol-metadata mismatch or indicate that the TIMESTAMP\_NTZ data type is unsupported.

This limitation does not affect Nilus pipeline execution or merge operations. The underlying data is written successfully and remains queryable through the Fabric SQL analytics endpoint.

Workaround: Avoid using TIMESTAMP\_NTZ columns when access through the Fabric Table View is required, or query the table through the SQL analytics endpoint.
{% endhint %}

## Limitations

The Fabric Lakehouse **destination** covers batch data movement only. The following are **not** supported by this connector:

* CDC and streaming pipelines
* Automatic schema evolution
* Automatic partition selection or partition evolution
* Microsoft Fabric **Warehouse** as a destination (this connector targets the **Lakehouse**)
* Spark-based ingestion

## Troubleshooting

| Symptom                                                                                                                                     | Likely cause                                                                                                                                                  | Resolution                                                                                              |
| ------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| `Invalid Fabric Lakehouse URI: missing required parameters: …`                                                                              | One or more of `tenant_id`, `client_id`, `client_secret`, `workspace_id`, `lakehouse_id`, or the Lakehouse name is absent from the URI.                       | Add every required parameter; verify the projected env vars resolved.                                   |
| `Table must be in <schema>.<table> format`                                                                                                  | `dest_table` is missing the schema prefix or has more than two parts.                                                                                         | Use exactly `<schema>.<table>`.                                                                         |
| `AADSTS7000215: Invalid client secret` / `Login failed`                                                                                     | Wrong or expired client secret, or the service principal is not enabled in Fabric.                                                                            | Rotate the client secret; confirm the tenant setting allows service principals to use Fabric APIs.      |
| Write fails with `403` / authorization error                                                                                                | The service principal lacks a write role on the workspace, or the workspace/Lakehouse GUIDs are wrong.                                                        | Add the SP to the workspace with Contributor (or higher); re-check `workspace_id` / `lakehouse_id`.     |
| Table with a `timestamp_ntz` column fails to open in the Fabric Table View (`invariants` protocol mismatch / `TIMESTAMP_NTZ` not supported) | Upstream `delta-rs` bug writes `timestampNtz` without the `invariants` writer feature (see the known limitation above). Data is intact and queryable via SQL. | Avoid `timestamp_ntz` columns (use a timezone-aware timestamp) or query via the SQL analytics endpoint. |
| Lineage is not produced                                                                                                                     | `sql_endpoint` is missing from the URI while lineage is enabled.                                                                                              | Add `sql_endpoint=<sql-analytics-endpoint-host>` to the address.                                        |

## Related Docs

* [Destinations](https://v2.dataos.info/references/resources/nilus/destinations.md): all destination connectors and their supported incremental strategies.
* [Microsoft Fabric Lakehouse](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources/microsoft-fabric-lakehouse.md): cataloging Fabric Lakehouse assets as a metadata source.
* [Batch pipeline config](https://v2.dataos.info/references/resources/nilus/batch/pipeline-config.md)
* [Optimize sink datasets](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimize-sink-datasets.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/references/resources/nilus/destinations/microsoft-fabric-lakehouse.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
