> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/v1/resources/nilus/roles-and-permissions/databricks-minimum-permissions.md).

# Databricks

Platform administrators use this reference to grant a Nilus service principal or token the minimum Databricks Unity Catalog permissions required for each Nilus pipeline mode: batch read, batch write, and Hera metadata ingestion. All access runs through a Databricks SQL warehouse.

## Read data: Databricks as source (type: batch)

Nilus reads from Databricks when a Databricks URI is configured in `source.address` of a pipeline config. Nilus issues `SELECT` queries (full scan or incremental, using a cursor column) against the target table through the SQL warehouse identified by `http_path`.

{% hint style="warning" %}
The identity running the pipeline must have "Can use" permission on the target SQL warehouse. This permission is granted through the Databricks workspace UI (SQL Warehouses → warehouse → Permissions), not through SQL `GRANT` statements. Without it, Databricks rejects query execution regardless of the Unity Catalog grants below.
{% endhint %}

### Minimum grants

The token or service principal needs the following Unity Catalog privileges.

```sql
-- Catalog access
GRANT USE CATALOG ON CATALOG <catalog_name> TO `<user_or_service_principal>`;

-- Schema access
GRANT USE SCHEMA ON SCHEMA <catalog_name>.<schema_name> TO `<user_or_service_principal>`;

-- Table read access
GRANT SELECT ON TABLE <catalog_name>.<schema_name>.<table_name> TO `<user_or_service_principal>`;
```

To grant access to every table in a schema instead of one table at a time, use:

```sql
GRANT SELECT ON SCHEMA <catalog_name>.<schema_name> TO `<user_or_service_principal>`;
```

## Write data: Databricks as destination (type: batch)

Nilus writes to Databricks when a Databricks URI is configured in `sink.address` of a pipeline config. Nilus manages table creation and data loading through the SQL warehouse.

### Minimum grants (covers all strategies)

```sql
-- Catalog access
GRANT USE CATALOG ON CATALOG <catalog_name> TO `<user_or_service_principal>`;

-- Schema access
GRANT USE SCHEMA ON SCHEMA <catalog_name>.<schema_name> TO `<user_or_service_principal>`;

-- Table creation (required if Nilus auto-creates the destination table)
GRANT CREATE TABLE ON SCHEMA <catalog_name>.<schema_name> TO `<user_or_service_principal>`;

-- Staging volume: Nilus stages Parquet files in a Unity Catalog managed volume
-- in the target schema, then loads them with COPY INTO
GRANT CREATE VOLUME ON SCHEMA <catalog_name>.<schema_name> TO `<user_or_service_principal>`;

-- DML on existing tables
GRANT MODIFY ON TABLE <catalog_name>.<schema_name>.<table_name> TO `<user_or_service_principal>`;
GRANT SELECT ON TABLE <catalog_name>.<schema_name>.<table_name> TO `<user_or_service_principal>`;
```

### Grants by incremental strategy

| Strategy  | Required privileges                                                                                                      |
| --------- | ------------------------------------------------------------------------------------------------------------------------ |
| `append`  | `USE CATALOG`, `USE SCHEMA`, `MODIFY` (INSERT), `CREATE TABLE` (if new table), `CREATE VOLUME`                           |
| `merge`   | `USE CATALOG`, `USE SCHEMA`, `SELECT`, `MODIFY` (INSERT, UPDATE, DELETE), `CREATE TABLE` (if new table), `CREATE VOLUME` |
| `replace` | `USE CATALOG`, `USE SCHEMA`, `MODIFY` (INSERT, DELETE), `CREATE TABLE` (if new table), `CREATE VOLUME`                   |

## Hera metadata ingestion: Databricks source connector (type: metadata)

Nilus drives the Hera Databricks source connector when a `metadata+databricks://` URI is configured in `source.address` of a `type: metadata` pipeline. Authentication uses the same PAT or OAuth M2M modes as batch pipelines, and the same SQL warehouse "Can use" permission described above is required for metadata ingestion.

### Basic metadata and profiling (tables, schemas, columns)

```sql
-- Catalog access
GRANT USE CATALOG ON CATALOG <catalog_name> TO `<user_or_service_principal>`;

-- Schema access
GRANT USE SCHEMA ON SCHEMA <catalog_name>.<schema_name> TO `<user_or_service_principal>`;

-- Table read access
GRANT SELECT ON TABLE <catalog_name>.<schema_name>.<table_name> TO `<user_or_service_principal>`;
```

### Additional grants by workflow type

| Workflow                        | Grants required                                                    | What it accesses                                                                      |
| ------------------------------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------------------- |
| Usage and lineage               | See the system table grants below                                  | `system.query.history`, `system.access.table_lineage`, `system.access.column_lineage` |
| View definitions                | See the `GRANT SELECT ON INFORMATION_SCHEMA.VIEWS` statement below | `INFORMATION_SCHEMA.VIEWS` in the target catalog                                      |
| Tags (Databricks Runtime 13.3+) | See the tag grants below                                           | `information_schema.*_tags` tables in the target catalog                              |

#### Usage and lineage: system table grants

```sql
-- Catalog access: required before any system.* schema or table grant can be used
GRANT USE CATALOG ON CATALOG system TO `<user_or_service_principal>`;

-- Query history for usage analytics and SQL-based lineage
GRANT SELECT ON system.query.history TO `<user_or_service_principal>`;
GRANT USE SCHEMA ON SCHEMA system.query TO `<user_or_service_principal>`;

-- System lineage tables for table-level and column-level lineage
GRANT SELECT ON system.access.table_lineage TO `<user_or_service_principal>`;
GRANT SELECT ON system.access.column_lineage TO `<user_or_service_principal>`;
GRANT USE SCHEMA ON SCHEMA system.access TO `<user_or_service_principal>`;
```

#### View definitions

```sql
GRANT SELECT ON INFORMATION_SCHEMA.VIEWS TO `<user_or_service_principal>`;
```

#### Tags (Databricks Runtime 13.3 or higher only)

```sql
GRANT SELECT ON `<catalog_name>`.information_schema.catalog_tags TO `<user_or_service_principal>`;
GRANT SELECT ON `<catalog_name>`.information_schema.schema_tags TO `<user_or_service_principal>`;
GRANT SELECT ON `<catalog_name>`.information_schema.table_tags TO `<user_or_service_principal>`;
GRANT SELECT ON `<catalog_name>`.information_schema.column_tags TO `<user_or_service_principal>`;
```

{% hint style="info" %}
Tag extraction requires Databricks Runtime 13.3 or higher. On earlier runtimes, Hera skips tags without failing the pipeline.
{% endhint %}

## Permission matrix

| Use case              | `CAN USE` WH | `USE CATALOG` | `USE SCHEMA` | `SELECT` | `MODIFY` | `CREATE TABLE` | `CREATE VOLUME` | System tables             |
| --------------------- | ------------ | ------------- | ------------ | -------- | -------- | -------------- | --------------- | ------------------------- |
| Read (source, batch)  | Yes          | Yes           | Yes          | Yes      | No       | No             | No              | No                        |
| Write, `append`       | Yes          | Yes           | Yes          | No       | Yes      | Yes            | Yes             | No                        |
| Write, `merge`        | Yes          | Yes           | Yes          | Yes      | Yes      | Yes            | Yes             | No                        |
| Write, `replace`      | Yes          | Yes           | Yes          | No       | Yes      | Yes            | Yes             | No                        |
| Hera metadata (basic) | Yes          | Yes           | Yes          | Yes      | No       | No             | No              | No                        |
| Hera usage/lineage    | Yes          | Yes           | Yes          | Yes      | No       | No             | No              | Yes                       |
| Hera tags (13.3+)     | Yes          | Yes           | Yes          | Yes      | No       | No             | No              | Yes (information\_schema) |

## Notes

* **SQL warehouse permission is UI-managed.** The `CAN USE` permission on the SQL warehouse is granted through the Databricks workspace UI (SQL Warehouses → warehouse → Permissions). It cannot be set through SQL `GRANT` statements, and it is required for every pipeline mode.
* **`MODIFY` covers all DML.** In Unity Catalog, `MODIFY` on a Delta table grants INSERT, UPDATE, and DELETE. There is no separate `INSERT`, `UPDATE`, or `DELETE` privilege.
* **Write staging volume.** Nilus loads data by uploading Parquet files to a Unity Catalog managed volume in the destination schema, then runs `COPY INTO` on the table. `CREATE VOLUME` on the destination schema is required in addition to `CREATE TABLE`.
* **System table access.** The `system.query` and `system.access` schemas must be explicitly enabled in the Databricks account settings before grants can be issued on them. Contact the Databricks account administrator if these schemas are not visible.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://v2.dataos.info/references/v1/resources/nilus/roles-and-permissions/databricks-minimum-permissions.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.