For the complete documentation index, see llms.txt. Markdown versions of documentation pages are available by appending .md to page URLs.
Skip to content

Databricks

Minimum Databricks Unity Catalog grants for each Nilus pipeline mode: batch read and write, and metadata ingestion.

Platform administrators use this reference to grant a Nilus service principal or token the minimum Databricks Unity Catalog permissions required for each Nilus pipeline mode: batch read, batch write, and metadata ingestion. All access runs through a Databricks SQL warehouse.

Read data: Databricks as source (type: batch)

Nilus reads from Databricks when a Databricks URI is configured in source.address of a pipeline config. Nilus issues SELECT queries (full scan or incremental, using a cursor column) against the target table through the SQL warehouse identified by http_path.

Minimum grants

The token or service principal needs the following Unity Catalog privileges.

-- Catalog access
GRANT USE CATALOG ON CATALOG <catalog_name> TO `<user_or_service_principal>`;

-- Schema access
GRANT USE SCHEMA ON SCHEMA <catalog_name>.<schema_name> TO `<user_or_service_principal>`;

-- Table read access
GRANT SELECT ON TABLE <catalog_name>.<schema_name>.<table_name> TO `<user_or_service_principal>`;

To grant access to every table in a schema instead of one table at a time, use:

GRANT SELECT ON SCHEMA <catalog_name>.<schema_name> TO `<user_or_service_principal>`;

Write data: Databricks as destination (type: batch)

Nilus writes to Databricks when a Databricks URI is configured in sink.address of a pipeline config. Nilus manages table creation and data loading through the SQL warehouse.

Minimum grants (covers all strategies)

Grants by incremental strategy

Strategy
Required privileges

append

USE CATALOG, USE SCHEMA, MODIFY (INSERT), CREATE TABLE (if new table), CREATE VOLUME

merge

USE CATALOG, USE SCHEMA, SELECT, MODIFY (INSERT, UPDATE, DELETE), CREATE TABLE (if new table), CREATE VOLUME

replace

USE CATALOG, USE SCHEMA, MODIFY (INSERT, DELETE), CREATE TABLE (if new table), CREATE VOLUME

Metadata ingestion: Databricks source connector (type: metadata)

Nilus drives the Databricks source connector when a metadata+databricks:// URI is configured in source.address of a type: metadata pipeline. Authentication uses the same PAT or OAuth M2M modes as batch pipelines, and the same SQL warehouse "Can use" permission described above is required for metadata ingestion.

Basic metadata and profiling (tables, schemas, columns)

Additional grants by workflow type

Workflow
Grants required
What it accesses

Usage and lineage

See the system table grants below

system.query.history, system.access.table_lineage, system.access.column_lineage

View definitions

See the GRANT SELECT ON INFORMATION_SCHEMA.VIEWS statement below

INFORMATION_SCHEMA.VIEWS in the target catalog

Tags (Databricks Runtime 13.3+)

See the tag grants below

information_schema.*_tags tables in the target catalog

Usage and lineage: system table grants

View definitions

Tags (Databricks Runtime 13.3 or higher only)

Tag extraction requires Databricks Runtime 13.3 or higher. On earlier runtimes, nilus metadata scanner skips tags without failing the pipeline.

Permission matrix

Use case

CAN USE WH

USE CATALOG

USE SCHEMA

SELECT

MODIFY

CREATE TABLE

CREATE VOLUME

System tables

Read (source, batch)

Yes

Yes

Yes

Yes

No

No

No

No

Write, append

Yes

Yes

Yes

No

Yes

Yes

Yes

No

Write, merge

Yes

Yes

Yes

Yes

Yes

Yes

Yes

No

Write, replace

Yes

Yes

Yes

No

Yes

Yes

Yes

No

metadata (basic)

Yes

Yes

Yes

Yes

No

No

No

No

usage/lineage

Yes

Yes

Yes

Yes

No

No

No

Yes

tags (13.3+)

Yes

Yes

Yes

Yes

No

No

No

Yes (information_schema)

Notes

  • SQL warehouse permission is UI-managed. The CAN USE permission on the SQL warehouse is granted through the Databricks workspace UI (SQL Warehouses → warehouse → Permissions). It cannot be set through SQL GRANT statements, and it is required for every pipeline mode.

  • MODIFY covers all DML. In Unity Catalog, MODIFY on a Delta table grants INSERT, UPDATE, and DELETE. There is no separate INSERT, UPDATE, or DELETE privilege.

  • Write staging volume. Nilus loads data by uploading Parquet files to a Unity Catalog managed volume in the destination schema, then runs COPY INTO on the table. CREATE VOLUME on the destination schema is required in addition to CREATE TABLE.

  • System table access. The system.query and system.access schemas must be explicitly enabled in the Databricks account settings before grants can be issued on them. Contact the Databricks account administrator if these schemas are not visible.

Last updated

Was this helpful?