Databricks
Minimum Databricks Unity Catalog grants for each Nilus pipeline mode: batch read and write, and metadata ingestion.
Platform administrators use this reference to grant a Nilus service principal or token the minimum Databricks Unity Catalog permissions required for each Nilus pipeline mode: batch read, batch write, and metadata ingestion. All access runs through a Databricks SQL warehouse.
Read data: Databricks as source (type: batch)
Nilus reads from Databricks when a Databricks URI is configured in source.address of a pipeline config. Nilus issues SELECT queries (full scan or incremental, using a cursor column) against the target table through the SQL warehouse identified by http_path.
Minimum grants
The token or service principal needs the following Unity Catalog privileges.
-- Catalog access
GRANT USE CATALOG ON CATALOG <catalog_name> TO `<user_or_service_principal>`;
-- Schema access
GRANT USE SCHEMA ON SCHEMA <catalog_name>.<schema_name> TO `<user_or_service_principal>`;
-- Table read access
GRANT SELECT ON TABLE <catalog_name>.<schema_name>.<table_name> TO `<user_or_service_principal>`;To grant access to every table in a schema instead of one table at a time, use:
GRANT SELECT ON SCHEMA <catalog_name>.<schema_name> TO `<user_or_service_principal>`;Write data: Databricks as destination (type: batch)
Nilus writes to Databricks when a Databricks URI is configured in sink.address of a pipeline config. Nilus manages table creation and data loading through the SQL warehouse.
Minimum grants (covers all strategies)
Grants by incremental strategy
append
USE CATALOG, USE SCHEMA, MODIFY (INSERT), CREATE TABLE (if new table), CREATE VOLUME
merge
USE CATALOG, USE SCHEMA, SELECT, MODIFY (INSERT, UPDATE, DELETE), CREATE TABLE (if new table), CREATE VOLUME
replace
USE CATALOG, USE SCHEMA, MODIFY (INSERT, DELETE), CREATE TABLE (if new table), CREATE VOLUME
Metadata ingestion: Databricks source connector (type: metadata)
Nilus drives the Databricks source connector when a metadata+databricks:// URI is configured in source.address of a type: metadata pipeline. Authentication uses the same PAT or OAuth M2M modes as batch pipelines, and the same SQL warehouse "Can use" permission described above is required for metadata ingestion.
Basic metadata and profiling (tables, schemas, columns)
Additional grants by workflow type
Usage and lineage
See the system table grants below
system.query.history, system.access.table_lineage, system.access.column_lineage
View definitions
See the GRANT SELECT ON INFORMATION_SCHEMA.VIEWS statement below
INFORMATION_SCHEMA.VIEWS in the target catalog
Tags (Databricks Runtime 13.3+)
See the tag grants below
information_schema.*_tags tables in the target catalog
Usage and lineage: system table grants
View definitions
Tags (Databricks Runtime 13.3 or higher only)
Permission matrix
Use case
CAN USE WH
USE CATALOG
USE SCHEMA
SELECT
MODIFY
CREATE TABLE
CREATE VOLUME
System tables
Read (source, batch)
Yes
Yes
Yes
Yes
No
No
No
No
Write, append
Yes
Yes
Yes
No
Yes
Yes
Yes
No
Write, merge
Yes
Yes
Yes
Yes
Yes
Yes
Yes
No
Write, replace
Yes
Yes
Yes
No
Yes
Yes
Yes
No
metadata (basic)
Yes
Yes
Yes
Yes
No
No
No
No
usage/lineage
Yes
Yes
Yes
Yes
No
No
No
Yes
tags (13.3+)
Yes
Yes
Yes
Yes
No
No
No
Yes (information_schema)
Notes
SQL warehouse permission is UI-managed. The
CAN USEpermission on the SQL warehouse is granted through the Databricks workspace UI (SQL Warehouses → warehouse → Permissions). It cannot be set through SQLGRANTstatements, and it is required for every pipeline mode.MODIFYcovers all DML. In Unity Catalog,MODIFYon a Delta table grants INSERT, UPDATE, and DELETE. There is no separateINSERT,UPDATE, orDELETEprivilege.Write staging volume. Nilus loads data by uploading Parquet files to a Unity Catalog managed volume in the destination schema, then runs
COPY INTOon the table.CREATE VOLUMEon the destination schema is required in addition toCREATE TABLE.System table access. The
system.queryandsystem.accessschemas must be explicitly enabled in the Databricks account settings before grants can be issued on them. Contact the Databricks account administrator if these schemas are not visible.
Last updated
Was this helpful?