> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/build/productize/connect-engine/databricks.md).

# Databricks

Configure Databricks as the engine for your Data Product.

Databricks suits large-scale data engineering and analytics on Delta Lake and Unity Catalog. Use the `databricks` adapter. This page covers the connection. For the full reference, see the [Databricks engine manual](https://v2.dataos.info/references/engine-guide/databricks) in References.

## Before you start

* A Databricks workspace with SQL warehouse or cluster access.
* A personal access token or service principal credentials.
* The HTTP path to your SQL warehouse or cluster. Find it under **SQL Warehouses → \[your warehouse] → Connection details**.
* The target Unity Catalog catalog must already exist: Vulcan creates schemas and tables inside it, but not the catalog itself.
* An external Postgres `state_connection`. Vulcan state can't live in the SQL warehouse, so this is required for both local development and production.
* DataOS access: a `roles:id:data-dev`-equivalent role tag, `depot:rw:<databricks-depot-name>` (or `depot:r` for read-only), **Can Use** on Compute/Depots/Secrets, and a git-sync Secret.

## Permissions

| Permission                                                        | Required for                                                                                             |
| ----------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| `USE CATALOG` on the target catalog                               | Using the catalog                                                                                        |
| `CREATE SCHEMA` on the target catalog                             | Creating schemas                                                                                         |
| `USE SCHEMA` on source and target schemas                         | Using schemas                                                                                            |
| `CREATE TABLE`, `CREATE VIEW` on target schemas                   | Creating tables and views                                                                                |
| `SELECT` on source and target schemas, `MODIFY` on target schemas | Reading and writing model output (`MODIFY` covers `INSERT`, `UPDATE`, `DELETE`, `TRUNCATE`, and `MERGE`) |

Three roles are required on the Databricks side — don't collapse them into one: a workspace **admin** (creates the SQL warehouse, catalog, and grants), the **Vulcan service principal** (runs models, manages target schemas/tables), and a **consumer role** (reads Data Product tables). Grant the service principal as a workspace admin, on the warehouse configured by `http_path`:

```sql
GRANT USE CATALOG ON CATALOG <your_catalog> TO `<principal>`;
GRANT CREATE SCHEMA ON CATALOG <your_catalog> TO `<principal>`;
GRANT CREATE TABLE ON CATALOG <your_catalog> TO `<principal>`;
GRANT CREATE VIEW ON CATALOG <your_catalog> TO `<principal>`;
```

Grant consumers read-only access the same way, with `USE CATALOG` / `USE SCHEMA` / `SELECT` instead. Verify with `SHOW SCHEMAS IN <your_catalog>;` on the same warehouse.

{% hint style="warning" %}
The target catalog must exist before `vulcan plan` — Vulcan creates schemas and tables inside it, but not the catalog itself. A missing catalog raises `NO_SUCH_CATALOG_EXCEPTION`.
{% endhint %}

## Connection options

| Option            | Description                                                     |
| ----------------- | --------------------------------------------------------------- |
| `type`            | Must be `databricks`                                            |
| `server_hostname` | Workspace hostname, for example `adb-xxxxx.azuredatabricks.net` |
| `http_path`       | HTTP path to the SQL warehouse or cluster                       |
| `access_token`    | Personal access token or service principal token                |
| `catalog`         | The Unity Catalog to use as default                             |

### Authentication methods

* **Personal access token (PAT)**: set `access_token` to a token generated under User Settings → Developer → Access tokens. The PAT's workspace must match `server_hostname`, or auth fails silently.
* **Service principal token**: set `access_token` to the service principal's token.

## Example

```yaml
gateways:
  default:
    connection:
      type: databricks
      server_hostname: <workspace-hostname>
      http_path: <sql-warehouse-or-cluster-http-path>
      access_token: "{{ env_var('DATABRICKS_ACCESS_TOKEN') }}"
      catalog: <catalog-name>
    state_connection:
      type: postgres
      host: "{{ env_var('STATESTORE_HOST') }}"
      port: "{{ env_var('STATESTORE_PORT', '5432') }}"
      user: "{{ env_var('STATESTORE_USER') }}"
      password: "{{ env_var('STATESTORE_PASSWORD') }}"
      database: "{{ env_var('STATESTORE_DATABASE') }}"
    state_schema: <project_name>
```

Never commit the token. Pull it from the environment.

## Supported model kinds

`FULL`, `VIEW`, `INCREMENTAL_BY_TIME_RANGE`, `INCREMENTAL_BY_UNIQUE_KEY`, `INCREMENTAL_BY_PARTITION`, and `SCD_TYPE_2`. Each compiles to a Delta Lake operation on Unity Catalog: `FULL` is `INSERT OVERWRITE`, `VIEW` is `CREATE OR REPLACE VIEW`, `INCREMENTAL_BY_TIME_RANGE` overwrites the time-column partition, `INCREMENTAL_BY_UNIQUE_KEY` is a `MERGE` upsert, `INCREMENTAL_BY_PARTITION` is `REPLACE WHERE`, and `SCD_TYPE_2` is a `MERGE` that expires and re-inserts changed rows. For the full mapping and a typical Bronze/staging/gold layering example, see the [Databricks engine manual](https://v2.dataos.info/references/engine-guide/databricks).

> Vulcan state (plans, snapshots, intervals) cannot live in the SQL warehouse: Databricks requires an external Postgres `state_connection` and a `state_schema`, both locally and in production. Databricks tables commonly use lowercase column names; alias them to UPPERCASE in staging and use UPPERCASE everywhere in semantic models, filters, joins, and metric expressions, or queries fail with `column not found`.

Wrap any DDL that should run only on real execution (not `vulcan plan`) in `pre_statements` / `post_statements`, guarded with `@IF(@runtime_stage = 'evaluating', ...)`. To make forward-only structural changes such as dropping a column, set `physical_properties ('delta.columnMapping.mode' = 'name')` on the model, or `ALTER` fails with `DELTA_UNSUPPORTED_DROP_COLUMN`.

## Limits and performance

Use a small SQL warehouse with 10-minute auto-stop for development, and right-size daily incrementals to partition volume; scale up temporarily for first backfills, then revert. Start `concurrent_tasks` at 2 for daily incrementals, and add API replicas (1–2 for a single team, 3–5 multi-team, 5+ high volume) before raising warehouse concurrency; keep `api.resources.limits.memory` at 3 GiB. Expect a latency floor of 2–10 seconds for warehouse cold start after auto-stop, plus 1.5–3 seconds for the semantic API.

Databricks bills SQL warehouse DBUs — enable auto-stop, set a query timeout, tag queries for attribution, and run `VACUUM` to reclaim deleted Delta files. If concurrency degrades, fix in order: remove cross-join-shaped queries, add API replicas, raise warehouse cluster count, split build and query warehouses, then increase warehouse size last.

## Troubleshooting

| Symptom                                 | Likely cause                                        | Fix                                                              |
| --------------------------------------- | --------------------------------------------------- | ---------------------------------------------------------------- |
| `NO_SUCH_CATALOG_EXCEPTION`             | Catalog missing or misnamed                         | Create the UC catalog; fix `connection.catalog`.                 |
| Auth / 403 on warehouse                 | Expired PAT or wrong workspace URL                  | Issue a new PAT; ensure `server_hostname` matches its workspace. |
| `Table ... not found` on a source table | Missing UC grant, or table not declared as external | Grant `SELECT` on the schema; add the table to `inputs.yaml`.    |
| Semantic `column not found`             | Identifier casing mismatch                          | Use UPPERCASE identifiers in semantic models.                    |
| `vulcan plan` runs DDL                  | Lifecycle DDL not guarded                           | Wrap it in `@IF(@runtime_stage = 'evaluating', ...)`.            |
| Endpoint OOM on large result            | `api.resources.limits.memory` below 1.5 GiB         | Raise it to 3 GiB.                                               |

For the full failure-mode table, health-check queries, and recovery procedures, see [Troubleshooting](https://v2.dataos.info/references/engine-guide/databricks/troubleshooting-databricks) in the Databricks engine manual.

Once `vulcan info` succeeds, continue to [Assets](https://v2.dataos.info/build/productize/assets.md).


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/build/productize/connect-engine/databricks.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
