> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/build/v1/productize/connect-engine/databricks.md).

# Databricks

Databricks suits large-scale data engineering and analytics on Delta Lake and Unity Catalog. Use the `databricks` adapter. This page covers the connection. For the full reference, see the [Databricks engine manual](https://v2.dataos.info/references/engine-guide/databricks) in References.

## Before you start

* A Databricks workspace with SQL warehouse or cluster access.
* A personal access token or service principal credentials.
* The HTTP path to your SQL warehouse or cluster. Find it under **SQL Warehouses → \[your warehouse] → Connection details**.
* The target Unity Catalog catalog must already exist: Vulcan creates schemas and tables inside it, but not the catalog itself.
* An external Postgres `state_connection`. Vulcan state can't live in the SQL warehouse, so this is required for both local development and production.
* DataOS access: a `roles:id:data-dev`-equivalent role tag, `depot:rw:<databricks-depot-name>` (or `depot:r` for read-only), **Can Use** on Compute/Depots/Secrets, and a git-sync Secret.

## Permissions

| Permission                                       | Required for                     |
| ------------------------------------------------ | -------------------------------- |
| `USE CATALOG` on the target catalog              | Using the catalog                |
| `USE SCHEMA`, `CREATE SCHEMA` on target schemas  | Using and creating schemas       |
| `CREATE TABLE`, `CREATE VIEW` on schemas         | Creating tables and views        |
| `SELECT`, `INSERT`, `UPDATE`, `DELETE` on tables | Reading and writing model output |

## Connection options

| Option            | Description                                                     |
| ----------------- | --------------------------------------------------------------- |
| `type`            | Must be `databricks`                                            |
| `server_hostname` | Workspace hostname, for example `adb-xxxxx.azuredatabricks.net` |
| `http_path`       | HTTP path to the SQL warehouse or cluster                       |
| `access_token`    | Personal access token or service principal token                |
| `catalog`         | The Unity Catalog to use as default                             |

### Authentication methods

* **Personal access token (PAT)**: set `access_token` to a token generated under User Settings → Developer → Access tokens. The PAT's workspace must match `server_hostname`, or auth fails silently.
* **Service principal token**: set `access_token` to the service principal's token.

## Example

```yaml
gateways:
  default:
    connection:
      type: databricks
      server_hostname: <workspace-hostname>
      http_path: <sql-warehouse-or-cluster-http-path>
      access_token: "{{ env_var('DATABRICKS_ACCESS_TOKEN') }}"
      catalog: <catalog-name>
    state_connection:
      type: postgres
      host: "{{ env_var('STATESTORE_HOST') }}"
      port: "{{ env_var('STATESTORE_PORT', '5432') }}"
      user: "{{ env_var('STATESTORE_USER') }}"
      password: "{{ env_var('STATESTORE_PASSWORD') }}"
      database: "{{ env_var('STATESTORE_DATABASE') }}"
    state_schema: <project_name>
```

Never commit the token. Pull it from the environment.

## Supported model kinds

`FULL`, `VIEW`, `INCREMENTAL_BY_TIME_RANGE`, `INCREMENTAL_BY_UNIQUE_KEY`, `INCREMENTAL_BY_PARTITION`, and `SCD_TYPE_2`. For how each kind materializes, see the [Databricks engine manual](https://v2.dataos.info/references/engine-guide/databricks).

> Vulcan state (plans, snapshots, intervals) cannot live in the SQL warehouse: Databricks requires an external Postgres `state_connection` and a `state_schema`, both locally and in production. Databricks tables commonly use lowercase column names; alias them to UPPERCASE in staging and use UPPERCASE everywhere in semantic models, filters, joins, and metric expressions, or queries fail with `column not found`.

Once `vulcan info` succeeds, continue to [Assets](/build/v1/productize/assets.md).


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://v2.dataos.info/build/v1/productize/connect-engine/databricks.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.