> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/engine-guide/databricks/limits-and-performance.md).

# Limits & Performance

Configure Databricks warehouse sizing, concurrency, cost guardrails, and performance baselines for reliable production workloads.

Set warehouse and concurrency limits before production demand grows. Use these guardrails to keep Databricks workloads reliable and predictable as demand increases.

## Operational boundaries

### Compute sizing

Match the SQL warehouse to the workload. Use a small warehouse with 10-minute auto-stop for development. Right-size daily incrementals to partition volume. Scale up temporarily for first backfills, then revert. Use a dedicated or higher-tier warehouse for the semantic API.

Always enable auto-stop on development warehouses: **Workspace UI** → **SQL Warehouses** → **Edit** → **Auto Stop**.

### Concurrency and API replicas

Keep `concurrent_tasks` below what the warehouse can admit without queue timeouts; start at 2 for daily incrementals. The API is stateless: add replicas (1 to 2 single team, 3 to 5 multi-team, 5 or more high volume) before raising warehouse concurrency. Keep `api.resources.limits.memory` at 3 GiB.

### Scheduling and latency floor

Schedule after upstream lands, set `timezone: UTC`, set `endOn` 1 to 2 years out, and use `concurrencyPolicy: Forbid`. The latency floor: SQL warehouse cold start after auto-stop adds 2 to 10 seconds (tier dependent); the semantic API adds about 1.5 to 3 seconds.

### Cost guardrails

Databricks bills SQL warehouse DBUs. Enable auto-stop, set a query timeout, tag queries for attribution, and use `VACUUM` to reclaim deleted Delta files.

```sql
SET spark.sql.session.timeoutMs = 3600000;       -- 1 hour, in pre_statements
SET spark.sql.query.tag = 'vulcan:<project>:<model>';
```

## Performance reference

Latency is warehouse-tier and data-volume shaped, so validate on a known dataset before publishing SLOs. Measure warehouse queue time (concurrency saturation), plan/run duration per model, semantic query wall time, rows touched per incremental window (the primary DBU driver), and cold-start overhead. When concurrency degrades, fix in order: remove cross-join-shaped queries, add API replicas, raise warehouse cluster count, split build and query warehouses, then increase warehouse size last.

### Next step

When workload behavior needs investigation, use [Troubleshooting](https://v2.dataos.info/references/engine-guide/databricks/troubleshooting-databricks.md).


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/references/engine-guide/databricks/limits-and-performance.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
