Skip to content
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Limits & Performance

Configure Databricks warehouse sizing, concurrency, cost guardrails, and performance baselines for reliable production workloads.

Set warehouse and concurrency limits before production demand grows. Use these guardrails to keep Databricks workloads reliable and predictable as demand increases.

Operational boundaries

Compute sizing

Match the SQL warehouse to the workload. Use a small warehouse with 10-minute auto-stop for development. Right-size daily incrementals to partition volume. Scale up temporarily for first backfills, then revert. Use a dedicated or higher-tier warehouse for the semantic API.

Always enable auto-stop on development warehouses: Workspace UI → SQL Warehouses → Edit → Auto Stop.

Concurrency and API replicas

Keep concurrent_tasks below what the warehouse can admit without queue timeouts; start at 2 for daily incrementals. The API is stateless: add replicas (1 to 2 single team, 3 to 5 multi-team, 5 or more high volume) before raising warehouse concurrency. Keep api.resources.limits.memory at 3 GiB.

Scheduling and latency floor

Schedule after upstream lands, set timezone: UTC, set endOn 1 to 2 years out, and use concurrencyPolicy: Forbid. The latency floor: SQL warehouse cold start after auto-stop adds 2 to 10 seconds (tier dependent); the semantic API adds about 1.5 to 3 seconds.

Cost guardrails

Databricks bills SQL warehouse DBUs. Enable auto-stop, set a query timeout, tag queries for attribution, and use VACUUM to reclaim deleted Delta files.

SET spark.sql.session.timeoutMs = 3600000;       -- 1 hour, in pre_statements
SET spark.sql.query.tag = 'vulcan:<project>:<model>';

Performance reference

Latency is warehouse-tier and data-volume shaped, so validate on a known dataset before publishing SLOs. Measure warehouse queue time (concurrency saturation), plan/run duration per model, semantic query wall time, rows touched per incremental window (the primary DBU driver), and cold-start overhead. When concurrency degrades, fix in order: remove cross-join-shaped queries, add API replicas, raise warehouse cluster count, split build and query warehouses, then increase warehouse size last.

Next step

When workload behavior needs investigation, use Troubleshooting.

Last updated

Was this helpful?