> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/engine-guide/redshift/limits-and-performance.md).

# Limits and Performance

Set Redshift resource, concurrency, and performance guardrails.

Set practical resource and concurrency limits before production demand grows. These guardrails protect cluster stability as workloads, API traffic, and BI usage increase.

## Operational boundaries

### Compute sizing

| Workload            | Guidance                                             |
| ------------------- | ---------------------------------------------------- |
| Dev / small fixture | Small node or Serverless low RPU                     |
| Full backfill       | Scale up temporarily; watch disk / sort skew         |
| Daily incrementals  | Right-size WLM; avoid table scans without predicates |
| Semantic API + BI   | Dedicated queue or concurrency scaling               |

### Concurrency and API replicas

Align Vulcan `concurrent_tasks` with WLM queue slots. Raise API replicas before maxing out cluster concurrency.

```yaml
api:
  replicas: 2
  resources:
    limits:
      cpu: "4000m"
      memory: "3Gi"
```

### Scheduling and latency floor

| Recommendation                        | Why                                            |
| ------------------------------------- | ---------------------------------------------- |
| Schedule after upstream ETL completes | Avoid partial reads                            |
| `timezone: UTC` in production         | Avoid DST shifts                               |
| `endOn` ≥ 1 to 2 years out            | An expired `endOn` halts the workflow silently |
| `concurrencyPolicy: Forbid`           | Prevents overlapping incrementals              |

WLM queue wait dominates under load; monitor `STL_WLM_QUERY`. The semantic API adds roughly 1.5 to 3 seconds of overhead (validate in your stack). A first query after idle is usually faster than a cold Databricks warehouse, but it's cluster-dependent.

## Performance reference

Measure `plan`/`run` duration (WLM + table size), queue time (concurrency saturation), semantic query wall time (API + Redshift execution), and rows merged per incremental (CPU + I/O driver).

{% hint style="info" %}
**Flagged for review:** the source manual committed to formal SLO targets, which no other converted engine page states numerically; the rest describe latency floors and guardrails instead of hard percentages. The original targets are kept here rather than rewritten into prose; decide whether they belong on this page, on a tenant-level SLA page, or should be dropped until they're validated on your own cluster.

| SLO                                                                      | Target                                                                      |
| ------------------------------------------------------------------------ | --------------------------------------------------------------------------- |
| Scheduled incremental run success rate (cluster healthy, grants correct) | ≥ 99%                                                                       |
| `vulcan plan` / `apply` success (after schemas + depot validated)        | ≥ 99%                                                                       |
| Semantic API availability                                                | Same as the deployed Vulcan API track                                       |
| Query queue time under load                                              | Not yet measured; benchmark on your own cluster (WLM / concurrency scaling) |

Redshift cost and latency are shaped by cluster size, WLM queue, and table design (sort/dist keys, `ANALYZE`). Validate targets on a known dataset in your own repo (for example, a TPC-H-style QA fixture (`vulcan plan`/`apply`, an API smoke test)) before committing to external SLOs.
{% endhint %}

## Cost guardrails

Redshift bills for provisioned cluster hours or Serverless RPUs, plus storage. Set limits by environment before workloads grow. Review usage after backfills, new BI workloads, and changes to query concurrency.

### WLM and concurrency scaling

Keep development workloads in a capped WLM queue. Reserve production capacity for scheduled incrementals and API traffic.

Enable concurrency scaling only for approved production queues. Set a usage limit and alert threshold first. Review scaled-query usage against the workload it protects.

### Query timeouts

Set `statement_timeout` in `before_all` to bound runaway queries. Choose a value that permits the largest expected backfill. Reset the setting in `after_all` so it does not affect later workloads.

### Cost attribution

Tag DataOS resources with the environment, team, and workload. Use separate depots or schemas for development, staging, and production. This makes usage reviews and chargeback reports actionable.

### Table design

Use a sort key on the incremental time column. Filter incrementals on that same column to reduce scanned blocks. Run `ANALYZE` after significant loads, typically in `after_all`, so the optimizer has current statistics.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/references/engine-guide/redshift/limits-and-performance.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
