> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/engine-guide/spark-1/limits-and-performance.md).

# Limits & Performance

Size and tune Vulcan Spark workloads using observed driver, executor, shuffle, file-layout, and Iceberg commit behavior.

Spark sizing depends on query shape, input files, partitions, shuffle volume, source throughput, and available Compute resources. There is no reliable driver or executor size based only on row count or total data size.

Use representative workloads and Spark UI evidence to establish a baseline. Change one setting at a time and compare the same workload before adopting the change.

## Before you begin

Confirm that:

* The Spark connection and catalogs work.
* The model materializes correctly with representative data.
* Driver and executor metrics are available in the Spark UI.
* The test environment uses the same Spark, Iceberg, connector, and object-storage path as production.
* The workload has a measurable runtime or reliability objective.

See [Connect and configure Spark](https://v2.dataos.info/references/engine-guide/spark-1/connection-options.md) for resource and Spark property syntax.

## Understand the Spark resource roles

### Driver

The driver creates the Spark session, builds query plans, coordinates stages, tracks tasks, handles catalog operations, and runs Python model code.

Increase or tune driver resources when evidence shows:

* Driver out-of-memory failures.
* Long garbage-collection pauses.
* Large query plans or catalog metadata pressure.
* Large results returned to or constructed on the driver.
* Python logic exceeding driver CPU or memory.

Adding executor memory does not fix driver memory pressure.

### Executors

Executors scan data, run transformations, perform joins and aggregations, shuffle data, and write output files.

Tune executor resources when evidence shows:

* Executor out-of-memory failures.
* Shuffle spill to disk.
* Fetch failures or lost executors.
* Long-running or skewed tasks.
* Low parallelism or idle executors.

Executor parallelism is influenced by executor instances, cores per executor, and the number of input and shuffle partitions.

## Start with a measured baseline

Use this process for a new workload:

1. Select a representative model and time range.
2. Run it in a non-production environment.
3. Record input size, runtime, driver and executor resources, shuffle read/write, spill, task count, failed tasks, output files, and Iceberg commit duration.
4. Identify the first observed bottleneck.
5. Change one resource or Spark property.
6. Run the same model and interval again.
7. Keep the change only when the measured result improves without introducing failures or excessive cost.

Do not publish an SLO from a single successful run. Repeat the workload under expected data volume and concurrency.

## Configure Spark resources

Configure driver and executor resources in `config.yaml` or in the generated `deploy.yaml`:

{% tabs %}
{% tab title="config.yaml" %}

```yaml
deployment:
  workflow:
    resources:
      driver:
        cores: 2
        coreLimit: "3"
        memory: 4g
      executor:
        cores: 2
        coreLimit: "3"
        memory: 4g
        instances: 2
```

{% endtab %}

{% tab title="deploy.yaml" %}

```yaml
spec:
  workflow:
    resources:
      driver:
        cores: 2
        coreLimit: "3"
        memory: 4g
      executor:
        cores: 2
        coreLimit: "3"
        memory: 4g
        instances: 2
```

{% endtab %}
{% endtabs %}

Treat these values as an example shape, not a production recommendation. Use the Spark UI to determine whether the driver, executor size, executor count, or partitioning limits the workload.

## Tune shuffle behavior

### Adaptive query execution

Adaptive query execution can revise parts of a query plan using runtime statistics:

{% tabs %}
{% tab title="config.yaml" %}

```yaml
deployment:
  sparkConf:
    spark.sql.adaptive.enabled: "true"
    spark.sql.adaptive.coalescePartitions.enabled: "true"
    spark.sql.adaptive.skewJoin.enabled: "true"
```

{% endtab %}

{% tab title="deploy.yaml" %}

```yaml
spec:
  workflow:
    sparkConf:
      spark.sql.adaptive.enabled: "true"
      spark.sql.adaptive.coalescePartitions.enabled: "true"
      spark.sql.adaptive.skewJoin.enabled: "true"
```

{% endtab %}
{% endtabs %}

Validate adaptive behavior in the final executed plan. Enabling a property does not guarantee that Spark changes every query.

### Shuffle partitions

`spark.sql.shuffle.partitions` sets the initial partition count for shuffle operations:

{% tabs %}
{% tab title="config.yaml" %}

```yaml
deployment:
  sparkConf:
    spark.sql.shuffle.partitions: "200"
```

{% endtab %}

{% tab title="deploy.yaml" %}

```yaml
spec:
  workflow:
    sparkConf:
      spark.sql.shuffle.partitions: "200"
```

{% endtab %}
{% endtabs %}

Use task and stage evidence:

* Increase the count when shuffle tasks are too large, spill heavily, or exceed executor memory.
* Decrease the count when the stage creates many very small tasks and scheduler overhead dominates.
* Keep enough partitions to use available executor cores.

Do not select this value from input size alone. Join cardinality and aggregation shape can make shuffle data much larger or smaller than the source.

### Skewed joins

One or a few tasks running much longer than the rest usually indicates data skew.

Before adding executors:

1. Identify the skewed key or partition in the Spark UI.
2. Confirm join cardinality and filter placement.
3. Enable adaptive skew handling when appropriate.
4. Change the data or query design if one key still dominates the stage.

Additional executors do not remove a single oversized task.

## Tune input scans

`spark.sql.files.maxPartitionBytes` influences how file-based input is split into Spark tasks:

{% tabs %}
{% tab title="config.yaml" %}

```yaml
deployment:
  sparkConf:
    spark.sql.files.maxPartitionBytes: "134217728"
```

{% endtab %}

{% tab title="deploy.yaml" %}

```yaml
spec:
  workflow:
    sparkConf:
      spark.sql.files.maxPartitionBytes: "134217728"
```

{% endtab %}
{% endtabs %}

This example is 128 MiB. Treat it as an example value.

* Lower it when scan tasks are too large or exceed executor memory.
* Raise it when many small scan tasks create excessive scheduling overhead.
* Evaluate it together with source file size, open-file cost, executor memory, and available cores.

## Choose static or dynamic executors

### Static executor count

Use `executor.instances` when the workload needs a predictable number of executors:

{% tabs %}
{% tab title="config.yaml" %}

```yaml
deployment:
  workflow:
    resources:
      driver:
        cores: 2
        coreLimit: "3"
        memory: 4g
      executor:
        cores: 2
        coreLimit: "3"
        memory: 4g
        instances: 4
```

{% endtab %}

{% tab title="deploy.yaml" %}

```yaml
spec:
  workflow:
    resources:
      driver:
        cores: 2
        coreLimit: "3"
        memory: 4g
      executor:
        cores: 2
        coreLimit: "3"
        memory: 4g
        instances: 4
```

{% endtab %}
{% endtabs %}

### Dynamic allocation

When supported by the deployed runtime, dynamic allocation changes executor count within configured bounds:

{% tabs %}
{% tab title="config.yaml" %}

```yaml
deployment:
  sparkConf:
    spark.dynamicAllocation.enabled: "true"
    spark.dynamicAllocation.shuffleTracking.enabled: "true"
    spark.dynamicAllocation.initialExecutors: "1"
    spark.dynamicAllocation.minExecutors: "1"
    spark.dynamicAllocation.maxExecutors: "4"
```

{% endtab %}

{% tab title="deploy.yaml" %}

```yaml
spec:
  workflow:
    sparkConf:
      spark.dynamicAllocation.enabled: "true"
      spark.dynamicAllocation.shuffleTracking.enabled: "true"
      spark.dynamicAllocation.initialExecutors: "1"
      spark.dynamicAllocation.minExecutors: "1"
      spark.dynamicAllocation.maxExecutors: "4"
```

{% endtab %}
{% endtabs %}

Dynamic allocation requires compatible shuffle tracking or external shuffle-service configuration. Validate executor removal and shuffle stability under load.

Dynamic allocation scales executors only. It does not scale:

* The Spark driver.
* API replicas.
* Request concurrency.

`spec.api.autoscale` is not supported for the Spark engine.

## Control file layout

Performance degrades when Iceberg tables contain many small files or a few oversized files.

Monitor:

* Number of files written per run.
* File-size distribution.
* Planning time.
* Object-storage listing and read latency.
* Number and size of scan tasks.

Control file creation through query partitioning, shuffle configuration, and the model's partition design. Use an Iceberg maintenance procedure supported by the deployed runtime when existing tables require compaction or snapshot cleanup.

Do not over-partition tables. High-cardinality partitioning increases metadata and small-file overhead, while insufficient partitioning can force broad scans.

## Tune incremental and MERGE workloads

### Time-range and partition overwrite

For overwrite-based models:

* Filter source data to the intended interval or partition.
* Confirm the target partition specification.
* Test the effective overwrite scope with controlled data.
* Monitor output file count and Iceberg commit duration.

Overwrite scope depends on Spark and catalog partition-overwrite configuration.

### Unique-key MERGE

For `INCREMENTAL_BY_UNIQUE_KEY`:

* Confirm that the target provider supports Spark `MERGE INTO`.
* Keep the unique key selective and stable.
* Filter unnecessary source rows before the merge.
* Inspect the join and shuffle stages used by the merge.
* Stagger workflows that update the same target table.

Concurrent Iceberg writers can produce optimistic commit conflicts. `concurrencyPolicy: Forbid` prevents overlapping runs of one schedule; it does not coordinate separate resources or external writers.

## Plan backfills

Backfills can scan more history, create larger shuffles, and write more files than regular incremental runs.

Before a backfill:

1. Estimate the interval and source volume.
2. Run a smaller representative interval.
3. Confirm object-storage and source-system throughput.
4. Check temporary disk and shuffle spill requirements.
5. Schedule the run according to available Compute and source capacity.
6. Restore temporary resource changes after the backfill.

Split a backfill into bounded intervals when one run would exceed resource or operational limits.

## Understand API query cost

Queries against Spark-backed models can start Spark work and consume driver and executor resources.

The latency floor can include:

* Driver and executor startup.
* Dependency loading.
* Iceberg REST metadata requests.
* Object-storage listing and reads.
* Query planning.
* Shuffle and task execution.
* Result serialization.

Spark does not guarantee sub-second query latency. Pre-aggregate common query shapes or use a lower-latency serving engine when the consumer SLO requires it.

## Diagnose performance symptoms

| Symptom                    | Evidence to check                                    | Initial action                                         |
| -------------------------- | ---------------------------------------------------- | ------------------------------------------------------ |
| Driver out of memory       | Driver logs, heap pressure, collected result size    | Reduce driver-side data or increase driver memory      |
| Executor out of memory     | Failed executor logs, task size, scan partition size | Reduce task size or increase executor memory           |
| Heavy shuffle spill        | Stage spill metrics, shuffle partition size          | Increase shuffle partitions or change the query        |
| One task dominates a stage | Task duration distribution and input size            | Investigate skewed keys or partitions                  |
| Executors remain idle      | Active tasks, partition count, executor utilization  | Increase useful parallelism or reduce executor count   |
| Many small output files    | File count and file-size distribution                | Adjust write partitioning and use supported compaction |
| Slow scan planning         | File count, partition count, catalog latency         | Improve file layout and partition pruning              |
| Iceberg commit conflicts   | Commit errors and overlapping writer history         | Stagger and coordinate writers                         |

## Validate a tuning change

For every change, record:

* Model and interval.
* Input data and file count.
* Driver and executor configuration.
* Spark properties.
* Runtime and stage duration.
* Shuffle read/write and spill.
* Failed or retried tasks.
* Output file count and file sizes.
* Iceberg commit duration.

Compare results from the same workload. Keep the configuration as a project baseline only after repeated runs produce stable behavior.

Use [deployment recipe fragments](https://v2.dataos.info/references/engine-guide/spark-1/recipes/deployment-recipes.md) as examples, then replace their values with measurements from the target environment.

## Related

* [Connect and configure Spark](https://v2.dataos.info/references/engine-guide/spark-1/connection-options.md)
* [Spark materialization behavior](https://v2.dataos.info/references/engine-guide/spark-1/materialization-behaviour.md)
* [Spark Lakehouse limitations](https://v2.dataos.info/references/engine-guide/spark-1/spark-lakehouse-limitations.md)
* [Deployment recipe fragments](https://v2.dataos.info/references/engine-guide/spark-1/recipes/deployment-recipes.md)
* [Troubleshooting](https://v2.dataos.info/references/engine-guide/spark-1/troubleshooting.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/references/engine-guide/spark-1/limits-and-performance.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
