> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/engine-guide/spark-1/recipes/deployment-recipes.md).

# Deployment recipes

Example Spark configuration fragments for daily incremental workloads, Lakehouse marts, large backfills, and MERGE-heavy projects.

Use these partial `spec` fragments as starting points, not as complete deployment manifests or fixed production requirements. Merge a fragment into a valid Vulcan resource that defines `runAsUser`, `engine`, `compute`, `repo`, and `depots`. Validate every value with representative data and adjust it using the [limits and performance guide](https://v2.dataos.info/references/engine-guide/spark-1/limits-and-performance.md).

Every recipe sets `sparkConf` in both `spec.workflow.sparkConf` (tunes `migrate`/`plan`/`run`) and `spec.api.sparkConf` (tunes the API track's separate Spark session). The API track uses driver and executor resources for its single Spark application. Spark dynamic allocation may scale executors, but it does not scale the API driver, replicas, or request concurrency.

## Prerequisites

Before using a recipe:

* Attach an active DataOS Lakehouse Depot.
* Select a Spark-capable Compute resource.
* Replace example resource names, schedules, and catalog names.
* Confirm that the selected Spark and Iceberg versions support each configured property.
* Test the configuration in a non-production environment.

## Small daily incremental workload

```yaml
spec:
  engine: spark
  compute: spark-compute
  depots:
    - dataos://lakehousedepot?purpose=rw
  workflow:
    schedule:
      crons: ['0 3 * * *']
      timezone: UTC
      endOn: "<ISO-8601-end-time>"
      concurrencyPolicy: Forbid
    resources:
      driver: { coreLimit: "2000m", cores: 1, memory: "2G" }
      executor: { coreLimit: "4000m", cores: 2, memory: "4G", instances: 2 }
    sparkConf:
      spark.sql.adaptive.enabled: "true"
      spark.sql.adaptive.coalescePartitions.enabled: "true"
      spark.serializer: org.apache.spark.serializer.KryoSerializer
  api:
    replicas: 1
    resources:
      driver: { coreLimit: "2000m", cores: 1, memory: "2G" }
      executor: { coreLimit: "4000m", cores: 2, memory: "4G", instances: 1 }
    sparkConf:
      spark.sql.adaptive.enabled: "true"
      spark.sql.adaptive.coalescePartitions.enabled: "true"
      spark.serializer: org.apache.spark.serializer.KryoSerializer
```

## Medium Lakehouse mart

```yaml
spec:
  engine: spark
  compute: spark-compute
  depots:
    - dataos://lakehousedepot?purpose=rw
  workflow:
    resources:
      driver: { coreLimit: "4000m", cores: 2, memory: "6G" }
      executor: { coreLimit: "4000m", cores: 3, memory: "6G", instances: 4 }
    sparkConf:
      spark.sql.shuffle.partitions: "300"
      spark.sql.adaptive.enabled: "true"
      spark.sql.adaptive.skewJoin.enabled: "true"
      spark.sql.adaptive.coalescePartitions.enabled: "true"
      spark.sql.files.maxPartitionBytes: "134217728"
      spark.serializer: org.apache.spark.serializer.KryoSerializer
  api:
    replicas: 1
    resources:
      driver: { coreLimit: "2000m", cores: 1, memory: "2G" }
      executor: { coreLimit: "4000m", cores: 2, memory: "4G", instances: 1 }
    sparkConf:
      spark.sql.shuffle.partitions: "300"
      spark.sql.adaptive.enabled: "true"
      spark.sql.adaptive.skewJoin.enabled: "true"
      spark.sql.adaptive.coalescePartitions.enabled: "true"
      spark.sql.files.maxPartitionBytes: "134217728"
      spark.serializer: org.apache.spark.serializer.KryoSerializer
```

## Large backfill

```yaml
spec:
  engine: spark
  compute: spark-compute
  depots:
    - dataos://lakehousedepot?purpose=rw
  workflow:
    resources:
      driver: { coreLimit: "4000m", cores: 4, memory: "8G" }
      executor: { coreLimit: "6000m", cores: 4, memory: "8G", instances: 8 }
    sparkConf:
      spark.sql.shuffle.partitions: "1000"
      spark.sql.files.maxPartitionBytes: "268435456"
      spark.sql.adaptive.enabled: "true"
      spark.sql.adaptive.skewJoin.enabled: "true"
  api:
    replicas: 1
    resources:
      driver: { coreLimit: "2000m", cores: 1, memory: "2G" }
      executor: { coreLimit: "4000m", cores: 2, memory: "4G", instances: 1 }
    sparkConf:
      spark.sql.shuffle.partitions: "1000"
      spark.sql.files.maxPartitionBytes: "268435456"
      spark.sql.adaptive.enabled: "true"
      spark.sql.adaptive.skewJoin.enabled: "true"
```

Apply the temporary resource settings, run selected models for a controlled time range, and then restore the normal executor count. Schedule backfills according to the source-system and cluster capacity available in the target environment.

## MERGE-heavy CDC project

```yaml
spec:
  engine: spark
  compute: spark-compute
  depots:
    - dataos://lakehousedepot?purpose=rw
  workflow:
    schedule:
      crons: ['30 2 * * *']
      timezone: UTC
      concurrencyPolicy: Forbid
    resources:
      driver: { coreLimit: "4000m", cores: 2, memory: "6G" }
      executor: { coreLimit: "4000m", cores: 3, memory: "6G", instances: 4 }
    sparkConf:
      spark.sql.shuffle.partitions: "300"
      spark.sql.adaptive.enabled: "true"
  api:
    replicas: 1
    resources:
      driver: { coreLimit: "2000m", cores: 1, memory: "2G" }
      executor: { coreLimit: "4000m", cores: 2, memory: "4G", instances: 1 }
    sparkConf:
      spark.sql.shuffle.partitions: "300"
      spark.sql.adaptive.enabled: "true"
```

The following model header fragment explicitly declares the Iceberg format:

```sql
MODEL (
  name lakehousedepot.crm.customer_snapshot,
  kind INCREMENTAL_BY_UNIQUE_KEY (
    unique_key c_custkey,
    batch_size 1,
    when_matched (
      WHEN MATCHED AND source.last_op = 'D' THEN DELETE
      WHEN MATCHED AND source.last_op IN ('I', 'U') THEN UPDATE SET
        c_name       = source.c_name,
        c_mktsegment = source.c_mktsegment,
        last_op      = source.last_op
    )
  ),
  table_format iceberg,
  partitioned_by (source_ds),
  grain (c_custkey)
);
```

`when_matched` branches use `source.` and `target.` aliases. New keys fall through to implicit insert, so delete tombstones for missing keys insert rows unless filtered upstream.

## Verify a recipe

Run the workload with representative data in a non-production environment. In the Spark UI, check:

* Driver or executor out-of-memory failures.
* Shuffle spill and skewed or slow stages.
* Executor utilization and task parallelism.
* Output file count and file sizes.
* Iceberg commit duration and commit conflicts.

Adjust one setting at a time, rerun the same workload, and record the configuration that meets the required runtime and reliability targets.

## Next steps

Return to the [Spark guide](https://v2.dataos.info/references/engine-guide/spark-1.md) for operations and troubleshooting guidance.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/references/engine-guide/spark-1/recipes/deployment-recipes.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
