> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/engine-guide/spark-1.md).

# Spark

Apache Spark is a unified analytics engine for large-scale data processing. Vulcan integrates with Spark to manage transformations as Apache Iceberg tables over object storage, with version control and safe deployments. This page is the full manual: platform dependencies, connection and storage setup, local development, how each model kind materialises to Iceberg, the semantic layer, data quality, deployment recipes, driver and executor tuning, and the failure modes you hit in real projects.

Use the [Connect engine → Spark](broken://pages/04ff1cff579c5d0fa2fceda647adf1095da653f7) Build page for the quick connection steps.

{% hint style="info" %}
Engine adapter type: `spark`. Model dialect: `spark2`. Support level: GA for S3 and ABFSS lakehouse deployments. Table format: Apache Iceberg through a REST catalog. Object storage: Amazon S3 and Azure ABFSS. Tested Vulcan image: `tmdcio/vulcan-spark:0.228.1.26`. Serving images: `tmdcio/vulcan-graphql:0.228.1.26`, `tmdcio/mysql-wire:0.228.1.26`. Tested DataOS release: Draco 1.38 series.
{% endhint %}

## When to use Spark

Choose Spark when you run a lakehouse: large batch transforms, shuffle-heavy joins, multi-year backfills, and Iceberg-backed marts over S3 or ABFSS. Spark is a batch/lakehouse engine, not a sub-second serving engine; if your downstream SLO is sub-second on arbitrary semantic queries, pre-aggregate into a serving store or use a lower-latency engine for that surface. 

Spark performance is workload-shaped — a 10 GB dimension update, a 100 GB shuffle-heavy join, and a 1 TB backfill have different ceilings. Use [Performance reference](#performance-reference) to size from the data shape, not from row count alone.

### Supported model kinds

`FULL`, `VIEW`, `SEED` / `EMBEDDED`, `INCREMENTAL_BY_TIME_RANGE`, `INCREMENTAL_BY_PARTITION`, `INCREMENTAL_BY_UNIQUE_KEY` (requires Iceberg `physical_properties`), and `SCD_TYPE_2`.

{% hint style="warning" %}
VDE is not supported on Spark. Setting `vde: true` is rejected by validation when the gateway type is `spark`. Spark gateways must run in simple mode (`vde: false`, the default). Vulcan state cannot live in the Iceberg catalog either; use a transactional database (DuckDB locally, external Postgres in production) for `state_connection`.
{% endhint %}

### Not supported on Spark via Vulcan

| Feature                                          | Why not supported                                                          | Alternative                                                                           |
| ------------------------------------------------ | -------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
| Spark compute lifecycle (create / resize / stop) | Vulcan has no compute pool management API                                  | Provision compute pools via DataOS admin or Terraform                                 |
| Materialized views                               | Not the default Spark/Iceberg path; version-dependent                      | Use `kind FULL` incremental models as materialized equivalents                        |
| Sub-second API serving for arbitrary queries     | Spark/Iceberg is a batch/lakehouse engine, not optimised for point lookups | Pre-aggregate into a serving store or use a lower-latency engine for that surface     |
| Concurrent writes to the same Iceberg table      | Optimistic commit; concurrent MERGE writers can conflict                   | Use `concurrencyPolicy: Forbid`, stagger heavy writers, configure commit retry        |
| DLT / streaming tables                           | Separate concern; not in Vulcan's model kinds                              | Build streaming pipelines natively; land results as Iceberg tables for Vulcan to read |

### Service-level expectations

Reference targets, assuming the recommended configuration below and workload validation in your own tenant:

| Signal                                                           | Target                                                                     |
| ---------------------------------------------------------------- | -------------------------------------------------------------------------- |
| Daily incremental run success rate (stable upstream and compute) | ≥ 99%                                                                      |
| Plan/run driver startup success                                  | ≥ 99% after dependency and catalog validation                              |
| API track availability                                           | Same as DataOS service-level availability for the deployed Vulcan resource |
| Iceberg MERGE conflict recovery                                  | Commit retry configured before concurrent MERGE workloads go live          |
| Executor OOM rate                                                | 0 on representative production windows after sizing validation             |

## Platform dependencies

Spark on Vulcan needs platform resources that an SRE team provisions before you write any model code.

| Requirement                        | Notes                                                                                                       |
| ---------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| Spark-capable compute pool         | Verify with `dataos-ctl resource -t compute get -a`; set `spec.compute` to this pool name.                  |
| Tenant-level `vulcan-spark` stack  | Provides the runtime image, serving sidecars, dependency loading, and catalog templating.                   |
| DataOS lakehouse resource          | Backed by S3 or ABFSS object storage.                                                                       |
| Lakehouse depot                    | Attached as `dataos://<depot>?purpose=rw` in the Vulcan resource.                                           |
| External Postgres for Vulcan state | Provisioned and wired automatically by the platform in production; nothing to add to project `config.yaml`. |
| Object-storage credentials         | Stored on the lakehouse/depot, never hard-coded in project code.                                            |
| Git-sync secret                    | Used by the Vulcan resource to pull model code.                                                             |

Three roles are required: an admin role (platform SRE provisions the compute pool, installs the stack, creates the lakehouse and depot), the Vulcan service role (the `runAsUser`; runs Spark jobs and writes Iceberg tables through the lakehouse depot), and a consumer role (read-only via API endpoints).

Minimum object-storage grants:

```
S3:         s3:GetObject, s3:PutObject, s3:DeleteObject, s3:ListBucket on the warehouse bucket
ABFSS:  Storage Blob Data Contributor on the container; Storage Account Key or SPN credentials on the depot
```

### DataOS permissions

| Permission                        | What it unlocks                                               | Who to request from                 |
| --------------------------------- | ------------------------------------------------------------- | ----------------------------------- |
| `roles:id:data-dev` or equivalent | Create and apply Vulcan resources (workflow, API)             | DataOS operator / admin             |
| Access to the target workspace    | Apply secrets, depots, and Vulcan resources in that workspace | DataOS operator                     |
| `depot:rw:<lakehouse-depot-name>` | Read/write access to the lakehouse depot                      | DataOS operator                     |
| `depot:r:<lakehouse-depot-name>`  | Read-only access (consumer)                                   | DataOS operator                     |
| Git repository access             | Vulcan pulls model code via git-sync                          | Your VCS admin (GitHub / Bitbucket) |

Run `dataos-ctl get depot` before starting; if the lakehouse depot appears in the output, your read access is confirmed.

### Python version

Local development requires Python 3.10 (`python --version` must report `3.10.x`). The Spark runtime inside the Vulcan image manages its own Python version.

## Connection options

Set these under `gateways.<name>.connection`. Spark is single-catalog: all models must be defined under one catalog.

| Option       | Description                                                                       | Type   | Required |
| ------------ | --------------------------------------------------------------------------------- | ------ | -------- |
| `type`       | Engine type name. Must be `spark`.                                                | string | Yes      |
| `config_dir` | Value to set for `SPARK_CONFIG_DIR`.                                              | string | No       |
| `catalog`    | The catalog to use when issuing commands.                                         | string | No       |
| `config`     | Key/value pairs for the Spark configuration, including the Iceberg catalog setup. | dict   | No       |

If `catalog` is not set, the default depends on the Spark version: Spark 3.4 and above determine it at runtime; earlier versions default to `spark_catalog`. Authentication is configuration-based: credentials come through the `config` parameter or the catalog (S3, HDFS).

{% hint style="info" %}
In production, do not hand-author the catalog block. Attach the lakehouse depot in `spec.depots[]` and let the `vulcan-spark` stack generate the catalog config automatically. The local examples below are for Docker-only development.
{% endhint %}

## Local development

The local stack mirrors production: a Spark standalone cluster, MinIO (S3-compatible storage), an Iceberg REST catalog, and Vulcan acting as the Spark driver. Running the driver inside Linux avoids Windows Hadoop and `winutils.exe` issues.

### Install Vulcan

Vulcan is distributed as a Python wheel and installed with `pip`. Get the latest `.whl` for the Spark engine from your DataOS distribution channel.

```bash
pip install "/path/to/vulcan-<version>-py3-none-any.whl[spark,postgres]"
vulcan --version   # verify after install
```

For the Docker flows below, place the wheel at the project root; the `vulcan-cli` container installs from it automatically.

### Choose a local flow

Two flows are supported. Use **Local Python** to run the Vulcan driver in a virtual environment on your machine. Use **Python in Docker** to run the Python runtime and Vulcan CLI inside the `vulcan-cli` container, with no host Python or Java required.

|                   | Local Python                             | Python in Docker                             |
| ----------------- | ---------------------------------------- | -------------------------------------------- |
| Compose file      | `docker/docker-compose.local-python.yml` | `docker/docker-compose.python-in-docker.yml` |
| Vulcan runs on    | Your machine (`venv` + Java 17)          | The `vulcan-cli` container                   |
| Host requirements | Docker, Java 17, Python 3.10             | Docker only                                  |
| Hosts-file entry  | Required once                            | Not required                                 |
| Config file       | `config.yaml`                            | `config.yaml`                                |

Both flows start the same core services: Spark standalone cluster, MinIO, and Iceberg REST catalog. Local Vulcan state uses DuckDB, a local file, so there is nothing extra to run for it.

### Local service map

| Service                  | Image                                | Host ports                 | Purpose                                                   |
| ------------------------ | ------------------------------------ | -------------------------- | --------------------------------------------------------- |
| `spark-master`           | `tmdcio/vulcan-spark-base:<version>` | 7077, 8080                 | Spark standalone master.                                  |
| `spark-worker` (1 or 2)  | `tmdcio/vulcan-spark-base:<version>` | 8081                       | Local executor worker(s).                                 |
| `minio-warehouse`        | `minio/minio:latest`                 | 9000 (API), 9001 (console) | S3-compatible object storage.                             |
| `iceberg-rest-warehouse` | `tabulario/iceberg-rest:latest`      | 8181                       | Local Iceberg REST catalog.                               |
| `iceberg-rest-sample`    | `tabulario/iceberg-rest:latest`      | 8182                       | Second local catalog for sample data (Local Python flow). |
| `vulcan-cli`             | Python runtime plus Vulcan wheel     | n/a                        | Python-in-Docker flow only; acts as the Spark driver.     |

### Docker Compose

Start Spark, MinIO, Iceberg REST, and the Vulcan CLI container together:

```bash
docker compose -f docker/docker-compose.python-in-docker.yml up -d
docker compose -f docker/docker-compose.python-in-docker.yml logs -f vulcan-cli
```

```yaml
# Run Vulcan with Python in Docker: Spark + MinIO + Iceberg REST + vulcan-cli.
# Place a vulcan-*.whl in the project root before first boot.

services:
  spark-master:
    image: tmdcio/vulcan-spark-base:0.228.1.26
    command: ["/bin/bash", "-lc", "/opt/spark/sbin/start-master.sh --host 0.0.0.0 --port 7077 --webui-port 8080 && tail -f /opt/spark/logs/*"]
    ports:
      - "7077:7077"
      - "8080:8080"
    networks: [vulcan-spark-local-net]

  spark-worker:
    image: tmdcio/vulcan-spark-base:0.228.1.26
    command: ["/bin/bash", "-lc", "/opt/spark/sbin/start-worker.sh spark://spark-master:7077 --webui-port 8081 && tail -f /opt/spark/logs/*"]
    depends_on: [spark-master]
    ports:
      - "8081:8081"
    networks: [vulcan-spark-local-net]

  minio-warehouse:
    image: minio/minio:latest
    environment:
      MINIO_ROOT_USER: admin
      MINIO_ROOT_PASSWORD: password
      MINIO_DOMAIN: minio-warehouse
    ports:
      - "9000:9000"
      - "9001:9001"
    volumes:
      - minio_data:/data
    command: server /data --console-address ":9001"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 5s
      timeout: 5s
      retries: 10
    networks:
      vulcan-spark-local-net:
        aliases: [minio-warehouse]

  minio-warehouse-init:
    image: minio/mc:latest
    depends_on:
      minio-warehouse:
        condition: service_healthy
    entrypoint: >
      /bin/sh -c "
        mc alias set minio http://minio-warehouse:9000 admin password;
        mc mb --ignore-existing minio/warehouse;
        mc mb --ignore-existing minio/sample;
        mc anonymous set download minio/warehouse;
        mc anonymous set download minio/sample;
        exit 0;
      "
    networks: [vulcan-spark-local-net]

  iceberg-rest-warehouse:
    image: tabulario/iceberg-rest:latest
    ports:
      - "8181:8181"
    environment:
      AWS_ACCESS_KEY_ID: admin
      AWS_SECRET_ACCESS_KEY: password
      AWS_REGION: us-east-1
      CATALOG_WAREHOUSE: s3://warehouse/
      CATALOG_IO__IMPL: org.apache.iceberg.aws.s3.S3FileIO
      CATALOG_S3_ENDPOINT: http://minio-warehouse:9000
      CATALOG_S3_PATH__STYLE__ACCESS: "true"
    depends_on:
      minio-warehouse:
        condition: service_healthy
    networks:
      vulcan-spark-local-net:
        aliases: [iceberg-rest-warehouse]

  vulcan-cli:
    image: python:3.10-bookworm
    depends_on: [spark-master, spark-worker, iceberg-rest-warehouse]
    environment:
      LOCAL_DEV_HOST: vulcan-cli
      VULCAN_LOCAL_JARS_DIR: /workspace/dependencies/java
      VULCAN_SPARK_EXECUTOR_EXTRA_JARS_DIR: /workspace/dependencies/java
    volumes:
      - ..:/workspace
      - vulcan_ivy_cache:/root/.ivy2
      - vulcan_m2_cache:/root/.m2
    working_dir: /workspace
    networks: [vulcan-spark-local-net]
    command:
      - sh
      - -lc
      - |
        set -eu
        if ! command -v java >/dev/null 2>&1; then
          apt-get update && apt-get install -y openjdk-17-jre-headless curl
          rm -rf /var/lib/apt/lists/*
        fi
        python -m pip install -U pip setuptools wheel
        if ! command -v vulcan >/dev/null 2>&1; then
          WHEEL=$$(ls /workspace/vulcan-*.whl 2>/dev/null | head -1)
          [ -z "$$WHEEL" ] && echo "ERROR: No vulcan-*.whl found in /workspace." && exit 1
          python -m pip install --no-cache-dir "vulcan[spark,postgres] @ file://$$WHEEL"
        fi
        tail -f /dev/null

networks:
  vulcan-spark-local-net:
    driver: bridge

volumes:
  minio_data:
  vulcan_ivy_cache:
  vulcan_m2_cache:
```

Set an alias to run Vulcan through the compose service:

```bash
alias vulcan='docker compose -f docker/docker-compose.python-in-docker.yml exec -T vulcan-cli vulcan'
```

If you use the **Local Python** flow instead, the driver runs on your machine against the same Spark/MinIO/Iceberg-REST services, so add the local service names to your hosts file once:

```bash
sudo tee -a /etc/hosts >/dev/null <<'EOF'
127.0.0.1 spark-master minio-warehouse iceberg-rest-warehouse iceberg-rest-sample
EOF
```

then install Vulcan into a venv and run it directly:

```bash
python3.10 -m venv .venv
source .venv/bin/activate
pip install -U pip setuptools wheel
pip install "/path/to/vulcan-<version>-py3-none-any.whl[spark,postgres]"
```

Host-driver mode runs the Spark driver on your machine while executors run in Docker. If Spark logs show bind failures or executors cannot reach the driver, set `LOCAL_DEV_HOST` to your machine's LAN IPv4 address before running Vulcan (`ipconfig getifaddr en0` on macOS, `hostname -I` on Linux, `ipconfig` on Windows).

### Configure config.yaml

Run the project initialisation flow from the [LDK setup guide](https://v2.dataos.info/build/readme/ldk-setup) before your first `vulcan info`, `vulcan plan`, or `vulcan run` — it creates the project structure and `config.yaml`. Then update the generated file with the local Spark connection below, and adjust project identity fields (`name`, `display_name`, `tenant`, `domain`, `model_defaults.start`).

```yaml
gateways:
  default:
    connection:
      type: spark
      config:
        spark.master: spark://spark-master:7077
        spark.app.name: "<spark-app-name>"
        spark.driver.host: "{{ env_var('LOCAL_DEV_HOST', '127.0.0.1') }}"
        spark.driver.bindAddress: 0.0.0.0
        spark.driver.extraJavaOptions: "-Daws.region=us-east-1 -Daws.defaultRegion=us-east-1 -Djava.io.tmpdir=/tmp/iceberg"
        spark.executor.extraJavaOptions: "-Daws.region=us-east-1 -Daws.defaultRegion=us-east-1 -Djava.io.tmpdir=/tmp/iceberg"
        spark.driver.extraClassPath: "{{ env_var('VULCAN_LOCAL_JARS_DIR', 'dependencies/java') }}/*:{{ env_var('VULCAN_LOCAL_JARS_DIR', 'dependencies/java') }}/nested/*"
        spark.executor.extraClassPath: "{{ env_var('VULCAN_SPARK_EXECUTOR_EXTRA_JARS_DIR', '/etc/dataos/work/dependencies/java') }}/*:{{ env_var('VULCAN_SPARK_EXECUTOR_EXTRA_JARS_DIR', '/etc/dataos/work/dependencies/java') }}/nested/*"
        spark.jars.packages: org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.10.1,org.apache.iceberg:iceberg-aws-bundle:1.10.1,org.apache.hadoop:hadoop-aws:3.3.4,com.amazonaws:aws-java-sdk-bundle:1.12.262
        spark.sql.catalog.warehouse: org.apache.iceberg.spark.SparkCatalog
        spark.sql.catalog.warehouse.type: rest
        spark.sql.catalog.warehouse.uri: http://iceberg-rest-warehouse:8181
        spark.sql.catalog.warehouse.warehouse: s3://warehouse/
        spark.sql.catalog.warehouse.io-impl: org.apache.iceberg.io.ResolvingFileIO
        spark.sql.catalog.warehouse.s3.endpoint: http://minio-warehouse:9000
        spark.sql.catalog.warehouse.s3.path-style-access: "true"
        spark.sql.catalog.warehouse.s3.access-key-id: admin
        spark.sql.catalog.warehouse.s3.secret-access-key: password
        spark.sql.catalog.warehouse.client.region: us-east-1
        spark.sql.extensions: org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
        spark.sql.legacy.charVarcharAsString: "true"
        spark.dynamicAllocation.enabled: "true"
        spark.dynamicAllocation.shuffleTracking.enabled: "true"
        spark.dynamicAllocation.initialExecutors: "1"
        spark.dynamicAllocation.minExecutors: "1"
        spark.dynamicAllocation.maxExecutors: "2"
    state_connection:
      type: duckdb                  # local dev only
      database: ./.state.db
    state_schema: spark

default_gateway: default
model_defaults:
  dialect: spark2
  start: '<project-start-date>'
```

{% hint style="info" %}
`state_connection` is a local-only key. DuckDB is a plain file, nothing to spin up beyond the path above — drop this block entirely when you edit `config.yaml` for deployment; in DataOS the platform provisions and manages the production state store automatically. The same relative `./.state.db` path resolves correctly under both local flows, since Python-in-Docker mounts the whole project at `/workspace`.
{% endhint %}

{% hint style="warning" %}
Do not remove the driver network and Java option settings. `spark.driver.host`, `spark.driver.bindAddress`, `spark.driver.extraJavaOptions`, and `spark.executor.extraJavaOptions` let Docker executors call back to the host driver and keep AWS/Iceberg temp-file behavior stable in the local stack.
{% endhint %}

Keep local JARs in `dependencies/java/` — `spark.driver.extraClassPath` loads them for the driver and `spark.executor.extraClassPath` makes them available to Spark workers. If you test REST/GraphQL query responses locally, keep an `object_store` block in `config.yaml` so query results are written to the local MinIO bucket and returned through presigned URLs. As before, the catalog block itself should not be hand-authored for production — attach the lakehouse depot in `spec.depots[]` and let the `vulcan-spark` stack generate it.

### Hello-world starter

The local MinIO warehouse starts empty — there is no `warehouse.raw.orders` until something lands it. A `kind SEED` model loads a CSV straight into an Iceberg table, so it doubles as the starting point when you have a CSV and no upstream source yet: seed the raw table, then build the gold model in the same project on top of it.

```csv
# seeds/orders.csv
order_id,customer_id,total_price,order_date
O001,C001,120.50,2025-01-05
O002,C002,89.99,2025-01-06
O003,C001,45.00,2025-01-07
```

```sql
-- models/seeds/raw_orders.sql
MODEL (
  name warehouse.raw.orders,
  kind SEED (
    path '../../seeds/orders.csv'
  ),
  columns (
    order_id STRING,
    customer_id STRING,
    total_price DOUBLE,
    order_date DATE
  ),
  grain order_id
);
```

```sql
-- models/orders.sql
MODEL (
  name warehouse.gold.orders,
  kind FULL,
  physical_properties (format = 'iceberg')
);
SELECT order_id, customer_id, total_price, order_date
FROM   warehouse.raw.orders
```

The [Semantic models](#semantic-models) and [Business metrics](#business-metrics) sections below use the same `warehouse.gold.orders` table as their example, so this starter project is enough to try both.

### Validate your connection

Run these only after the Docker stack is running, `config.yaml` exists in the project root, and the hello-world models are in place.

```bash
vulcan migrate             # initialises Vulcan state (DuckDB locally, Postgres in production)
vulcan plan --auto-apply   # dry-run + apply against the local Iceberg catalog
vulcan run                 # materialises warehouse.gold.orders
```

If `vulcan plan` succeeds, your local setup is complete. Call the REST endpoint to confirm end to end. Common failures at this step are covered under [Failure modes and troubleshooting](#failure-modes-and-troubleshooting).

## Dependencies

Place JVM dependencies under `dependencies/java/` and Python wheels under `dependencies/python/`; the stack loads them for driver and executors automatically. Nested folders under `dependencies/java/` are not picked up. If you have multiple JAR folders, set `spark.driver.extraClasspath` (and `spark.executor.extraClasspath` for executor-side UDFs). Register a Java UDF by fully qualified class name to call it in SQL:

```java
context.spark.udf.registerJavaFunction(
    "my_udf_name",
    "com.yourorg.udf.YourUdfClass",
    types.StringType(),
)
```

When using ABFSS, add the Azure Iceberg bundle JAR under `dependencies/java/`, or runtime fails with `NoClassDefFoundError: DataLakeStorageException`.

For SEED models on DataOS, download vendor Derby before deployment and commit the JAR so git-sync delivers it to the Spark pod:

1. Download [derby-10.14.2.0.jar](https://repo1.maven.org/maven2/org/apache/derby/derby/10.14.2.0/derby-10.14.2.0.jar) from Maven Central.
2. Save it into your Vulcan project at `dependencies/java/derby-10.14.2.0.jar` (create the `dependencies/java/` folder if it doesn't exist).
3. Commit that file so git-sync ships it to the Spark pod on deploy.

## Production deployment

Apply resources in this order — each step depends on the previous:

```
Lakehouse resource → Lakehouse depot → Git secret → config.yaml → Vulcan resource
```

| Manifest                                  | Purpose                                      | Key fields                                                                                                                                          |
| ----------------------------------------- | -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| Lakehouse resource                        | Object-storage-backed Iceberg storage        | S3 or ABFSS configuration                                                                                                                           |
| Lakehouse depot                           | Exposes the lakehouse to Spark and Vulcan    | `name`, `spec.type`, storage credentials, `purpose: rw`                                                                                             |
| `secret-git-sync.yaml`                    | Repo credentials for git-sync                | `GITSYNC_USERNAME`, `GITSYNC_PASSWORD`                                                                                                              |
| `config.yaml`                             | Project config, trimmed to production fields | `engine: spark`, depot gateway, `model_defaults`                                                                                                    |
| Vulcan resource (`workspace-deploy.yaml`) | Spark driver + executors + API + schedule    | `spec.engine: spark`, `spec.compute`, `spec.repo`, `spec.depots`, `spec.workflow.resource`, `spec.api`, `spec.workflow.sparkConf`, `spec.sparkConf` |

The `vulcan-spark` stack generates the Iceberg REST catalog config from the lakehouse depot, so the depot must exist before `vulcan apply`. Set `state_connection` to `type: postgres` for production, or omit it entirely if the platform provisions it automatically for your tenant. Depot purposes are `rw` (Vulcan workflow), `scan` (metadata scanner), and `query` (consumer read-only).

Generate the deploy manifest from the project root:

```bash
vulcan create_deploy_yaml --output workspace-deploy.yaml
```

Before editing `config.yaml` for deployment, keep a copy of the working local version for reference:

```bash
cp config.yaml config.local.yaml   # keep out of the deploy commit; add to .gitignore
```

Then edit `config.yaml` in place. Drop the local-only keys — `state_connection`, `object_store`, and the `spark.master` / `spark.driver.*` / `spark.executor.*` / `spark.sql.catalog.*` block — and switch to a DataOS depot gateway:

```yaml
gateways:
  default:
    connection:
      type: depot
      address: dataos://<lakehouse-depot-name>
```

{% hint style="warning" %}
Rename the catalog prefix on every model. A local catalog name like `warehouse` (declared under `spark.sql.catalog.warehouse`) does not exist in DataOS. Once the gateway switches to `type: depot`, the catalog prefix becomes the depot name instead, so every fully qualified model, seed, and semantic-layer reference built during local development must be renamed from `warehouse.<schema>.<table>` to `<lakehouse-depot-name>.<schema>.<table>` before deploying.
{% endhint %}

Fill the generated placeholders in `workspace-deploy.yaml`:

| Field                    | What to set                                                           |
| ------------------------ | --------------------------------------------------------------------- |
| `type`                   | `vulcan`                                                              |
| `name`                   | Lowercase DataOS resource name, usually matching `config.yaml` `name` |
| `spec.runAsUser`         | DataOS user/service account to run the workflow                       |
| `spec.compute`           | Spark-capable DataOS compute pool                                     |
| `spec.engine`            | `spark`                                                               |
| `spec.repo.url`          | Git repository URL containing the project                             |
| `spec.repo.syncFlags`    | Branch/tag ref, e.g. `--ref=<branch>`                                 |
| `spec.repo.baseDir`      | Path from repo root to the Vulcan project                             |
| `spec.repo.secretId`     | Git-sync secret, if the repo is private                               |
| `spec.depots`            | `dataos://<lakehouse-depot-name>?purpose=rw`                          |
| `spec.workflow.schedule` | Cron, `endOn`, timezone, and `concurrencyPolicy`                      |
| `spec.workflow.resource` | Spark driver/executor CPU and memory sizing                           |
| `spec.api`               | API replicas and request/limit resources                              |

{% hint style="warning" %}
Set Spark tuning in **both** `spec.workflow.sparkConf` and `spec.sparkConf`. The workflow (`migrate`/`plan`/`run`) and the API track run as separate Spark sessions, and neither inherits the other's config — setting only `spec.workflow.sparkConf` tunes `plan`/`run` but leaves the API session on defaults, which can serve queries without the same classpath or catalog implementation.
{% endhint %}

```yaml
spec:
  workflow:
    sparkConf:
      spark.sql.catalogImplementation: "in-memory"
      spark.executor.extraClassPath: "/etc/dataos/work/<repo-base-dir>/dependencies/java/*"
  sparkConf:
    spark.sql.catalogImplementation: "in-memory"
    spark.executor.extraClassPath: "/etc/dataos/work/<repo-base-dir>/dependencies/java/*"
```

Apply once `config.yaml`, dependencies, and `workspace-deploy.yaml` are committed and pushed:

```bash
dataos-ctl resource apply -f workspace-deploy.yaml
dataos-ctl resource -t vulcan -n <resource_name> get -d          # detailed status and logs
dataos-ctl resource -t vulcan -n <resource_name> get runtime     # container runtime behavior
```

The `vulcan-spark` stack can only generate catalog config from a depot that already exists; the Vulcan resource can only sync code if the Git secret is available. A depot name mismatch in `spec.depots[]` passes `apply` but fails at every workflow run — this is why the apply order above matters.

## Materialisation behaviour per model kind

SQL models compile to Spark SQL and write Iceberg tables through the lakehouse REST catalog. The Vulcan workflow acts as the Spark driver; executors perform scans, joins, shuffles, and writes.

| Model kind                  | Spark / Iceberg operation                          | Notes                                                                           |
| --------------------------- | -------------------------------------------------- | ------------------------------------------------------------------------------- |
| `FULL`                      | `INSERT OVERWRITE` (replace target content)        | Good for small tables, aggregates, rebuildable marts.                           |
| `VIEW`                      | Spark view (where the catalog/runtime supports it) | Staging and intermediate layer.                                                 |
| `SEED` / `EMBEDDED`         | File-backed or embedded reference data             | Small reference tables.                                                         |
| `INCREMENTAL_BY_TIME_RANGE` | `INSERT OVERWRITE` by time-window partition        | Keep `time_column` UTC; filter source by interval macros.                       |
| `INCREMENTAL_BY_PARTITION`  | `INSERT OVERWRITE` by partition key                | Best when the partition key is the natural restatement boundary.                |
| `INCREMENTAL_BY_UNIQUE_KEY` | Iceberg `MERGE INTO`                               | Requires Iceberg `physical_properties`; supports upsert and conditional delete. |
| `SCD_TYPE_2`                | Spark/Iceberg slowly changing dimension            | Dimension history with current and historical rows.                             |

{% hint style="warning" %}
Declare Iceberg `physical_properties` on all production tables. `INCREMENTAL_BY_UNIQUE_KEY` compiles to Iceberg `MERGE INTO`: without `physical_properties(format = 'iceberg')` or `table_format iceberg`, the model silently does nothing. Align `grain`/`grains` with `unique_key`.
{% endhint %}

```sql
MODEL (
  name lakehouse_depot.tpch_lakehouse.customer_360,
  kind FULL,
  owner 'data-product-owner',
  grains (customer_key),
  physical_properties (
    format = 'iceberg',
    'write.format.default' = 'parquet',
    'write.parquet.compression-codec' = 'snappy'
  ),
  assertions (
    unique_values(columns := customer_key),
    not_null(columns := (customer_key, customer_name))
  )
);
SELECT
  cn.c_custkey                                               AS customer_key,
  cn.c_name                                                  AS customer_name,
  cn.c_mktsegment                                            AS market_segment,
  COUNT(DISTINCT oi.l_orderkey)                              AS order_count,
  COALESCE(SUM(oi.net_amount), 0)                            AS total_revenue
FROM lakehouse_depot.tpch_lakehouse.int_customer_nation cn
LEFT JOIN lakehouse_depot.tpch_lakehouse.int_order_items oi
  ON oi.o_custkey = cn.c_custkey
GROUP BY cn.c_custkey, cn.c_name, cn.c_mktsegment;
```

A MERGE-heavy unique-key model uses `source.` and `target.` aliases; new keys fall through to an implicit insert, so delete tombstones for missing keys insert rows unless filtered upstream:

```sql
kind INCREMENTAL_BY_UNIQUE_KEY (
  unique_key c_custkey,
  when_matched (
    WHEN MATCHED AND source.last_op = 'D' THEN DELETE
    WHEN MATCHED AND source.last_op IN ('I', 'U') THEN UPDATE SET ...
  )
),
table_format iceberg,
grain (c_custkey)
```

### Python models

Python models should return a Spark DataFrame, using `context.spark.sql()` or the DataFrame API. Do not convert large inputs to Pandas on the driver: this pulls data to the driver and causes OOM. A Pandas return is safe only for a bounded summary (one row). RDD APIs are supported for legacy or custom logic but bypass the Catalyst optimizer and Iceberg column pruning, so prefer DataFrames in production models; avoid `collect()` and `toPandas()` on large tables.

### Identifier casing

Use UPPERCASE dimensions and measure expressions to match the physical column casing in the underlying Iceberg tables (as aliased in staging). Lowercase causes `column not found`. `COUNT(DISTINCT …)` over large Iceberg tables is shuffle-heavy and the primary sizing driver; declare explicit joins, because cross-join-shaped queries are the biggest avoidable latency risk.

### Semantic models

Semantic models wrap Vulcan-managed Iceberg tables and expose dimensions, measures, segments, and joins to API and SQL-wire consumers. The API track translates semantic queries into Spark/Iceberg reads. Declare explicit joins in semantics — cross-join-shaped queries are the biggest avoidable latency risk on Spark. Identifier casing follows the same UPPERCASE rule as models above.

```yaml
kind: semantic
name: CUSTOMER_ORDERS_SUMMARY
depends_on: lakehouse_depot.tpch_lakehouse.customer_360
dimensions:
  - MARKET_SEGMENT
measures:
  - name: TOTAL_REVENUE
    type: sum
    expression: "{CUSTOMER_ORDERS_SUMMARY.TOTAL_REVENUE}"
  - name: CUSTOMER_COUNT
    type: count_distinct
    expression: "{CUSTOMER_ORDERS_SUMMARY.CUSTOMER_KEY}"
segments:
  - name: positive_balance
    expression: "{CUSTOMER_ORDERS_SUMMARY.ACCT_BAL} > 0"
```

### Business metrics

A metric is a YAML manifest that references a measure already defined in a semantic model. Vulcan compiles it to a Spark SQL aggregation executed against Iceberg.

```yaml
kind: metric
name: DAILY_REVENUE
measure: CUSTOMER_ORDERS_SUMMARY.TOTAL_REVENUE
ts: CUSTOMER_ORDERS_SUMMARY.ORDER_DATE
granularity: day
dimensions:
  - name: MARKET_SEGMENT
    ref: CUSTOMER_ORDERS_SUMMARY.MARKET_SEGMENT
```

Two Spark-specific behaviors matter at runtime: `COUNT(DISTINCT …)` over large Iceberg tables is shuffle-heavy and the primary compute sizing driver, so size up executors or pre-aggregate distinct counts into a mart before rewriting the metric; and identifier casing in metric `filters`/`expressions` follows the same UPPERCASE rule as semantic models — lowercase references cause "column not found" errors.

## Data quality

Vulcan quality is layered on Spark. Live checks execute as Spark SQL against Iceberg tables and consume compute from the workflow run or a dedicated quality track.

| Layer                        | Where it runs                                  | Spark cost                | When it catches a problem                 |
| ---------------------------- | ---------------------------------------------- | ------------------------- | ----------------------------------------- |
| Linter                       | Locally, before Spark execution                | None                      | Authoring time                            |
| Assertions (in `MODEL(...)`) | Spark during model materialisation             | Counted in the model run  | Every run                                 |
| Unit tests (`tests/`)        | Locally (DuckDB in-process)                    | No cluster cost           | Pre-deploy logic regressions              |
| Audits (`audits/`)           | Spark after model materialises                 | One Spark query per audit | Post-run relationship failures            |
| Data quality (`kind: dq`)    | Spark SQL after model execution or on schedule | One or more Spark jobs    | Observability, drift, freshness, accuracy |

Use `assertions` inside `MODEL(...)` for invariants that make the table unsafe if they fail (primary key uniqueness, required columns, non-negative measures). Use `kind: dq` for non-blocking drift, freshness, and distribution checks.

```yaml
kind: dq
name: customer_360_dq
depends_on: lakehouse_depot.tpch_lakehouse.customer_360
profiles:
  - order_count
  - total_revenue
  - market_segment
rules:
  - row_count > 0:
      name: customer_360_not_empty
      dimension: completeness
  - missing_count(customer_key) = 0:
      name: no_missing_customer_key
      dimension: completeness
  - failed rows:
      name: negative_revenue
      dimension: accuracy
      fail query: |
        SELECT customer_key, total_revenue
        FROM lakehouse_depot.tpch_lakehouse.customer_360
        WHERE total_revenue < 0
      samples limit: 10
```

Keep `failed rows` queries narrow and always set `samples limit` so incident payloads stay bounded. Profile only operationally useful columns — profiling high-cardinality string columns on large Iceberg tables is expensive. Use fully qualified lakehouse table names in all DQ and audit SQL, and for partitioned Iceberg tables add partition predicates where the rule only needs the latest window.

## Endpoints

The Spark stack serves API traffic through three dedicated serving images on the deployed `api` track:

| Endpoint   | Runtime image                                 |
| ---------- | --------------------------------------------- |
| REST       | Vulcan API (`tmdcio/vulcan-spark:0.228.1.26`) |
| GraphQL    | `tmdcio/vulcan-graphql:0.228.1.26`            |
| MySQL-wire | `tmdcio/mysql-wire:0.228.1.26` (port 3306)    |

Endpoint queries read the semantic layer and push execution to Spark/Iceberg. Size the API track separately from the workflow cluster, but remember that expensive semantic queries still consume Spark compute. Keep `api.limit.memory` at or above 1.5 GiB (recommended 2 GiB) to avoid OOM on large result sets.

### MCP tools

| MCP tool   | Spark behaviour                                    |
| ---------- | -------------------------------------------------- |
| `about`    | Static; no engine call                             |
| `lineage`  | Parsed graph; no live Spark job                    |
| `quality`  | Last check/audit results; typically no live call   |
| `data`     | Live Spark/Iceberg query; consumes compute         |
| `run`      | Triggers workflow; same cost as a scheduled run    |
| `activity` | Reads workflow history from DataOS; no engine call |

Treat agent-driven `data` tool queries like production dashboard queries: they can trigger Spark scans, shuffles, and Iceberg reads through the semantic layer.

## Metadata scanning and catalog

DataOS scans two Spark/Iceberg sources: the Iceberg REST catalog (real-time structural metadata — tables, schemas, columns, partition specs) and object-storage scan logs or Iceberg history (lineage).

| Object                             | Catalogue                    | Lineage             |
| ---------------------------------- | ---------------------------- | ------------------- |
| Namespaces (schemas)               | ✓                            | —                   |
| Iceberg tables                     | ✓                            | ✓ as nodes          |
| Columns with types                 | ✓                            | ✓ column-level      |
| Partition specs                    | ✓ as metadata                | —                   |
| Lineage edges (from parsed models) | —                            | ✓ from Vulcan parse |
| Iceberg snapshots / history        | Partially shown              | Used internally     |
| Views                              | Ingested if catalog supports | ✓ if declared       |

Lineage is primarily Vulcan-computed: DataOS parses the model SQL graph and `external_models.yaml`. Iceberg metadata contributes snapshot and schema evolution history.

The scanner role is read-only and separate from the Vulcan service role. Grant the scanner principal `purpose: scan` on the lakehouse depot, with:

```
Iceberg REST catalog:   GET /v1/catalogs/*, /v1/namespaces/*, /v1/tables/*
S3:                     s3:GetObject, s3:ListBucket on the warehouse bucket
ABFSS:              Storage Blob Data Reader on the container
```

The scanner runs every 6 to 12 hours (configurable); structural changes (new tables, columns) appear after the next scan. Lineage from model runs is computed from Vulcan's SQL parse and is available immediately after a successful `vulcan run`. Iceberg snapshot-level lineage (for example partition evolution) may lag behind the catalog scan cycle.

### Lineage and version rollback

Spark does not infer missing lineage from the Iceberg catalog automatically. Declare external tables in `external_models.yaml`:

```yaml
- name: lakehouse_depot.tpch_lakehouse.int_customer_nation
  columns:
    c_custkey:    BIGINT
    c_name:       STRING
    c_mktsegment: STRING
```

Without the declaration, the linter throws `Table not found` and lineage stops at the model boundary. Keep fully qualified lakehouse names consistent across models, DQ files, and semantic files.

Rollback replays the previous materialisation strategy and produces new Iceberg snapshots. For `INCREMENTAL_BY_UNIQUE_KEY`, plan rollbacks carefully: the MERGE path is not a cheap metadata pointer flip, it consumes Spark compute and Iceberg commits.

## Engine-native feature support

| Spark / Iceberg feature            | Vulcan pattern                                                          | Notes                                                                                                |
| ---------------------------------- | ----------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| Spark runtime                      | Tenant-level `vulcan-spark` stack                                       | Configured once per tenant by SRE                                                                    |
| Spark driver                       | `spec.workflow.resource.driver`                                         | Runs migrate, plan, run                                                                              |
| Spark executors                    | `spec.workflow.resource.executor`                                       | Distributed scan, shuffle, join, write                                                               |
| Iceberg REST catalog               | Lakehouse depot in `spec.depots[]`                                      | Stack generates `spark.sql.catalog.*` automatically                                                  |
| S3 object storage                  | Lakehouse with `storageType: s3`                                        | —                                                                                                    |
| ABFSS object storage               | Lakehouse with `storageType: abfss`                                     | May need Azure bundle JAR                                                                            |
| Full-refresh Iceberg table         | kind `FULL` + `physical_properties(format = 'iceberg')`                 | Rebuildable; good for aggregates and marts                                                           |
| Time-range incremental             | `INCREMENTAL_BY_TIME_RANGE`                                             | Insert overwrite by interval partition                                                               |
| Partition incremental              | `INCREMENTAL_BY_PARTITION`                                              | Insert overwrite by partition key                                                                    |
| Unique-key upsert                  | `INCREMENTAL_BY_UNIQUE_KEY` + `physical_properties(format = 'iceberg')` | Compiles to Iceberg `MERGE INTO`                                                                     |
| Conditional delete/update in MERGE | `when_matched` inside unique-key model                                  | Uses `source.` and `target.` aliases                                                                 |
| SCD Type 2                         | `SCD_TYPE_2` kind                                                       | Dimension history with current and historical rows                                                   |
| View                               | kind `VIEW`                                                             | Staging / intermediate layer                                                                         |
| SEED / EMBEDDED                    | kind `SEED` or `EMBEDDED`                                               | File-backed reference data                                                                           |
| Python models (Spark DataFrame)    | `.py` model returning `SparkDataFrame`                                  | Return Spark DataFrames; avoid Pandas for large outputs                                              |
| JVM dependencies                   | `dependencies/java/`                                                    | JARs auto-loaded by stack for driver + executors                                                     |
| Python wheel dependencies          | `dependencies/python/`                                                  | Wheels auto-loaded by stack                                                                          |
| Spark tuning — workflow            | `spec.workflow.sparkConf`                                               | Merged with generated catalog config                                                                 |
| Spark tuning — API track           | `spec.sparkConf`                                                        | Separate Spark session; set matching keys in both if the API needs the same classpath/catalog config |
| Serving                            | `spec.api` + sidecar images                                             | REST, GraphQL, MySQL-wire                                                                            |
| Spark compute lifecycle            | Boundary                                                                | Provision compute pools via DataOS admin                                                             |
| DLT / Streaming tables             | Boundary                                                                | Separate concern; land results as Iceberg tables                                                     |

Wrap `pre_statements` / `post_statements` lifecycle DDL that should only run on real execution, or `vulcan plan` dry-runs will execute it:

```sql
@IF(@runtime_stage = 'evaluating',
  ALTER TABLE lakehouse_depot.gold.orders SET TBLPROPERTIES ('write.metadata.delete-after-commit.enabled' = 'true')
);
```

## Operational boundaries

### Driver and executor sizing

| Workload                           | Starting shape                                    | Rationale                                                 |
| ---------------------------------- | ------------------------------------------------- | --------------------------------------------------------- |
| Dev / local plan (\~1 GB)          | 1 driver, 1 worker.                               | Validate syntax, catalog, and small data paths            |
| Small production models (\~1 GB)   | Driver 1 core / 2G; Executor 2× 2 cores / 4G.     | Mostly scheduler and catalog overhead                     |
| Medium production models (\~10 GB) | Driver 2 cores / 4G; Executor 3× 2 cores / 4G.    | Enough parallelism for moderate joins                     |
| Large production models (\~100 GB) | Driver 2 cores / 6G; Executor 4× 3 cores / 6G.    | Baseline for shuffle-heavy lakehouse workloads            |
| XL backfill (\~1 TB)               | Driver 4 cores / 8G; Executor 8–16× 4 cores / 8G. | Scale executor instances after tuning shuffle/read splits |

Executor parallelism is roughly `executor.cores × executor.instances`. Add 10 to 15% memory overhead. Fix shuffle partitions and adaptive query execution (AQE) before adding more executors.

### Spark tuning

| Setting                                         | Guidance                                                       |
| ----------------------------------------------- | -------------------------------------------------------------- |
| `spark.sql.shuffle.partitions`                  | First lever for shuffle-heavy stages; raise when stages spill. |
| `spark.sql.adaptive.enabled`                    | Keep `true`.                                                   |
| `spark.sql.adaptive.coalescePartitions.enabled` | Keep `true` to reduce tiny output files.                       |
| `spark.sql.adaptive.skewJoin.enabled`           | Enable for skewed joins.                                       |
| `spark.sql.files.maxPartitionBytes`             | Raise for huge scans; lower if executors OOM.                  |
| `spark.serializer`                              | Use `org.apache.spark.serializer.KryoSerializer`.              |
| `spark.sql.catalog.<depot>.commit.retry.*`      | Configure for concurrent MERGE / Iceberg commit conflicts.     |

Tuning order: shuffle partitions, AQE plus coalescing, skew-join handling, `maxPartitionBytes`, executor memory/instances, then Iceberg commit retry.

### API replicas and Kubernetes resources

| Traffic                           | API replicas |
| --------------------------------- | ------------ |
| Single team                       | 1–2          |
| Multi-team / scheduled dashboards | 3–5          |
| Enterprise / high-volume          | 5+           |

The API track is sized separately from the Spark workflow cluster:

```yaml
api:
  replicas: 2
  resource:
    request: { cpu: "200m", memory: "512Mi" }
    limit:   { cpu: "2000m", memory: "2Gi" }
```

Add API replicas for request concurrency and add workflow executors for Spark query execution — these are different bottlenecks.

### Scheduling and latency floor

Schedule after upstream files land, set `timezone: UTC`, use `concurrencyPolicy: Forbid` (prevents overlapping writers and Iceberg commit conflicts), set `endOn` 1 to 2 years out, and stagger MERGE-heavy models. The latency floor is driver and executor startup plus dependency load, Iceberg REST metadata round trips, object-storage listing/read latency, API parsing/planning/serialisation, and small-file overhead. If your downstream SLO is sub-second, Spark/Iceberg is the wrong serving path — pre-aggregate, cache into a serving store, or use a lower-latency engine for that surface.

## Performance reference

Reference sizing tiers for TPC-H-shaped joins on Iceberg over object storage, with AQE enabled. Validate on tenant data before publishing SLOs externally.

| Tier | Approx data | Driver       | Executor           | Total exec cores | `shuffle.partitions` | `maxPartitionBytes` | Expected complex-join time |
| ---- | ----------- | ------------ | ------------------ | ---------------- | -------------------- | ------------------- | -------------------------- |
| S    | \~1 GB      | 1 core / 2G  | 2× 2 cores / 4G    | 4                | 64–100               | 128 MB              | seconds to \~1 min         |
| M    | \~10 GB     | 2 cores / 4G | 3× 2 cores / 4G    | 6                | 200                  | 128 MB              | \~1–3 min                  |
| L    | \~100 GB    | 2 cores / 6G | 4× 3 cores / 6G    | 12               | 200–400              | 128 MB              | \~5–15 min                 |
| XL   | \~1 TB      | 4 cores / 8G | 8–16× 4 cores / 8G | 32–64            | 1000–2000            | 256 MB              | \~30–90 min                |

Track driver startup time, executor allocation time, shuffle read/write size, spill to disk, straggler tasks, output file count, and Iceberg commit duration as the core sizing signals. A simple validation loop: pick the closest tier, run a representative `vulcan plan` + `vulcan run` window, inspect the Spark UI for spill/stragglers/OOM/shuffle size, apply one tuning change, re-run and compare, then record the stable driver/executor/`sparkConf` combination as the workload baseline.

When concurrency degrades, fix in order: prevent overlapping writers, add Iceberg commit retry, stagger heavy schedules, add executor instances, then split across compute pools only after query shape and scheduling are clean.

### Performance ceilings

| Ceiling                                                 | Tune via                                                          |
| ------------------------------------------------------- | ----------------------------------------------------------------- |
| Shuffle-heavy joins spill to disk                       | Increase `shuffle.partitions`, executor memory, and skew handling |
| Object-storage small files dominate planning            | Keep AQE coalescing enabled; compact/rewrite tables when needed   |
| High-cardinality semantic queries run slowly            | Pre-aggregate marts; constrain governed query shapes              |
| Iceberg MERGE commit conflicts under concurrent writers | Stagger writers; raise `commit.retry.*` values                    |
| Executor OOM on wide scans                              | Lower `maxPartitionBytes` or raise executor memory                |
| Low cluster utilisation                                 | Reduce executor instances or increase partition parallelism       |

### Cost guardrails

Size from the Spark UI, not instinct: if stages spill, tune shuffle partitions and memory before adding executors; if executors sit idle, reduce instances. Use `concurrencyPolicy: Forbid` for all scheduled workflows — overlapping runs are expensive and can conflict on Iceberg commits.

Where supported, tag Spark sessions to attribute spend by project and model:

```python
context.spark.conf.set("spark.sql.session.description", "vulcan:<project>:<model>")
```

For storage cost: keep AQE coalescing enabled to control output file counts, run `VACUUM` on high-churn Iceberg tables to reclaim deleted-file storage, set short snapshot retention on Bronze/staging tables that can be re-derived, and partition deliberately (over-partitioning creates small files; under-partitioning forces full scans).

## Failure modes and troubleshooting

| Symptom                                          | Likely cause                                                | Fix                                                                                 |
| ------------------------------------------------ | ----------------------------------------------------------- | ----------------------------------------------------------------------------------- |
| Unique-key model "does nothing"                  | Iceberg format missing                                      | Add `physical_properties(format = 'iceberg')`.                                      |
| `NoClassDefFoundError: DataLakeStorageException` | ABFSS FileIO classes missing from classpath                 | Add the `iceberg-azure-bundle` JAR under `dependencies/java/`.                      |
| Dependency not available at runtime              | JAR/wheel not shipped or not imported                       | Place under `dependencies/java/` or `dependencies/python/`; redeploy.               |
| `CommitFailedException` (Iceberg)                | Concurrent writers to the same table                        | Raise commit retry; stagger schedules; `concurrencyPolicy: Forbid`.                 |
| Delete tombstones appear as rows after MERGE     | Missing-key DELETE falls through to implicit insert         | Filter tombstones before MERGE or handle downstream.                                |
| Many tiny output files slow queries              | Shuffle partitions too high or AQE coalescing off           | Keep AQE coalescing enabled; tune `shuffle.partitions`.                             |
| One task runs far longer than others             | Skewed join key                                             | Enable `spark.sql.adaptive.skewJoin.enabled`; review join strategy.                 |
| Executor OOM                                     | Partitions too large or memory too low                      | Lower `maxPartitionBytes` or raise executor memory.                                 |
| Driver OOM                                       | Python model collects a large DataFrame to the driver       | Return Spark DataFrames; avoid `toPandas()` on large outputs.                       |
| API query times out                              | Semantic query launches an expensive job                    | Pre-aggregate, add segments, constrain query shape.                                 |
| DataOS fails to resolve catalog (works locally)  | Depot missing or misnamed in `spec.depots[]`                | Verify `spec.depots[]` and the fully qualified catalog prefix.                      |
| Production `vulcan migrate` fails                | Platform-managed state store not provisioned or unreachable | Escalate to the SRE/platform team, then re-run `vulcan migrate`.                    |
| `Table not found` for external source            | External model not declared                                 | Add to `external_models.yaml` with the fully qualified lakehouse name.              |
| Time-range incremental reprocesses all history   | Missing or wrong `time_column`; interval macros not used    | Verify `kind INCREMENTAL_BY_TIME_RANGE (time_column <col>)` and the source filter.  |
| Straggler stage on a large join                  | Data skew on the join key                                   | Enable AQE skew join; review partition distribution; pre-aggregate the skewed side. |
| First job after an idle period is slow           | Spark startup plus dependency load                          | Expected cold-start cost; document in the SLO or pre-warm the cluster.              |

### Logs and where to look

| Symptom                                        | Where                                  | How                                                                                                      |
| ---------------------------------------------- | -------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| Local Docker startup issue                     | Docker compose services                | `docker compose -f docker/docker-compose.local-python.yml logs <service>` (or the python-in-docker file) |
| Spark scheduler / catalog / auth failure       | DataOS driver container                | `dataos-ctl resource log -t Vulcan -n <name> --container-group <name>-run-execute -c main`               |
| Executor OOM, shuffle-fetch failure, UDF error | Spark executor logs                    | Spark UI → application → executors                                                                       |
| API / semantic query issue                     | API track containers                   | `dataos-ctl resource log ... --container-group <name>-api`                                               |
| Iceberg commit conflict                        | Driver logs + Spark SQL execution logs | Search for `CommitFailedException` and the target table name                                             |

### Recovery procedures

| Situation                                    | Procedure                                                                                |
| -------------------------------------------- | ---------------------------------------------------------------------------------------- |
| Incremental time-range run failed mid-window | Re-run the affected window; partition overwrite is safe to replay                        |
| Unique-key MERGE failed before commit        | Re-run after checking target table snapshots and commit conflict logs                    |
| Unique-key MERGE committed bad data          | Restore with Iceberg snapshot rollback, coordinated with the platform SRE team           |
| Dependency artifact missing                  | Add the JAR/wheel under `dependencies/`, redeploy, run `vulcan plan` before `vulcan run` |
| Compute exhausted                            | Reduce concurrency, tune shuffle/read splits, then scale executor instances              |
| Schedule overlap caused an Iceberg conflict  | Stop the overlapping run; set `concurrencyPolicy: Forbid`; stagger heavy writers         |
| Workflow halted because `endOn` expired      | Update the Vulcan resource with a new `endOn` and re-apply                               |

## Deployment recipes

Every recipe sets `sparkConf` in both `spec.workflow.sparkConf` (tunes `migrate`/`plan`/`run`) and `spec.sparkConf` (tunes the API track's separate Spark session). `spec.api` is sized independently — request/limit resources plus replica count for REST/GraphQL/MySQL-wire concurrency.

### Daily incremental, small project (5–20 models)

```yaml
spec:
  engine: spark
  compute: spark-compute
  depots:
    - dataos://lakehouse-depot?purpose=rw
  workflow:
    schedule:
      crons: ['0 3 * * *']
      timezone: UTC
      endOn: '2028-01-01T00:00:00Z'
      concurrencyPolicy: Forbid
    resource:
      driver: { coreLimit: "2000m", cores: 1, memory: "2G" }
      executor: { coreLimit: "4000m", cores: 2, memory: "4G", instances: 2 }
    sparkConf:
      spark.sql.adaptive.enabled: "true"
      spark.sql.adaptive.coalescePartitions.enabled: "true"
      spark.serializer: org.apache.spark.serializer.KryoSerializer
  sparkConf:
    spark.sql.adaptive.enabled: "true"
    spark.sql.adaptive.coalescePartitions.enabled: "true"
    spark.serializer: org.apache.spark.serializer.KryoSerializer
  api:
    replicas: 2
    resource:
      request: { cpu: "200m", memory: "512Mi" }
      limit:   { cpu: "2000m", memory: "2Gi" }
```

### Medium lakehouse mart (\~10–100 GB)

```yaml
spec:
  engine: spark
  compute: spark-compute
  workflow:
    resource:
      driver: { coreLimit: "4000m", cores: 2, memory: "6G" }
      executor: { coreLimit: "4000m", cores: 3, memory: "6G", instances: 4 }
    sparkConf:
      spark.sql.shuffle.partitions: "300"
      spark.sql.adaptive.enabled: "true"
      spark.sql.adaptive.skewJoin.enabled: "true"
      spark.sql.adaptive.coalescePartitions.enabled: "true"
      spark.sql.files.maxPartitionBytes: "134217728"
      spark.serializer: org.apache.spark.serializer.KryoSerializer
  sparkConf:
    spark.sql.shuffle.partitions: "300"
    spark.sql.adaptive.enabled: "true"
    spark.sql.adaptive.skewJoin.enabled: "true"
    spark.sql.adaptive.coalescePartitions.enabled: "true"
    spark.sql.files.maxPartitionBytes: "134217728"
    spark.serializer: org.apache.spark.serializer.KryoSerializer
  api:
    replicas: 2
    resource:
      request: { cpu: "500m", memory: "1Gi" }
      limit:   { cpu: "2000m", memory: "4Gi" }
```

### Large backfill (first run, 1–5 years of history)

```yaml
spec:
  workflow:
    type: trigger                 # one-shot; no recurring schedule
    resource:
      driver: { coreLimit: "4000m", cores: 4, memory: "8G" }
      executor: { coreLimit: "6000m", cores: 4, memory: "8G", instances: 8 }
    sparkConf:
      spark.sql.shuffle.partitions: "1000"
      spark.sql.files.maxPartitionBytes: "268435456"
      spark.sql.adaptive.enabled: "true"
      spark.sql.adaptive.skewJoin.enabled: "true"
  sparkConf:
    spark.sql.shuffle.partitions: "1000"
    spark.sql.files.maxPartitionBytes: "268435456"
    spark.sql.adaptive.enabled: "true"
    spark.sql.adaptive.skewJoin.enabled: "true"
  api:
    replicas: 2
    resource:
      request: { cpu: "200m", memory: "512Mi" }
      limit:   { cpu: "2000m", memory: "2Gi" }
```

Run backfills off-peak, and return to the normal executor count immediately after the backfill completes. The API track keeps serving existing materialized data throughout — its sizing doesn't need to scale with the backfill's driver/executor resources.

### MERGE-heavy CDC project

```yaml
spec:
  workflow:
    schedule:
      crons: ['30 2 * * *']
      timezone: UTC
      concurrencyPolicy: Forbid
    resource:
      driver: { coreLimit: "4000m", cores: 2, memory: "6G" }
      executor: { coreLimit: "4000m", cores: 3, memory: "6G", instances: 4 }
    sparkConf:
      spark.sql.shuffle.partitions: "300"
      spark.sql.adaptive.enabled: "true"
      spark.sql.catalog.lakehouse_depot.commit.retry.num-retries: "10"
      spark.sql.catalog.lakehouse_depot.commit.retry.min-wait-ms: "1000"
      spark.sql.catalog.lakehouse_depot.commit.retry.max-wait-ms: "60000"
  sparkConf:
    spark.sql.shuffle.partitions: "300"
    spark.sql.adaptive.enabled: "true"
    spark.sql.catalog.lakehouse_depot.commit.retry.num-retries: "10"
    spark.sql.catalog.lakehouse_depot.commit.retry.min-wait-ms: "1000"
    spark.sql.catalog.lakehouse_depot.commit.retry.max-wait-ms: "60000"
  api:
    replicas: 2
    resource:
      request: { cpu: "500m", memory: "1Gi" }
      limit:   { cpu: "2000m", memory: "4Gi" }
```

Pair with unique-key models that explicitly declare Iceberg format:

```sql
MODEL (
  name lakehouse_depot.crm.customer_snapshot,
  kind INCREMENTAL_BY_UNIQUE_KEY (
    unique_key c_custkey,
    batch_size 1,
    when_matched (
      WHEN MATCHED AND source.last_op = 'D' THEN DELETE
      WHEN MATCHED AND source.last_op IN ('I', 'U') THEN UPDATE SET
        c_name       = source.c_name,
        c_mktsegment = source.c_mktsegment,
        last_op      = source.last_op
    )
  ),
  table_format iceberg,
  partitioned_by (source_ds),
  grain (c_custkey)
);
```

`when_matched` branches use `source.` and `target.` aliases. New keys fall through to implicit insert, so delete tombstones for missing keys insert rows unless filtered upstream.

## Related

* [Connect engine → Spark](broken://pages/04ff1cff579c5d0fa2fceda647adf1095da653f7) for the quick connection steps.
* [LDK setup guide](https://v2.dataos.info/build/readme/ldk-setup) for project initialisation.
* [Semantic models](#) and [Business metrics](#) for the semantic layer referenced above.
* [Tests](#), [Audits](#), and [Data quality](#) for the quality layers referenced above.
* [Engine guide overview](file:///) for the cross-engine rules.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://v2.dataos.info/references/engine-guide/spark-1.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.