> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/build/productize/connect-engine/spark.md).

# Spark

Configure Apache Spark as the engine for your Data Product.

Apache Spark suits large-scale distributed processing on DataOS Lakehouse. This page covers the connection. For the full Docker setup, catalog options, and reference, see the [Spark engine manual](https://v2.dataos.info/references/engine-guide/spark-1) in References.

## Before you start

* A Spark-capable compute pool provisioned by your SRE team (verify with `dataos-ctl resource -t compute get -a`), plus the tenant-level `vulcan-spark` stack, a DataOS lakehouse resource, and a lakehouse depot. Vulcan brings no compute, storage, or state store of its own — request these from your platform team before writing a model.
* Local development runs a small Docker stack next to your project (Spark master and workers, Iceberg REST catalog, object store, and a Postgres state store). Running Vulcan on your machine needs Docker, Java 17, and **Python 3.13**, the same minor version as the Spark workers; any other version fails with `PYTHON_VERSION_MISMATCH`. Running Vulcan in the `vulcan-cli` container needs Docker only.
* An external `state_connection`: Vulcan state can't live in the Iceberg catalog. Locally, use the stack's Postgres `statestore`; in production, use Postgres, or omit it if your tenant's platform provisions state automatically.
* DataOS access: a `roles:id:data-dev`-equivalent role tag, `depot:rw:<lakehouse-depot-name>` (or `depot:r` for read-only), **Can Use** on Compute/Depots/Secrets, and a git-sync Secret.

## Permissions

Vulcan needs to create and manage tables in the configured catalog, read and write the configured storage, and submit Spark applications.

{% hint style="warning" %}
Spark is single-catalog: every model in the project must be defined under one catalog. In production, don't hand-author the catalog block: attach the lakehouse depot in `deployment.depots` in `config.yaml` and let the `vulcan-spark` stack generate the Iceberg REST catalog config.
{% endhint %}

## Connection options

| Option       | Required | Description                                                                 |
| ------------ | :------: | --------------------------------------------------------------------------- |
| `type`       |    Yes   | Must be `spark`                                                             |
| `catalog`    |    No    | Catalog for commands; defaults to `spark_catalog` for Spark < 3.4           |
| `config_dir` |    No    | Value for `SPARK_CONFIG_DIR`                                                |
| `config`     |    No    | Key/value pairs for Spark configuration (S3 credentials, executor settings) |

### Authentication methods

Authentication is configuration-based, set through `config` or `SPARK_CONFIG_DIR`, depending on the catalog:

* **S3**: `spark.hadoop.fs.s3a.*` credentials.
* **HDFS**: Kerberos settings.
* **Iceberg REST catalog**: catalog-specific auth. This path is Iceberg-only — there is no Delta Lake support on the Spark adapter.

## Example

```yaml
gateways:
  default:
    connection:
      type: spark
      catalog: warehouse
      config:
        "spark.master": "spark://spark-master:7077"
        "spark.driver.host": "{{ env_var('LOCAL_DEV_HOST', '127.0.0.1') }}"
        "spark.driver.bindAddress": "0.0.0.0"
        "spark.driver.extraClassPath": "{{ env_var('HOME') }}/.cache/vulcan/spark-jars/{{ env_var('VULCAN_VERSION', '0.228.1.32') }}/*:dependencies/java/*"
        "spark.executor.extraClassPath": "/etc/dataos/work/dependencies/java/*"
        "spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions"
        "spark.sql.defaultCatalog": "warehouse"
        "spark.sql.catalog.warehouse": "org.apache.iceberg.spark.SparkCatalog"
        "spark.sql.catalog.warehouse.type": "rest"
        "spark.sql.catalog.warehouse.uri": "{{ env_var('LDK_CATALOG_URI', 'http://iceberg-rest-warehouse:8181') }}"
        "spark.sql.catalog.warehouse.warehouse": "{{ env_var('LDK_WAREHOUSE', 's3://warehouse/') }}"
        # ...plus the object-store keys for MinIO/S3, ABFSS, or GCS (see the full block)
    state_connection:
      type: postgres
      database: statestore
      host: "{{ env_var('STATESTORE_HOST', 'localhost') }}"
      port: "{{ env_var('STATESTORE_PORT', '5431') }}"
      user: vulcan
      password: vulcan
default_gateway: default
model_defaults:
  dialect: spark2
before_all:
  - SELECT current_catalog()
```

This is the local shape, trimmed. `catalog` names the Iceberg (REST) catalog and must match the `spark.sql.catalog.<name>` keys; Iceberg is the only supported catalog. Keep `before_all: SELECT current_catalog()` in every Spark project that sets `catalog`: it loads the catalog on the main thread so worker-thread connections don't fail with "Failed to discover Spark objects". JARs come from the image (copied to `~/.cache/vulcan/spark-jars/`) and your `dependencies/java/` folder, so don't add `spark.jars.packages`. For the full block, including the object-store keys and the `.env` store switch, see [Configure config.yaml](https://v2.dataos.info/references/engine-guide/spark-1/connection-options#configure-config.yaml) in the Spark engine manual. Pass all secrets through environment variables. `dialect` is `spark2`, not `spark` (a common typo since the connection `type` itself is `spark`).

{% hint style="warning" %}
`vde: true` is rejected by validation on a Spark gateway. Leave `vde` at its default `false` (or omit it). Spark must run in direct materialization mode.
{% endhint %}

## Supported model kinds

`FULL`, `SEED` / `EMBEDDED`, `INCREMENTAL_BY_TIME_RANGE`, `INCREMENTAL_BY_PARTITION`, `INCREMENTAL_BY_UNIQUE_KEY` (requires Iceberg `physical_properties`), and `SCD_TYPE_2`. `FULL` and the incremental kinds compile to `INSERT OVERWRITE` (full replace, or scoped by time window / partition key); `INCREMENTAL_BY_UNIQUE_KEY` compiles to an Iceberg `MERGE INTO`. For how each kind materializes, see the [Spark engine manual](https://v2.dataos.info/references/engine-guide/spark-1).

{% hint style="warning" %}
`VIEW` (and materialized views) are **not supported** on this path — use `kind FULL` for staging/intermediate layers instead. Declare Iceberg `physical_properties(format = 'iceberg')` on every production table: without it, `INCREMENTAL_BY_UNIQUE_KEY` silently does nothing instead of upserting.
{% endhint %}

Wrap lifecycle DDL that should only run on real execution in `pre_statements` / `post_statements`, guarded with `@IF(@runtime_stage = 'evaluating', ...)`. In Python models, return a Spark DataFrame and avoid `collect()` / `toPandas()` on large tables — pulling data to the driver causes OOM.

## Limits and performance

Size drivers and executors to the workload: roughly driver 1 core/2G + 2×2-core/4G executors for small models (\~1 GB), scaling up to driver 4 cores/8G + 8–16×4-core/8G executors for a \~1 TB backfill. Executor parallelism is roughly `executor.cores × executor.instances`; add 10–15% memory overhead, and fix `spark.sql.shuffle.partitions` and adaptive query execution (AQE) before adding more executors. Use `concurrencyPolicy: Forbid` on every scheduled workflow to prevent overlapping writers and Iceberg commit conflicts.

Spark/Iceberg is not a low-latency serving path — pre-aggregate for sub-second surfaces. If concurrency degrades, fix in order: prevent overlapping writers, add Iceberg commit retry, stagger heavy schedules, add executor instances, then split across compute pools only after query shape and scheduling are clean.

## Troubleshooting

| Symptom                                         | Likely cause                                          | Fix                                                                 |
| ----------------------------------------------- | ----------------------------------------------------- | ------------------------------------------------------------------- |
| Unique-key model "does nothing"                 | Iceberg format missing                                | Add `physical_properties(format = 'iceberg')`.                      |
| `CommitFailedException` (Iceberg)               | Concurrent writers to the same table                  | Raise commit retry; stagger schedules; `concurrencyPolicy: Forbid`. |
| Driver OOM                                      | Python model collects a large DataFrame to the driver | Return Spark DataFrames; avoid `toPandas()` on large outputs.       |
| DataOS fails to resolve catalog (works locally) | Depot missing or misnamed in `deployment.depots`      | Verify `deployment.depots` and the fully qualified catalog prefix.  |
| `vulcan plan` runs DDL                          | Lifecycle DDL not guarded                             | Wrap it in `@IF(@runtime_stage = 'evaluating', ...)`.               |

For the full failure-mode table, log locations, and recovery procedures, see [Troubleshooting](https://v2.dataos.info/references/engine-guide/spark-1/troubleshooting) in the Spark engine manual.

Once `vulcan info` succeeds, continue to [Assets](https://v2.dataos.info/build/productize/assets.md).


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/build/productize/connect-engine/spark.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
