> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/engine-guide/spark-1.md).

# Spark

Technical guide to building and running a Data Product on Apache Spark with Iceberg: connection setup, materialization, tuning, and troubleshooting.

Apache Spark is a distributed processing engine for transforming data across configured catalogs and data sources. Vulcan uses Spark to run SQL and Python models, combine data at scale, and materialize governed outputs in formats such as Apache Iceberg.

## When to use

Choose Spark for distributed processing across configured catalogs and data sources, especially for large transformations, shuffle-intensive joins, backfills, and governed materialization into targets such as Apache Iceberg. Spark can support interactive queries, but it is primarily optimized for distributed analytical workloads rather than consistently low-latency serving. For sub-second consumption requirements, consider pre-aggregating results or using a dedicated serving engine.

Spark performance depends on workload characteristics such as data volume, partitioning, shuffle size, file layout, source-system throughput, and available driver and executor resources. Use the [limits and performance guide](https://v2.dataos.info/references/engine-guide/spark-1/limits-and-performance.md) to size workloads from observed execution behavior rather than row count alone.

### Common model kinds

Common Spark model kinds include `FULL`, `SEED`, `EMBEDDED`, `INCREMENTAL_BY_TIME_RANGE`, `INCREMENTAL_BY_PARTITION`, `INCREMENTAL_BY_UNIQUE_KEY`, `SCD_TYPE_2_BY_TIME`, and `SCD_TYPE_2_BY_COLUMN`. See [Materialization behavior](https://v2.dataos.info/references/engine-guide/spark-1/materialization-behaviour.md) for the supported mappings and write behavior.

{% hint style="warning" %}
Virtual Data Environment (VDE) is not supported when the engine is Spark. Configuration validation rejects \`vde: true\` when a configured connection uses \`type: spark\`.
{% endhint %}

### Unsupported and out-of-scope capabilities

| Feature                                      | Limitation                                              | Alternative                                                     |
| -------------------------------------------- | ------------------------------------------------------- | --------------------------------------------------------------- |
| Views                                        | `VIEW` and materialized views are not supported         | Use a table-backed model such as `FULL` or an incremental model |
| Sub-second API serving for arbitrary queries | Spark does not guarantee consistently low query latency | Pre-aggregate results or use a lower-latency serving engine     |

## Before you begin

Before configuring a production Spark gateway, ensure that the required DataOS resources and access controls are in place. These include a Spark-capable Compute resource, the tenant-level `vulcan-spark` stack, a DataOS Lakehouse Depot, and the required Secrets.

See [Spark Lakehouse permissions](https://v2.dataos.info/references/engine-guide/spark-1/permissions-and-grants.md) for catalog and object-storage access requirements.

## Connect Spark

Connect Vulcan to Spark by configuring a Spark gateway. The gateway tells Vulcan how to start Spark and which catalogs to use. Add the JVM and Python dependencies required by your models. In DataOS, attach Lakehouse Depots and store credentials in Secrets. See [connection-options.md](https://v2.dataos.info/references/engine-guide/spark-1/connection-options.md) for setup instructions.

## Build Spark models

On the DataOS Lakehouse path, table-backed Spark SQL and Python models materialize as Iceberg tables. Other configured catalogs can be written according to connector capabilities. `EMBEDDED` models do not create physical tables. Review [materialization behavior](https://v2.dataos.info/references/engine-guide/spark-1/materialization-behaviour.md) for model-kind mappings, Spark/Iceberg write operations, Python DataFrame models, partitioning, and identifier casing.

## Tune and operate Spark

Set driver and executor limits before running production schedules or large backfills.

For Spark resource sizing, tuning, scheduling, and performance baselines, see [limits-and-performance.md](https://v2.dataos.info/references/engine-guide/spark-1/limits-and-performance.md).

Queries served from Spark-backed models can start Spark jobs and consume compute for scans, joins, and Iceberg reads.

### Before production

* Confirm least-privilege grants for the service and consumer roles.
* Validate the project with `vulcan info` and `vulcan plan`, then perform a controlled test run in a non-production environment.
* Prevent overlapping workflows from writing to the same Iceberg table.
* Test recovery procedures before scheduling production workloads.

## Troubleshoot Spark

For common failures, log locations, and recovery procedures, see [troubleshooting.md](https://v2.dataos.info/references/engine-guide/spark-1/troubleshooting.md).

## Start faster with a recipe

Use the [deployment recipe fragments](https://v2.dataos.info/references/engine-guide/spark-1/recipes/deployment-recipes.md) as initial Spark configurations for daily incremental workloads, Lakehouse marts, backfills, and MERGE-heavy projects. Validate and tune every value with representative data before production.

## Related

* [Connect and configure Spark](https://v2.dataos.info/references/engine-guide/spark-1/connection-options.md) for catalogs, Spark properties, resources, and project dependencies.
* [Local Spark development](https://v2.dataos.info/references/engine-guide/spark-1/local-development.md) for the Docker-based development environment.
* [Spark Lakehouse limitations](https://v2.dataos.info/references/engine-guide/spark-1/spark-lakehouse-limitations.md) for deployment-specific boundaries.
* [Engine guide](https://v2.dataos.info/references/engine-guide/engine-guide.md) for cross-engine guidance.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/references/engine-guide/spark-1.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
