> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/build/v1/productize/connect-engine/spark.md).

# Spark

Apache Spark suits large-scale distributed processing on DataOS Lakehouse. This page covers the connection. For the full Docker setup, catalog options, and reference, see the [Spark engine manual](https://v2.dataos.info/references/engine-guide/spark-1) in References.

## Before you start

* For every data product using the Spark engine, DataOS automatically provisions and attaches a managed Spark cluster (Spark 3.5.5). You don't need to stand up your own standalone cluster.
* Local development requires Python 3.10. The Spark runtime inside the Vulcan image manages its own Python version separately.
* An external `state_connection`: Vulcan state can't live in the Iceberg catalog. Use DuckDB locally and Postgres in production, or omit it if your tenant's platform provisions state automatically.
* DataOS access: a `roles:id:data-dev`-equivalent role tag, `depot:rw:<lakehouse-depot-name>` (or `depot:r` for read-only), **Can Use** on Compute/Depots/Secrets, and a git-sync Secret.

## Permissions

Vulcan needs to create and manage tables in the configured catalog, read and write the configured storage, and submit Spark applications.

{% hint style="warning" %}
Spark is single-catalog: every model in the project must be defined under one catalog. In production, don't hand-author the catalog block: attach the lakehouse depot in `spec.depots[]` and let the `vulcan-spark` stack generate the Iceberg REST catalog config.
{% endhint %}

## Connection options

| Option       | Required | Description                                                                 |
| ------------ | :------: | --------------------------------------------------------------------------- |
| `type`       |    Yes   | Must be `spark`                                                             |
| `catalog`    |    No    | Catalog for commands; defaults to `spark_catalog` for Spark < 3.4           |
| `config_dir` |    No    | Value for `SPARK_CONFIG_DIR`                                                |
| `config`     |    No    | Key/value pairs for Spark configuration (S3 credentials, executor settings) |

### Authentication methods

Authentication is configuration-based, set through `config` or `SPARK_CONFIG_DIR`, depending on the catalog:

* **S3**: `spark.hadoop.fs.s3a.*` credentials.
* **HDFS**: Kerberos settings.
* **Iceberg / Delta**: catalog-specific auth.

## Example

```yaml
gateways:
  default:
    connection:
      type: spark
      catalog: iceberg_catalog
      config:
        spark.hadoop.fs.s3a.endpoint: s3.amazonaws.com
        spark.hadoop.fs.s3a.access.key: "{{ env_var('S3_ACCESS_KEY') }}"
        spark.hadoop.fs.s3a.secret.key: "{{ env_var('S3_SECRET_KEY') }}"
    state_connection:
      type: duckdb
      database: ./.state.db
default_gateway: default
model_defaults:
  dialect: spark2
```

Set `catalog` to your Iceberg, Hive, or Delta catalog name. Pass all secrets through environment variables. `dialect` is `spark2`, not `spark` (a common typo since the connection `type` itself is `spark`).

{% hint style="warning" %}
`vde: true` is rejected by validation on a Spark gateway. Leave `vde` at its default `false` (or omit it). Spark must run in direct materialization mode.
{% endhint %}

## Supported model kinds

`FULL`, `VIEW`, `SEED` / `EMBEDDED`, `INCREMENTAL_BY_TIME_RANGE`, `INCREMENTAL_BY_PARTITION`, `INCREMENTAL_BY_UNIQUE_KEY` (requires Iceberg `physical_properties`), and `SCD_TYPE_2`. For how each kind materializes, see the [Spark engine manual](https://v2.dataos.info/references/engine-guide/spark-1).

Once `vulcan info` succeeds, continue to [Assets](/build/v1/productize/assets.md).


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://v2.dataos.info/build/v1/productize/connect-engine/spark.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.