For the complete documentation index, see llms.txt. This page is also available as Markdown.

Spark

Configure Apache Spark as the engine for your Data Product.

Apache Spark suits large-scale distributed processing on DataOS Lakehouse. This page covers the connection. For the full Docker setup, catalog options, and reference, see the Spark engine manual in References.

Before you start

  • A Spark-capable compute pool provisioned by your SRE team (verify with dataos-ctl resource -t compute get -a), plus the tenant-level vulcan-spark stack, a DataOS lakehouse resource, and a lakehouse depot. Vulcan brings no compute, storage, or state store of its own — request these from your platform team before writing a model.

  • Local development runs a small Docker stack next to your project (Spark master and workers, Iceberg REST catalog, object store, and a Postgres state store). Running Vulcan on your machine needs Docker, Java 17, and Python 3.13, the same minor version as the Spark workers; any other version fails with PYTHON_VERSION_MISMATCH. Running Vulcan in the vulcan-cli container needs Docker only.

  • An external state_connection: Vulcan state can't live in the Iceberg catalog. Locally, use the stack's Postgres statestore; in production, use Postgres, or omit it if your tenant's platform provisions state automatically.

  • DataOS access: a roles:id:data-dev-equivalent role tag, depot:rw:<lakehouse-depot-name> (or depot:r for read-only), Can Use on Compute/Depots/Secrets, and a git-sync Secret.

Permissions

Vulcan needs to create and manage tables in the configured catalog, read and write the configured storage, and submit Spark applications.

Connection options

Option
Required
Description

type

Yes

Must be spark

catalog

No

Catalog for commands; defaults to spark_catalog for Spark < 3.4

config_dir

No

Value for SPARK_CONFIG_DIR

config

No

Key/value pairs for Spark configuration (S3 credentials, executor settings)

Authentication methods

Authentication is configuration-based, set through config or SPARK_CONFIG_DIR, depending on the catalog:

  • S3: spark.hadoop.fs.s3a.* credentials.

  • HDFS: Kerberos settings.

  • Iceberg REST catalog: catalog-specific auth. This path is Iceberg-only — there is no Delta Lake support on the Spark adapter.

Example

This is the local shape, trimmed. catalog names the Iceberg (REST) catalog and must match the spark.sql.catalog.<name> keys; Iceberg is the only supported catalog. Keep before_all: SELECT current_catalog() in every Spark project that sets catalog: it loads the catalog on the main thread so worker-thread connections don't fail with "Failed to discover Spark objects". JARs come from the image (copied to ~/.cache/vulcan/spark-jars/) and your dependencies/java/ folder, so don't add spark.jars.packages. For the full block, including the object-store keys and the .env store switch, see Configure config.yaml in the Spark engine manual. Pass all secrets through environment variables. dialect is spark2, not spark (a common typo since the connection type itself is spark).

Supported model kinds

FULL, SEED / EMBEDDED, INCREMENTAL_BY_TIME_RANGE, INCREMENTAL_BY_PARTITION, INCREMENTAL_BY_UNIQUE_KEY (requires Iceberg physical_properties), and SCD_TYPE_2. FULL and the incremental kinds compile to INSERT OVERWRITE (full replace, or scoped by time window / partition key); INCREMENTAL_BY_UNIQUE_KEY compiles to an Iceberg MERGE INTO. For how each kind materializes, see the Spark engine manual.

Wrap lifecycle DDL that should only run on real execution in pre_statements / post_statements, guarded with @IF(@runtime_stage = 'evaluating', ...). In Python models, return a Spark DataFrame and avoid collect() / toPandas() on large tables — pulling data to the driver causes OOM.

Limits and performance

Size drivers and executors to the workload: roughly driver 1 core/2G + 2×2-core/4G executors for small models (~1 GB), scaling up to driver 4 cores/8G + 8–16×4-core/8G executors for a ~1 TB backfill. Executor parallelism is roughly executor.cores × executor.instances; add 10–15% memory overhead, and fix spark.sql.shuffle.partitions and adaptive query execution (AQE) before adding more executors. Use concurrencyPolicy: Forbid on every scheduled workflow to prevent overlapping writers and Iceberg commit conflicts.

Spark/Iceberg is not a low-latency serving path — pre-aggregate for sub-second surfaces. If concurrency degrades, fix in order: prevent overlapping writers, add Iceberg commit retry, stagger heavy schedules, add executor instances, then split across compute pools only after query shape and scheduling are clean.

Troubleshooting

Symptom
Likely cause
Fix

Unique-key model "does nothing"

Iceberg format missing

Add physical_properties(format = 'iceberg').

CommitFailedException (Iceberg)

Concurrent writers to the same table

Raise commit retry; stagger schedules; concurrencyPolicy: Forbid.

Driver OOM

Python model collects a large DataFrame to the driver

Return Spark DataFrames; avoid toPandas() on large outputs.

DataOS fails to resolve catalog (works locally)

Depot missing or misnamed in deployment.depots

Verify deployment.depots and the fully qualified catalog prefix.

vulcan plan runs DDL

Lifecycle DDL not guarded

Wrap it in @IF(@runtime_stage = 'evaluating', ...).

For the full failure-mode table, log locations, and recovery procedures, see Troubleshooting in the Spark engine manual.

Once vulcan info succeeds, continue to Assets.

Last updated

Was this helpful?