Spark
Configure Apache Spark as the engine for your Data Product.
Apache Spark suits large-scale distributed processing on DataOS Lakehouse. This page covers the connection. For the full Docker setup, catalog options, and reference, see the Spark engine manual in References.
Before you start
A Spark-capable compute pool provisioned by your SRE team (verify with
dataos-ctl resource -t compute get -a), plus the tenant-levelvulcan-sparkstack, a DataOS lakehouse resource, and a lakehouse depot. Vulcan brings no compute, storage, or state store of its own — request these from your platform team before writing a model.Local development runs a small Docker stack next to your project (Spark master and workers, Iceberg REST catalog, object store, and a Postgres state store). Running Vulcan on your machine needs Docker, Java 17, and Python 3.13, the same minor version as the Spark workers; any other version fails with
PYTHON_VERSION_MISMATCH. Running Vulcan in thevulcan-clicontainer needs Docker only.An external
state_connection: Vulcan state can't live in the Iceberg catalog. Locally, use the stack's Postgresstatestore; in production, use Postgres, or omit it if your tenant's platform provisions state automatically.DataOS access: a
roles:id:data-dev-equivalent role tag,depot:rw:<lakehouse-depot-name>(ordepot:rfor read-only), Can Use on Compute/Depots/Secrets, and a git-sync Secret.
Permissions
Vulcan needs to create and manage tables in the configured catalog, read and write the configured storage, and submit Spark applications.
Connection options
type
Yes
Must be spark
catalog
No
Catalog for commands; defaults to spark_catalog for Spark < 3.4
config_dir
No
Value for SPARK_CONFIG_DIR
config
No
Key/value pairs for Spark configuration (S3 credentials, executor settings)
Authentication methods
Authentication is configuration-based, set through config or SPARK_CONFIG_DIR, depending on the catalog:
S3:
spark.hadoop.fs.s3a.*credentials.HDFS: Kerberos settings.
Iceberg REST catalog: catalog-specific auth. This path is Iceberg-only — there is no Delta Lake support on the Spark adapter.
Example
This is the local shape, trimmed. catalog names the Iceberg (REST) catalog and must match the spark.sql.catalog.<name> keys; Iceberg is the only supported catalog. Keep before_all: SELECT current_catalog() in every Spark project that sets catalog: it loads the catalog on the main thread so worker-thread connections don't fail with "Failed to discover Spark objects". JARs come from the image (copied to ~/.cache/vulcan/spark-jars/) and your dependencies/java/ folder, so don't add spark.jars.packages. For the full block, including the object-store keys and the .env store switch, see Configure config.yaml in the Spark engine manual. Pass all secrets through environment variables. dialect is spark2, not spark (a common typo since the connection type itself is spark).
Supported model kinds
FULL, SEED / EMBEDDED, INCREMENTAL_BY_TIME_RANGE, INCREMENTAL_BY_PARTITION, INCREMENTAL_BY_UNIQUE_KEY (requires Iceberg physical_properties), and SCD_TYPE_2. FULL and the incremental kinds compile to INSERT OVERWRITE (full replace, or scoped by time window / partition key); INCREMENTAL_BY_UNIQUE_KEY compiles to an Iceberg MERGE INTO. For how each kind materializes, see the Spark engine manual.
Wrap lifecycle DDL that should only run on real execution in pre_statements / post_statements, guarded with @IF(@runtime_stage = 'evaluating', ...). In Python models, return a Spark DataFrame and avoid collect() / toPandas() on large tables — pulling data to the driver causes OOM.
Limits and performance
Size drivers and executors to the workload: roughly driver 1 core/2G + 2×2-core/4G executors for small models (~1 GB), scaling up to driver 4 cores/8G + 8–16×4-core/8G executors for a ~1 TB backfill. Executor parallelism is roughly executor.cores × executor.instances; add 10–15% memory overhead, and fix spark.sql.shuffle.partitions and adaptive query execution (AQE) before adding more executors. Use concurrencyPolicy: Forbid on every scheduled workflow to prevent overlapping writers and Iceberg commit conflicts.
Spark/Iceberg is not a low-latency serving path — pre-aggregate for sub-second surfaces. If concurrency degrades, fix in order: prevent overlapping writers, add Iceberg commit retry, stagger heavy schedules, add executor instances, then split across compute pools only after query shape and scheduling are clean.
Troubleshooting
Unique-key model "does nothing"
Iceberg format missing
Add physical_properties(format = 'iceberg').
CommitFailedException (Iceberg)
Concurrent writers to the same table
Raise commit retry; stagger schedules; concurrencyPolicy: Forbid.
Driver OOM
Python model collects a large DataFrame to the driver
Return Spark DataFrames; avoid toPandas() on large outputs.
DataOS fails to resolve catalog (works locally)
Depot missing or misnamed in deployment.depots
Verify deployment.depots and the fully qualified catalog prefix.
vulcan plan runs DDL
Lifecycle DDL not guarded
Wrap it in @IF(@runtime_stage = 'evaluating', ...).
For the full failure-mode table, log locations, and recovery procedures, see Troubleshooting in the Spark engine manual.
Once vulcan info succeeds, continue to Assets.
Last updated
Was this helpful?