Connect to Spark
Configure the Vulcan gateway connection to Spark and load the JVM and Python dependencies the driver and executors need.
With roles and platform resources in place from Permissions and grants, this is the connect stage: configure the gateway connection Vulcan uses to reach Spark, then load the dependencies the driver and executors need before you write a model.
Connection options
Set these under gateways.<name>.connection. Spark is single-catalog: all models must be defined under one catalog.
type
Engine type name. Must be spark.
string
Yes
config_dir
Value to set for SPARK_CONFIG_DIR.
string
No
catalog
The catalog to use when issuing commands.
string
No
config
Key/value pairs for the Spark configuration, including the Iceberg catalog setup.
dict
No
If catalog is not set, the default depends on the Spark version: Spark 3.4 and above determine it at runtime; earlier versions default to spark_catalog. Authentication is configuration-based: credentials come through the config parameter or the catalog (S3, HDFS).
Local development
Vulcan behaves the same on every engine: the same project, the same vulcan plan, vulcan run, and vulcan fetchdf. Spark is the exception. It is not a single server you connect to, so local development needs a small stack running next to your project:
Spark master and two workers
Run the queries Vulcan sends.
Iceberg REST catalog
Tracks your tables.
Object store
Holds table data: local MinIO, or AWS S3, Azure ABFSS, or Google GCS.
Postgres
Stores Vulcan state.
One Docker Compose file adds this stack to your project. Start against MinIO, which needs no cloud account. Moving to S3, ABFSS, or GCS later only changes a few lines in .env.
Choose how you run Vulcan
The stack always runs in Docker. Vulcan acts as the Spark driver and runs in one of two places:
Host requirements
Docker, Java 17, Python 3.13
Docker only
Vulcan runs in
A Python virtual environment on your machine
The vulcan-cli container
Run commands as
vulcan plan
docker exec ldk-vulcan-cli vulcan plan
Extra setup
One hosts-file entry and your LAN IP
None
IDE and debugger
Work natively
Limited
Best for
Everyday development
Machines without Java or Python
Both modes use the same project, compose file, and .env. Where the steps differ, they are split into Local and Containerized tabs. Running the driver inside Linux (Containerized) avoids Windows Hadoop and winutils.exe issues.
Prerequisites
Both modes need:
Docker Desktop with at least 8 GB of memory and about 6 GB of free disk.
The Vulcan wheel,
vulcan-0.228.1.32-py3-none-any.whl.On Apple Silicon, the Spark image is amd64-only and runs under emulation. It is slower but works.
Check Java and Python, then install Vulcan in a virtual environment.
macOS/Linux:
Windows PowerShell:
Python must be 3.13, the same minor version as the Spark workers. Any other version fails with
PYTHON_VERSION_MISMATCH.[spark,postgres]installs PySpark (the driver) and the Postgres client (state).--use-deprecated=legacy-resolveravoids pip'sResolutionTooDeeperror on this wheel.
Add the local Docker service names to your hosts file once, so your machine can reach the containers by name.
macOS/Linux:
Windows:
No Java, Python, or hosts-file entry is needed. Place the wheel in your project folder (see Create your Vulcan project). The vulcan-cli container installs it on first start.
Create your Vulcan project
If you already have a Spark Vulcan project, use it and skip to Add the Docker stack.
macOS/Linux:
Windows PowerShell:
Create the project folder and copy the wheel into it. Scaffold the project after the stack is up (see Start the stack and run Vulcan):
This works in a folder that already contains docker/, .env, and the wheel.
-t default creates three example models, seed data, and data quality checks, so you have something to run straight away.
Once you finish the next three sections, the project looks like this:
Add the Docker stack
Create a docker/ folder inside your project and save this file as docker/docker-compose.yml. Run every docker compose command from this folder.
Docker Compose: local stack (docker/docker-compose.yml)
ldk-spark-master
7077, 8080
Spark master. UI at http://localhost:8080.
ldk-spark-worker-1/2
n/a
Spark executors.
ldk-iceberg-rest
8181
Iceberg REST catalog.
ldk-minio
9000, 9001
MinIO (local S3). Console at http://localhost:9001 (admin / password).
ldk-minio-init
n/a
Creates the warehouse bucket, then exits.
ldk-statestore
5431
Postgres for Vulcan state.
ldk-driver-jars
n/a
Copies the image's JARs to ~/.cache/vulcan/spark-jars/<version>/ for the driver, then exits.
ldk-vulcan-cli
4040
Containerized Vulcan. Spark driver UI on 4040.
Local mode: comment out the
vulcan-cliservice at the bottom of the file and thevulcan-cli-venvvolume at the top, sodocker compose up -dskips building it.Driver JARs: the
driver-jarsservice copies Spark's Iceberg, AWS, Azure, and GCS JARs out of the Spark image on everyup, so Vulcan and the workers always load identical files.MinIO image: the official
minio/minioimages are no longer published.pgsty/miniois a community rebuild of the same software.Your own JARs: custom UDFs and JDBC drivers go in
dependencies/java/. Both Vulcan and the workers pick them up.
Choose where your data lives
Four object stores are wired in. Pick one with .env in your project root:
MinIO
Local development, runs inside the stack
Nothing
AWS S3
Your S3 bucket
Bucket, region, access key, secret key
Azure ABFSS
Your ADLS Gen2 container
Storage account name and key
Google GCS
Your GCS bucket
Bucket, service-account key file
Save the file below as .env.example and commit it. Then copy it to .env (never commit .env) and link it into docker/.
macOS/Linux:
Windows PowerShell (run as Administrator, or enable Developer Mode, to create the symlink):
Environment file: object-store selection (.env.example)
Follow these rules for .env:
Keep exactly one block uncommented. MinIO is active by default. To switch, see Switch to S3, ABFSS, or GCS.
Vulcan reads
.envfrom the project root by itself, and Docker Compose reads it through thedocker/.envsymlink. One file keeps both sides in agreement.Do not leave values blank. A blank value silently falls back to the MinIO default.
Use a dedicated prefix in every cloud bucket. Two catalogs managing one prefix corrupt each other's view of the tables.
GCS also needs the service-account key at
.secrets/gcs.json. Add.envand.secrets/to your.gitignore.
Configure config.yaml
In config.yaml, replace the gateways: block that vulcan init created with the block below, and add the before_all: block after it.
Vulcan config: gateways and before_all (config.yaml)
spark.master: spark://spark-master:7077
A container name. Works from your machine (through the hosts file) and inside vulcan-cli.
spark.driver.host
The address Spark workers use to call Vulcan back. Set through LOCAL_DEV_HOST.
spark.driver.extraClassPath
The JARs driver-jars copied out of the image, plus your dependencies/java/.
catalog: warehouse and the spark.sql.catalog.warehouse.* keys
Vulcan's tables live in the Iceberg REST catalog.
The # {% if env_var('LDK_STORE') %} part
The only store-specific part: S3/MinIO keys, ADLS keys, or the GCS key file. The Jinja tags sit in YAML comments, so editors still see valid YAML.
state_connection
Postgres from the stack. state_schema is set per store.
before_all: SELECT current_catalog()
Required. See the hint below.
Start the stack and run Vulcan
Start the stack from the docker/ folder:
Every container shows Up except ldk-minio-init and ldk-driver-jars. Those are one-off jobs and show Exited (0). http://localhost:8080 lists two workers. The first start pulls the images and takes a few minutes.
Return to the project root, set LOCAL_DEV_HOST to your machine's LAN IPv4 address, then run Vulcan.
macOS:
Linux: set the address with export LOCAL_DEV_HOST="$(hostname -I | awk '{print $1}')", then run the same vulcan commands.
Windows PowerShell:
On Windows, get the LAN IPv4 from ipconfig under your active Wi-Fi or Ethernet adapter.
The Spark workers run in containers and call Vulcan back on LOCAL_DEV_HOST. 127.0.0.1 would point at the container itself. The address changes after sleep, a network switch, or a VPN connection. If jobs hang, set it again. Always run vulcan from the project root.
Wait for the container to install the wheel (first start only), then run every command through it:
If you have not created the project yet, run docker exec ldk-vulcan-cli vulcan init spark -t default now, then update config.yaml as described in Configure config.yaml before you run vulcan plan.
To shorten the commands, set an alias.
macOS/Linux:
Windows PowerShell:
Validate your connection
Your setup works when:
vulcan infoends withData warehouse connection succeededandState backend connection succeeded.vulcan plan --auto-applyfinishes and every data quality check passes.vulcan fetchdfreturns rows.The data files are in the object store. For MinIO, run the command below (it works in Bash and PowerShell), or browse http://localhost:9001.
If a step fails, see Troubleshoot the local stack or Troubleshooting.
Switch to S3, ABFSS, or GCS
Use the same project, compose file, and commands. From docker/:
Then run the same vulcan commands as in Start the stack and run Vulcan.
Check that the catalog picked up the new store. This is the step that fails silently:
Confirm the files landed with your provider's tools (aws s3 ls, az storage blob list, gcloud storage ls).
Run down -v on every switch. The catalog keeps its table list in memory, while Vulcan's state lives in Postgres. down -v resets both together, so the new store starts clean instead of with tables that point at the old one.
FileIO
S3FileIO
S3FileIO
ADLSFileIO
GCSFileIO
Warehouse
s3://warehouse/
s3://<bucket>/<prefix>
abfss://<container>@<account>.dfs.core.windows.net/<prefix>
gs://<bucket>/<prefix>
Credentials
admin / password
Access key and secret key
Account name and key
Service-account JSON
Endpoint
http://minio-warehouse:9000
https://s3.<region>.amazonaws.com
n/a
n/a
Path-style access
true
false
n/a
n/a
For GCS, three processes touch the bucket: the catalog, Vulcan, and the Spark workers. All three read the key from .secrets/gcs.json. If the first write fails with StorageException: 401 Unauthorized, the key file was missing when the stack started. Add it, then run down -v and up -d.
Troubleshoot the local stack
Failed to discover Spark objects in …, with USE CATALOG or PARSE_SYNTAX_ERROR in .logs/
before_all is missing from config.yaml. See Configure config.yaml.
Invalid ADLS URI: s3://…, or any mismatch between the catalog and Spark
A store exported in your terminal overrode .env. Open a new terminal or clear the LDK_* variables (commands below), then run down -v, up -d, and plan again.
Jobs hang and workers never connect (Local)
LOCAL_DEV_HOST is unset or stale. Set it again. A firewall or VPN may also block the workers from reaching your machine; try Containerized.
PYTHON_VERSION_MISMATCH
Your virtual environment is not Python 3.13. See Prerequisites.
ClassNotFoundException: org.apache.iceberg.spark.SparkCatalog
Driver JARs are missing. Check docker logs ldk-driver-jars and confirm ~/.cache/vulcan/spark-jars/<version>/ is not empty.
InvalidClassException … local class incompatible
Vulcan and the workers load different JARs. Do not add spark.jars.packages; keep VULCAN_VERSION the same everywhere.
TABLE_OR_VIEW_NOT_FOUND on fetchdf
Check the table name. The example tables are vulcan_example.seed_model, full_model, and incremental_model.
port is already allocated
Another stack uses 8080, 9000, 5431, or 7077. Stop it, or change the host-side (left) port in the compose file.
ERROR: /workspace/vulcan-<version>-py3-none-any.whl not found (Containerized)
The wheel is not in your project folder.
pull access denied for minio/…
An old compose file. Use the one in Add the Docker stack.
Clear exported LDK_* variables from the current terminal:
macOS/Linux:
Windows PowerShell:
Everyday commands and debugging
Run these from the project root. In Containerized mode, prefix each with docker exec ldk-vulcan-cli.
Preview a change before applying
vulcan plan
Apply to prod
vulcan plan --auto-apply
Run scheduled models
vulcan run
Run unit tests
vulcan test
Query a table
vulcan fetchdf "select …"
Lint
vulcan lint
http://localhost:8080
Spark master: workers and applications.
http://localhost:4040
Spark jobs and SQL plans of the running vulcan command.
.logs/
Vulcan logs.
docker logs ldk-spark-worker-1
Worker-side errors. Cloud auth failures show up here.
docker logs ldk-iceberg-rest
Catalog errors, such as credentials or the wrong FileIO.
http://localhost:9001
MinIO console.
In Local mode, Vulcan is a normal Python process. Point your IDE at the virtual environment's Python and the debugger works as usual.
Tear down
From docker/:
Data in cloud buckets is never deleted. Remove your <prefix> there yourself. ~/.cache/vulcan/spark-jars/ is safe to delete; the next up recreates it.
Upgrade Vulcan
Get the new wheel. In Containerized mode, place it in the project folder.
Set
VULCAN_VERSIONto the new version. The stack picks the matching Spark image and driver JARs from it.macOS/Linux:
export VULCAN_VERSION=<new-version>Windows PowerShell:
$env:VULCAN_VERSION="<new-version>"
In Local mode, install the new wheel into your virtual environment. If the new Spark image uses a different Python minor version, recreate the virtual environment with that version. Check it with
docker run --rm --platform linux/amd64 tmdcio/vulcan-spark-base:<new-version> python3 --version.Run
docker compose up -d. Containerized Vulcan reinstalls the wheel by itself.
Object-store keys
The compose file wires only the keys the four stores need. For more, such as server-side encryption, temporary credentials, or SAS tokens, add each key in two places: on the catalog in docker/docker-compose.yml, and on Spark in config.yaml. The Spark workers write the data files themselves, so they need the keys too.
Catalog (environment:)
CATALOG_ + upper case, . → _, - → __
CATALOG_S3_ACCESS__KEY__ID
Spark (config.yaml)
spark.sql.catalog.warehouse. + the key
spark.sql.catalog.warehouse.s3.access-key-id
S3 and MinIO (S3FileIO)
s3.endpoint
MinIO: http://minio-warehouse:9000. AWS: https://s3.<region>.amazonaws.com.
s3.access-key-id, s3.secret-access-key
Static credentials.
s3.session-token
Temporary or STS credentials.
s3.path-style-access
true for MinIO, false for AWS.
s3.sse.type, s3.sse.key
Server-side encryption.
s3.acl, s3.write.storage-class
Canned ACL and storage class.
Region is not an s3.* key. Set AWS_REGION on the catalog and client.region on Spark.
ABFSS (ADLSFileIO)
adls.auth.shared-key.account.name, .account.key
Both are required together.
adls.sas-token.<account>
SAS token instead of the account key.
adls.connection-string.<account>
Connection string.
GCS (GCSFileIO)
GOOGLE_APPLICATION_CREDENTIALS
Path to the service-account JSON. Mount the file; do not paste it.
gcs.project-id
Google Cloud project ID.
gcs.oauth2.token
Short-lived token instead of a key file.
gcs.oauth2.token-expires-at
Must be a number if set. An empty value crashes the catalog.
gcs.user-project
Requester-pays buckets.
Read tables another catalog wrote
A REST catalog keeps each table's current state in its own database, so pointing a new catalog at an existing bucket does not show those tables. Register each table instead. This only adds a row to the local catalog; nothing is written to the bucket.
Use the newest metadata file. An older one shows an old snapshot. Do not read the data/ Parquet files directly: they include files from every past snapshot.
Dependencies
Place JVM dependencies under dependencies/java/ and Python wheels under dependencies/python/; the stack loads them for driver and executors automatically. Nested folders under dependencies/java/ are not picked up. If you have multiple JAR folders, set spark.driver.extraClasspath (and spark.executor.extraClasspath for executor-side UDFs). Register a Java UDF by fully qualified class name to call it in SQL:
When using ABFSS, add the Azure Iceberg bundle JAR under dependencies/java/, or runtime fails with NoClassDefFoundError: DataLakeStorageException.
For SEED models on DataOS, download vendor Derby before deployment and commit the JAR so git-sync delivers it to the Spark pod:
Download derby-10.14.2.0.jar from Maven Central.
Save it into your Vulcan project at
dependencies/java/derby-10.14.2.0.jar(create thedependencies/java/folder if it doesn't exist).Commit that file so git-sync ships it to the Spark pod on deploy.
Production deployment
Apply resources in this order. Each step depends on the previous:
Lakehouse resource
Object-storage-backed Iceberg storage
S3 or ABFSS configuration
Lakehouse depot
Exposes the lakehouse to Spark and Vulcan
name, spec.type, storage credentials, purpose: rw
secret-git-sync.yaml
Repo credentials for git-sync
GITSYNC_USERNAME, GITSYNC_PASSWORD
config.yaml
Project config, trimmed to production fields
engine: spark, depot gateway, model_defaults
Vulcan resource (workspace-deploy.yaml)
Spark driver + executors + API + schedule
spec.engine: spark, spec.compute, spec.repo, spec.depots, spec.workflow.resources, spec.api, spec.workflow.sparkConf, spec.sparkConf
The vulcan-spark stack generates the Iceberg REST catalog config from the lakehouse depot, so the depot must exist before vulcan apply. Set state_connection to type: postgres for production, or omit it entirely if the platform provisions it automatically for your tenant. Depot purposes are rw (Vulcan workflow), scan (metadata scanner), and query (consumer read-only).
Generate the deploy manifest from the project root:
Before editing config.yaml for deployment, keep a copy of the working local version for reference:
Then edit config.yaml in place. Drop the local-only keys: state_connection, object_store, catalog, and the spark.master / spark.driver.* / spark.executor.* / spark.sql.catalog.* / spark.sql.defaultCatalog block. Then switch to a DataOS depot gateway:
Fill the generated placeholders in workspace-deploy.yaml:
type
vulcan
name
Lowercase DataOS resource name, usually matching config.yaml name
spec.runAsUser
DataOS user/service account to run the workflow
spec.compute
Spark-capable DataOS compute pool
spec.engine
spark
spec.repo.url
Git repository URL containing the project
spec.repo.syncFlags
Branch/tag ref, e.g. --ref=<branch>
spec.repo.baseDir
Path from repo root to the Vulcan project
spec.repo.secretId
Git-sync secret, if the repo is private
spec.depots
dataos://<lakehouse-depot-name>?purpose=rw
spec.workflow.schedule
Cron, endOn, timezone, and concurrencyPolicy
spec.workflow.resources
Spark driver/executor CPU and memory sizing
spec.api
API replicas and request/limit resources
Apply once config.yaml, dependencies, and workspace-deploy.yaml are committed and pushed:
The vulcan-spark stack can only generate catalog config from a depot that already exists; the Vulcan resource can only sync code if the Git secret is available. A depot name mismatch in spec.depots[] passes apply but fails at every workflow run. This is why the apply order above matters.
Next steps
Choose the appropriate model kind in Materialization behaviour.
Last updated
Was this helpful?