For the complete documentation index, see llms.txt. This page is also available as Markdown.

Connect to Spark

Configure the Vulcan gateway connection to Spark and load the JVM and Python dependencies the driver and executors need.

With roles and platform resources in place from Permissions and grants, this is the connect stage: configure the gateway connection Vulcan uses to reach Spark, then load the dependencies the driver and executors need before you write a model.

Connection options

Set these under gateways.<name>.connection. Spark is single-catalog: all models must be defined under one catalog.

Option
Description
Type
Required

type

Engine type name. Must be spark.

string

Yes

config_dir

Value to set for SPARK_CONFIG_DIR.

string

No

catalog

The catalog to use when issuing commands.

string

No

config

Key/value pairs for the Spark configuration, including the Iceberg catalog setup.

dict

No

If catalog is not set, the default depends on the Spark version: Spark 3.4 and above determine it at runtime; earlier versions default to spark_catalog. Authentication is configuration-based: credentials come through the config parameter or the catalog (S3, HDFS).

In production, do not hand-author the catalog block. Attach the lakehouse depot in spec.depots[] and let the vulcan-spark stack generate the catalog config automatically. The local examples below are for Docker-only development.

Local development

Vulcan behaves the same on every engine: the same project, the same vulcan plan, vulcan run, and vulcan fetchdf. Spark is the exception. It is not a single server you connect to, so local development needs a small stack running next to your project:

Component
Role

Spark master and two workers

Run the queries Vulcan sends.

Iceberg REST catalog

Tracks your tables.

Object store

Holds table data: local MinIO, or AWS S3, Azure ABFSS, or Google GCS.

Postgres

Stores Vulcan state.

One Docker Compose file adds this stack to your project. Start against MinIO, which needs no cloud account. Moving to S3, ABFSS, or GCS later only changes a few lines in .env.

Choose how you run Vulcan

The stack always runs in Docker. Vulcan acts as the Spark driver and runs in one of two places:

Local (recommended)
Containerized

Host requirements

Docker, Java 17, Python 3.13

Docker only

Vulcan runs in

A Python virtual environment on your machine

The vulcan-cli container

Run commands as

vulcan plan

docker exec ldk-vulcan-cli vulcan plan

Extra setup

One hosts-file entry and your LAN IP

None

IDE and debugger

Work natively

Limited

Best for

Everyday development

Machines without Java or Python

Both modes use the same project, compose file, and .env. Where the steps differ, they are split into Local and Containerized tabs. Running the driver inside Linux (Containerized) avoids Windows Hadoop and winutils.exe issues.

On Windows, set HOME once per PowerShell session before running docker compose or vulcan. The compose file and config.yaml read it, and PowerShell does not set it by default: $env:HOME = $env:USERPROFILE. In Local mode, spark.driver.extraClassPath in config.yaml separates entries with :. Java on Windows expects ;.

Prerequisites

Both modes need:

  • Docker Desktop with at least 8 GB of memory and about 6 GB of free disk.

  • The Vulcan wheel, vulcan-0.228.1.32-py3-none-any.whl.

  • On Apple Silicon, the Spark image is amd64-only and runs under emulation. It is slower but works.

Check Java and Python, then install Vulcan in a virtual environment.

macOS/Linux:

Windows PowerShell:

  • Python must be 3.13, the same minor version as the Spark workers. Any other version fails with PYTHON_VERSION_MISMATCH.

  • [spark,postgres] installs PySpark (the driver) and the Postgres client (state). --use-deprecated=legacy-resolver avoids pip's ResolutionTooDeep error on this wheel.

Add the local Docker service names to your hosts file once, so your machine can reach the containers by name.

macOS/Linux:

Windows:

No Java, Python, or hosts-file entry is needed. Place the wheel in your project folder (see Create your Vulcan project). The vulcan-cli container installs it on first start.

Create your Vulcan project

If you already have a Spark Vulcan project, use it and skip to Add the Docker stack.

macOS/Linux:

Windows PowerShell:

Create the project folder and copy the wheel into it. Scaffold the project after the stack is up (see Start the stack and run Vulcan):

This works in a folder that already contains docker/, .env, and the wheel.

-t default creates three example models, seed data, and data quality checks, so you have something to run straight away.

Once you finish the next three sections, the project looks like this:

Add the Docker stack

Create a docker/ folder inside your project and save this file as docker/docker-compose.yml. Run every docker compose command from this folder.

Docker Compose: local stack (docker/docker-compose.yml)

Full YAML (click to expand)
Container
Host ports
Role

ldk-spark-master

7077, 8080

Spark master. UI at http://localhost:8080.

ldk-spark-worker-1/2

n/a

Spark executors.

ldk-iceberg-rest

8181

Iceberg REST catalog.

ldk-minio

9000, 9001

MinIO (local S3). Console at http://localhost:9001 (admin / password).

ldk-minio-init

n/a

Creates the warehouse bucket, then exits.

ldk-statestore

5431

Postgres for Vulcan state.

ldk-driver-jars

n/a

Copies the image's JARs to ~/.cache/vulcan/spark-jars/<version>/ for the driver, then exits.

ldk-vulcan-cli

4040

Containerized Vulcan. Spark driver UI on 4040.

  • Local mode: comment out the vulcan-cli service at the bottom of the file and the vulcan-cli-venv volume at the top, so docker compose up -d skips building it.

  • Driver JARs: the driver-jars service copies Spark's Iceberg, AWS, Azure, and GCS JARs out of the Spark image on every up, so Vulcan and the workers always load identical files.

  • MinIO image: the official minio/minio images are no longer published. pgsty/minio is a community rebuild of the same software.

  • Your own JARs: custom UDFs and JDBC drivers go in dependencies/java/. Both Vulcan and the workers pick them up.

Choose where your data lives

Four object stores are wired in. Pick one with .env in your project root:

Store
Use it for
Needs

MinIO

Local development, runs inside the stack

Nothing

AWS S3

Your S3 bucket

Bucket, region, access key, secret key

Azure ABFSS

Your ADLS Gen2 container

Storage account name and key

Google GCS

Your GCS bucket

Bucket, service-account key file

Save the file below as .env.example and commit it. Then copy it to .env (never commit .env) and link it into docker/.

macOS/Linux:

Windows PowerShell (run as Administrator, or enable Developer Mode, to create the symlink):

Environment file: object-store selection (.env.example)

Full file (click to expand)

Follow these rules for .env:

  • Keep exactly one block uncommented. MinIO is active by default. To switch, see Switch to S3, ABFSS, or GCS.

  • Vulcan reads .env from the project root by itself, and Docker Compose reads it through the docker/.env symlink. One file keeps both sides in agreement.

  • Do not leave values blank. A blank value silently falls back to the MinIO default.

  • Use a dedicated prefix in every cloud bucket. Two catalogs managing one prefix corrupt each other's view of the tables.

  • GCS also needs the service-account key at .secrets/gcs.json. Add .env and .secrets/ to your .gitignore.

Configure config.yaml

In config.yaml, replace the gateways: block that vulcan init created with the block below, and add the before_all: block after it.

Vulcan config: gateways and before_all (config.yaml)

Full YAML (click to expand)
Setting
Why

spark.master: spark://spark-master:7077

A container name. Works from your machine (through the hosts file) and inside vulcan-cli.

spark.driver.host

The address Spark workers use to call Vulcan back. Set through LOCAL_DEV_HOST.

spark.driver.extraClassPath

The JARs driver-jars copied out of the image, plus your dependencies/java/.

catalog: warehouse and the spark.sql.catalog.warehouse.* keys

Vulcan's tables live in the Iceberg REST catalog.

The # {% if env_var('LDK_STORE') %} part

The only store-specific part: S3/MinIO keys, ADLS keys, or the GCS key file. The Jinja tags sit in YAML comments, so editors still see valid YAML.

state_connection

Postgres from the stack. state_schema is set per store.

before_all: SELECT current_catalog()

Required. See the hint below.

state_connection points at the local statestore Postgres container. When you edit config.yaml for deployment, follow Production deployment for the state store.

Start the stack and run Vulcan

Start the stack from the docker/ folder:

Every container shows Up except ldk-minio-init and ldk-driver-jars. Those are one-off jobs and show Exited (0). http://localhost:8080 lists two workers. The first start pulls the images and takes a few minutes.

Return to the project root, set LOCAL_DEV_HOST to your machine's LAN IPv4 address, then run Vulcan.

macOS:

Linux: set the address with export LOCAL_DEV_HOST="$(hostname -I | awk '{print $1}')", then run the same vulcan commands.

Windows PowerShell:

On Windows, get the LAN IPv4 from ipconfig under your active Wi-Fi or Ethernet adapter.

The Spark workers run in containers and call Vulcan back on LOCAL_DEV_HOST. 127.0.0.1 would point at the container itself. The address changes after sleep, a network switch, or a VPN connection. If jobs hang, set it again. Always run vulcan from the project root.

Wait for the container to install the wheel (first start only), then run every command through it:

If you have not created the project yet, run docker exec ldk-vulcan-cli vulcan init spark -t default now, then update config.yaml as described in Configure config.yaml before you run vulcan plan.

To shorten the commands, set an alias.

macOS/Linux:

Windows PowerShell:

Validate your connection

Your setup works when:

  1. vulcan info ends with Data warehouse connection succeeded and State backend connection succeeded.

  2. vulcan plan --auto-apply finishes and every data quality check passes.

  3. vulcan fetchdf returns rows.

  4. The data files are in the object store. For MinIO, run the command below (it works in Bash and PowerShell), or browse http://localhost:9001.

vulcan run right after plan may report No models are ready to run. @daily models only run once a full day (UTC) has passed. This is expected.

Switch to S3, ABFSS, or GCS

Use the same project, compose file, and commands. From docker/:

Then run the same vulcan commands as in Start the stack and run Vulcan.

Check that the catalog picked up the new store. This is the step that fails silently:

Confirm the files landed with your provider's tools (aws s3 ls, az storage blob list, gcloud storage ls).

Run down -v on every switch. The catalog keeps its table list in memory, while Vulcan's state lives in Postgres. down -v resets both together, so the new store starts clean instead of with tables that point at the old one.

MinIO
AWS S3
ABFSS
GCS

FileIO

S3FileIO

S3FileIO

ADLSFileIO

GCSFileIO

Warehouse

s3://warehouse/

s3://<bucket>/<prefix>

abfss://<container>@<account>.dfs.core.windows.net/<prefix>

gs://<bucket>/<prefix>

Credentials

admin / password

Access key and secret key

Account name and key

Service-account JSON

Endpoint

http://minio-warehouse:9000

https://s3.<region>.amazonaws.com

n/a

n/a

Path-style access

true

false

n/a

n/a

For GCS, three processes touch the bucket: the catalog, Vulcan, and the Spark workers. All three read the key from .secrets/gcs.json. If the first write fails with StorageException: 401 Unauthorized, the key file was missing when the stack started. Add it, then run down -v and up -d.

Troubleshoot the local stack

Symptom
Cause and fix

Failed to discover Spark objects in …, with USE CATALOG or PARSE_SYNTAX_ERROR in .logs/

before_all is missing from config.yaml. See Configure config.yaml.

Invalid ADLS URI: s3://…, or any mismatch between the catalog and Spark

A store exported in your terminal overrode .env. Open a new terminal or clear the LDK_* variables (commands below), then run down -v, up -d, and plan again.

Jobs hang and workers never connect (Local)

LOCAL_DEV_HOST is unset or stale. Set it again. A firewall or VPN may also block the workers from reaching your machine; try Containerized.

PYTHON_VERSION_MISMATCH

Your virtual environment is not Python 3.13. See Prerequisites.

ClassNotFoundException: org.apache.iceberg.spark.SparkCatalog

Driver JARs are missing. Check docker logs ldk-driver-jars and confirm ~/.cache/vulcan/spark-jars/<version>/ is not empty.

InvalidClassException … local class incompatible

Vulcan and the workers load different JARs. Do not add spark.jars.packages; keep VULCAN_VERSION the same everywhere.

TABLE_OR_VIEW_NOT_FOUND on fetchdf

Check the table name. The example tables are vulcan_example.seed_model, full_model, and incremental_model.

port is already allocated

Another stack uses 8080, 9000, 5431, or 7077. Stop it, or change the host-side (left) port in the compose file.

ERROR: /workspace/vulcan-<version>-py3-none-any.whl not found (Containerized)

The wheel is not in your project folder.

pull access denied for minio/…

An old compose file. Use the one in Add the Docker stack.

Clear exported LDK_* variables from the current terminal:

macOS/Linux:

Windows PowerShell:

Everyday commands and debugging

Run these from the project root. In Containerized mode, prefix each with docker exec ldk-vulcan-cli.

Task
Command

Preview a change before applying

vulcan plan

Apply to prod

vulcan plan --auto-apply

Run scheduled models

vulcan run

Run unit tests

vulcan test

Query a table

vulcan fetchdf "select …"

Lint

vulcan lint

Look at
For

http://localhost:8080

Spark master: workers and applications.

http://localhost:4040

Spark jobs and SQL plans of the running vulcan command.

.logs/

Vulcan logs.

docker logs ldk-spark-worker-1

Worker-side errors. Cloud auth failures show up here.

docker logs ldk-iceberg-rest

Catalog errors, such as credentials or the wrong FileIO.

http://localhost:9001

MinIO console.

In Local mode, Vulcan is a normal Python process. Point your IDE at the virtual environment's Python and the debugger works as usual.

Tear down

From docker/:

Data in cloud buckets is never deleted. Remove your <prefix> there yourself. ~/.cache/vulcan/spark-jars/ is safe to delete; the next up recreates it.

Upgrade Vulcan

  1. Get the new wheel. In Containerized mode, place it in the project folder.

  2. Set VULCAN_VERSION to the new version. The stack picks the matching Spark image and driver JARs from it.

    • macOS/Linux: export VULCAN_VERSION=<new-version>

    • Windows PowerShell: $env:VULCAN_VERSION="<new-version>"

  3. In Local mode, install the new wheel into your virtual environment. If the new Spark image uses a different Python minor version, recreate the virtual environment with that version. Check it with docker run --rm --platform linux/amd64 tmdcio/vulcan-spark-base:<new-version> python3 --version.

  4. Run docker compose up -d. Containerized Vulcan reinstalls the wheel by itself.

Object-store keys

The compose file wires only the keys the four stores need. For more, such as server-side encryption, temporary credentials, or SAS tokens, add each key in two places: on the catalog in docker/docker-compose.yml, and on Spark in config.yaml. The Spark workers write the data files themselves, so they need the keys too.

Where
Spelling
Example

Catalog (environment:)

CATALOG_ + upper case, . → _, - → __

CATALOG_S3_ACCESS__KEY__ID

Spark (config.yaml)

spark.sql.catalog.warehouse. + the key

spark.sql.catalog.warehouse.s3.access-key-id

S3 and MinIO (S3FileIO)

Key
Notes

s3.endpoint

MinIO: http://minio-warehouse:9000. AWS: https://s3.<region>.amazonaws.com.

s3.access-key-id, s3.secret-access-key

Static credentials.

s3.session-token

Temporary or STS credentials.

s3.path-style-access

true for MinIO, false for AWS.

s3.sse.type, s3.sse.key

Server-side encryption.

s3.acl, s3.write.storage-class

Canned ACL and storage class.

Region is not an s3.* key. Set AWS_REGION on the catalog and client.region on Spark.

ABFSS (ADLSFileIO)

Key
Notes

adls.auth.shared-key.account.name, .account.key

Both are required together.

adls.sas-token.<account>

SAS token instead of the account key.

adls.connection-string.<account>

Connection string.

GCS (GCSFileIO)

Key
Notes

GOOGLE_APPLICATION_CREDENTIALS

Path to the service-account JSON. Mount the file; do not paste it.

gcs.project-id

Google Cloud project ID.

gcs.oauth2.token

Short-lived token instead of a key file.

gcs.oauth2.token-expires-at

Must be a number if set. An empty value crashes the catalog.

gcs.user-project

Requester-pays buckets.

Read tables another catalog wrote

A REST catalog keeps each table's current state in its own database, so pointing a new catalog at an existing bucket does not show those tables. Register each table instead. This only adds a row to the local catalog; nothing is written to the bucket.

Use the newest metadata file. An older one shows an old snapshot. Do not read the data/ Parquet files directly: they include files from every past snapshot.

  • In Local mode, a firewall or VPN can stop the Spark workers from reaching your machine. Run Vulcan in Containerized mode instead.

  • The Spark image is amd64-only. On Apple Silicon it runs under emulation.

  • Each stack runs one object store at a time.

Dependencies

Place JVM dependencies under dependencies/java/ and Python wheels under dependencies/python/; the stack loads them for driver and executors automatically. Nested folders under dependencies/java/ are not picked up. If you have multiple JAR folders, set spark.driver.extraClasspath (and spark.executor.extraClasspath for executor-side UDFs). Register a Java UDF by fully qualified class name to call it in SQL:

When using ABFSS, add the Azure Iceberg bundle JAR under dependencies/java/, or runtime fails with NoClassDefFoundError: DataLakeStorageException.

For SEED models on DataOS, download vendor Derby before deployment and commit the JAR so git-sync delivers it to the Spark pod:

  1. Download derby-10.14.2.0.jar from Maven Central.

  2. Save it into your Vulcan project at dependencies/java/derby-10.14.2.0.jar (create the dependencies/java/ folder if it doesn't exist).

  3. Commit that file so git-sync ships it to the Spark pod on deploy.

Production deployment

Apply resources in this order. Each step depends on the previous:

Manifest
Purpose
Key fields

Lakehouse resource

Object-storage-backed Iceberg storage

S3 or ABFSS configuration

Lakehouse depot

Exposes the lakehouse to Spark and Vulcan

name, spec.type, storage credentials, purpose: rw

secret-git-sync.yaml

Repo credentials for git-sync

GITSYNC_USERNAME, GITSYNC_PASSWORD

config.yaml

Project config, trimmed to production fields

engine: spark, depot gateway, model_defaults

Vulcan resource (workspace-deploy.yaml)

Spark driver + executors + API + schedule

spec.engine: spark, spec.compute, spec.repo, spec.depots, spec.workflow.resources, spec.api, spec.workflow.sparkConf, spec.sparkConf

The vulcan-spark stack generates the Iceberg REST catalog config from the lakehouse depot, so the depot must exist before vulcan apply. Set state_connection to type: postgres for production, or omit it entirely if the platform provisions it automatically for your tenant. Depot purposes are rw (Vulcan workflow), scan (metadata scanner), and query (consumer read-only).

Generate the deploy manifest from the project root:

Before editing config.yaml for deployment, keep a copy of the working local version for reference:

Then edit config.yaml in place. Drop the local-only keys: state_connection, object_store, catalog, and the spark.master / spark.driver.* / spark.executor.* / spark.sql.catalog.* / spark.sql.defaultCatalog block. Then switch to a DataOS depot gateway:

Fill the generated placeholders in workspace-deploy.yaml:

Field
What to set

type

vulcan

name

Lowercase DataOS resource name, usually matching config.yaml name

spec.runAsUser

DataOS user/service account to run the workflow

spec.compute

Spark-capable DataOS compute pool

spec.engine

spark

spec.repo.url

Git repository URL containing the project

spec.repo.syncFlags

Branch/tag ref, e.g. --ref=<branch>

spec.repo.baseDir

Path from repo root to the Vulcan project

spec.repo.secretId

Git-sync secret, if the repo is private

spec.depots

dataos://<lakehouse-depot-name>?purpose=rw

spec.workflow.schedule

Cron, endOn, timezone, and concurrencyPolicy

spec.workflow.resources

Spark driver/executor CPU and memory sizing

spec.api

API replicas and request/limit resources

Apply once config.yaml, dependencies, and workspace-deploy.yaml are committed and pushed:

The vulcan-spark stack can only generate catalog config from a depot that already exists; the Vulcan resource can only sync code if the Git secret is available. A depot name mismatch in spec.depots[] passes apply but fails at every workflow run. This is why the apply order above matters.

Next steps

Choose the appropriate model kind in Materialization behaviour.

Last updated

Was this helpful?