For the complete documentation index, see llms.txt. This page is also available as Markdown.

Deployment steps

Deploy a Vulcan Data Product into a DataOS environment: prerequisites, the deployment: section in config.yaml, linting, deploying, and checking status.

Deploy Vulcan data products into a DataOS environment by following these steps in order: check prerequisites, add a deployment: section to config.yaml, lint the generated manifest, deploy it, then check status.


Step 1: Check prerequisites

Confirm the following resources already exist in your DataOS environment before deploying.

1.1 DataOS CLI

# Verify CLI installation
dataos-ctl version

# Login to your DataOS instance
dataos-ctl login

1.2 Depot (data source connection)

dataos-ctl resource -t depot get -a

The depot needs read/write permissions for your data warehouse schema.

1.3 Compute resource

dataos-ctl resource -t compute get -a

Example compute resources: cyclone-compute (general purpose), minerva-compute (query engine), or a custom compute cluster.

1.4 Git-sync secret (private repositories only)

Needed only when your Vulcan project lives in a private Git repository. A public repository syncs without a secret, so skip this step and leave repo.secret_id out of the deployment: section.

Once the CLI, depot, and compute are ready (plus the Git-sync secret, for a private repository), move on to configuration.


Step 2: Add the deployment: section to config.yaml

A Vulcan deployment is driven entirely by config.yaml. There is no separate domain-resource.yaml to hand-edit. The deployment: section supplies everything vulcan create_deploy_yaml needs to render a DataOS Vulcan resource manifest, deterministically and with no <placeholder> values left to fill in.

For the rest of config.yaml (project identity, users, gateways, model_defaults, notifications), see Configurations. Business-facing usage guidance belongs in a separate usage.yaml, and the data-sharing agreement text in agreement.md.

Some fields are computed automatically and are never written under deployment::

Field
Source

name, description, tags

From config.yaml's top-level name, description, tags

engine

From the default gateway connection type, or model_defaults.dialect

depots (default)

From the depot address on a depot-typed gateway connection

workflow.schedule.crons

Union of every model's effective cron (explicit cron, or model_defaults.cron when a model doesn't set one), deduplicated

engine and the derived crons can't be overridden. depots is derived but may be overridden explicitly under deployment.depots.

A couple of fields are fixed and not configurable: version is always v1alpha, and workflow.schedule.concurrency_policy is always Forbid.

Everything else lives under deployment:. Only repo.url, repo.base_dir, compute, and run_as_user are required; everything else has a default.

spark_conf and trino are engine-gated

spark_conf is only valid when the resolved engine is spark; trino only when it's trino. Setting either under the wrong engine fails validation at create_deploy_yaml time, not at DataOS apply time. Depot cardinality is also engine-dependent: spark and trino allow one or more depots, every other engine requires exactly one.

Tenant can be set in config.yaml or via the environment

tenant is a valid config.yaml key (vulcan init writes it directly), and domain becomes required once tenant is set. Alternatively, set DATAOS_TENANT_ID in your shell or .env instead of putting tenant in config.yaml.

Once config.yaml is committed and pushed to your Git repository, move on to linting.


Step 3: Lint the deploy manifest

Before deploying, render the manifest and validate it with -l (lint / dry-run) instead of applying it. This catches a missing required field, an unsupported engine, or an unknown key under deployment: before anything reaches DataOS:

Fix any errors it reports in config.yaml and re-run this command until it passes cleanly.


Step 4: Deploy

Once the lint check passes, render the manifest and apply it for real:


Step 5: Check status

-r auto-refreshes the result. For the runtime status of each component (API, plan, and run), use dataos-ctl resource get runtime -t vulcan -n <name-of-dp>; see Monitor & delete.


Step 6: Update a deployed Data Product

Update a deployed Data Product in place. Don't delete the vulcan Resource and redeploy it to pick up a change: deleting also removes the API container groups, so every app, dashboard, and API client that queries the Data Product loses access until the redeploy finishes.

In-place updates need dataos-ctl 3.24.0 or later. Check your version with dataos-ctl version.

First, commit and push the change to the Git ref the Data Product syncs from. Then pick the path that matches what changed:

Use this when the change touches config.yaml, including the deployment: block, or when you aren't sure what changed. Render the manifest again and apply it with --re-run:

apply updates the existing Resource in place, and --re-run runs it again straight after, so the change is picked up without waiting for the next scheduled run.

Use this when only model files changed on the synced Git ref and the deploy manifest is unchanged. Re-run the plan workflow:

The plan workflow runs vulcan plan --auto-apply again against the latest commit on the synced ref.

Then check the runtime:

The vulcan-plan container group should reach completed, and the vulcan-api container groups should stay running throughout. If the plan fails, read its logs.


Verification

Confirm the deployment actually worked.

Verify models in data warehouse

Access Vulcan API


Last updated

Was this helpful?