Deployment steps
Deploy a Vulcan Data Product into a DataOS environment: prerequisites, the deployment: section in config.yaml, linting, deploying, and checking status.
Deploy Vulcan data products into a DataOS environment by following these steps in order: check prerequisites, add a deployment: section to config.yaml, lint the generated manifest, deploy it, then check status.
Step 1: Check prerequisites
Confirm the following resources already exist in your DataOS environment before deploying.
1.1 DataOS CLI
# Verify CLI installation
dataos-ctl version
# Login to your DataOS instance
dataos-ctl login1.2 Depot (data source connection)
dataos-ctl resource -t depot get -a1.3 Compute resource
dataos-ctl resource -t compute get -aExample compute resources: cyclone-compute (general purpose), minerva-compute (query engine), or a custom compute cluster.
1.4 Git-sync secret (private repositories only)
Needed only when your Vulcan project lives in a private Git repository. A public repository syncs without a secret, so skip this step and leave repo.secret_id out of the deployment: section.
Once the CLI, depot, and compute are ready (plus the Git-sync secret, for a private repository), move on to configuration.
Step 2: Add the deployment: section to config.yaml
A Vulcan deployment is driven entirely by config.yaml. There is no separate domain-resource.yaml to hand-edit. The deployment: section supplies everything vulcan create_deploy_yaml needs to render a DataOS Vulcan resource manifest, deterministically and with no <placeholder> values left to fill in.
For the rest of config.yaml (project identity, users, gateways, model_defaults, notifications), see Configurations. Business-facing usage guidance belongs in a separate usage.yaml, and the data-sharing agreement text in agreement.md.
Some fields are computed automatically and are never written under deployment::
name, description, tags
From config.yaml's top-level name, description, tags
engine
From the default gateway connection type, or model_defaults.dialect
depots (default)
From the depot address on a depot-typed gateway connection
workflow.schedule.crons
Union of every model's effective cron (explicit cron, or model_defaults.cron when a model doesn't set one), deduplicated
engine and the derived crons can't be overridden. depots is derived but may be overridden explicitly under deployment.depots.
A couple of fields are fixed and not configurable: version is always v1alpha, and workflow.schedule.concurrency_policy is always Forbid.
Everything else lives under deployment:. Only repo.url, repo.base_dir, compute, and run_as_user are required; everything else has a default.
Once config.yaml is committed and pushed to your Git repository, move on to linting.
Step 3: Lint the deploy manifest
Before deploying, render the manifest and validate it with -l (lint / dry-run) instead of applying it. This catches a missing required field, an unsupported engine, or an unknown key under deployment: before anything reaches DataOS:
Fix any errors it reports in config.yaml and re-run this command until it passes cleanly.
Step 4: Deploy
Once the lint check passes, render the manifest and apply it for real:
Step 5: Check status
-r auto-refreshes the result. For the runtime status of each component (API, plan, and run), use dataos-ctl resource get runtime -t vulcan -n <name-of-dp>; see Monitor & delete.
Step 6: Update a deployed Data Product
Update a deployed Data Product in place. Don't delete the vulcan Resource and redeploy it to pick up a change: deleting also removes the API container groups, so every app, dashboard, and API client that queries the Data Product loses access until the redeploy finishes.
First, commit and push the change to the Git ref the Data Product syncs from. Then pick the path that matches what changed:
Use this when the change touches config.yaml, including the deployment: block, or when you aren't sure what changed. Render the manifest again and apply it with --re-run:
apply updates the existing Resource in place, and --re-run runs it again straight after, so the change is picked up without waiting for the next scheduled run.
Use this when only model files changed on the synced Git ref and the deploy manifest is unchanged. Re-run the plan workflow:
The plan workflow runs vulcan plan --auto-apply again against the latest commit on the synced ref.
Then check the runtime:
The vulcan-plan container group should reach completed, and the vulcan-api container groups should stay running throughout. If the plan fails, read its logs.
Verification
Confirm the deployment actually worked.
Verify models in data warehouse
Access Vulcan API
Related pages
Configurations overview: full
config.yamlkey reference.
Last updated
Was this helpful?