> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/resources/vulcan/deployment.md).

# Deployment steps

Deploy a Data Product into a DataOS environment: prerequisites, the deployment: section in config.yaml, linting, deploying, and checking status.

Deploy Data Products into a DataOS environment by following these steps in order: check prerequisites, add a `deployment:` section to `config.yaml`, lint the generated manifest, deploy it, then check status.

***

## Step 1: Check prerequisites

Confirm the following resources already exist in your DataOS environment before deploying.

### 1.1 DataOS CLI

```bash
# Verify CLI installation
dataos-ctl version

# Login to your DataOS instance
dataos-ctl login
```

### 1.2 Depot (data source connection)

```bash
dataos-ctl resource -t depot get -a
```

{% hint style="info" %}
The depot needs read/write permissions for your data warehouse schema.
{% endhint %}

### 1.3 Compute resource

```bash
dataos-ctl resource -t compute get -a
```

**Example compute resources:** `cyclone-compute` (general purpose), `minerva-compute` (query engine), or a custom compute cluster.

### 1.4 Git-sync secret (private repositories only)

Needed only when your Vulcan project lives in a private Git repository. A public repository syncs without a secret, so skip this step and leave `repo.secret_id` out of the `deployment:` section.

```bash
dataos-ctl resource apply -f git-sync-secret.yaml
```

```yaml
name: git-sync
version: v2alpha
type: secret
workspace: system
layer: user
description: "Secret for git-sync authentication (Bitbucket)"
secret:
  type: key-value
  data:
    GITSYNC_USERNAME: "<your-git-username>"
    GITSYNC_PASSWORD: "<your-git-token-or-password>"
```

{% hint style="warning" %}
Replace `GITSYNC_USERNAME` and `GITSYNC_PASSWORD` with your actual Git repository credentials or access tokens.
{% endhint %}

Once the CLI, depot, and compute are ready (plus the Git-sync secret, for a private repository), move on to configuration.

***

## Step 2: Add the `deployment:` section to `config.yaml`

A Vulcan deployment is driven entirely by `config.yaml`. There is no separate `domain-resource.yaml` to hand-edit. The `deployment:` section supplies everything `vulcan create_deploy_yaml` needs to render a DataOS Vulcan resource manifest, deterministically and with no `<placeholder>` values left to fill in.

For the rest of `config.yaml` (project identity, `users`, `gateways`, `model_defaults`, notifications), see [Configurations](https://v2.dataos.info/references/resources/vulcan/configurations.md). Business-facing usage guidance belongs in a separate [usage.yaml](https://v2.dataos.info/references/resources/vulcan/configurations/usage.md), and the data-sharing agreement text in [agreement.md](https://v2.dataos.info/references/resources/vulcan/configurations/agreement.md).

{% tabs %}
{% tab title="What's derived vs. set" %}
Some fields are computed automatically and are never written under `deployment:`:

| Field                         | Source                                                                                                                       |
| ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| `name`, `description`, `tags` | From `config.yaml`'s top-level `name`, `description`, `tags`                                                                 |
| `engine`                      | From the default gateway connection type, or `model_defaults.dialect`                                                        |
| `depots` (default)            | From the depot address on a depot-typed gateway connection                                                                   |
| `workflow.schedule.crons`     | Union of every model's effective cron (explicit `cron`, or `model_defaults.cron` when a model doesn't set one), deduplicated |

`engine` and the derived `crons` can't be overridden. `depots` is derived but may be overridden explicitly under `deployment.depots`.

A couple of fields are fixed and not configurable: `version` is always `v1alpha`, and `workflow.schedule.concurrency_policy` is always `Forbid`.

Everything else lives under `deployment:`. Only `repo.url`, `repo.base_dir`, `compute`, and `run_as_user` are required; everything else has a default.
{% endtab %}

{% tab title="Minimal" %}

```yaml
deployment:
  repo:
    url: git@bitbucket.org:org/repo.git
    base_dir: repo/vulcan-project
  compute: general-purpose-shared
  run_as_user: svc-user
```

{% endtab %}

{% tab title="Complete" %}

```yaml
deployment:
  type: vulcan                      # default vulcan; override for a custom resource type

  # ── repo (url + base_dir required) ───────────────────────────────
  repo:
    url:      <string>              # required: git remote
    base_dir:  <string>              # required: project path inside the repo
    secret_id: <tenant>:<name>       # optional; only needed for private repos (DataOS secret ref)
    sync_flags:                      # optional, any git-sync flag, verbatim
      - --ref=main                  # default when sync_flags omitted
      - --submodules=off            # default when sync_flags omitted

  # ── spec-level ──────────────────────────────────────────────────
  compute:   general-purpose-shared # required
  run_as_user: <string>               # required
  state_store_connection:  <name>     # advanced; omit → vulcan-state-connection
  object_store_connection: <name>     # advanced; omit → vulcan-object-store-connection

  # ── depots (override; else derived from config.yaml depot conn) ──
  depots:
    - dataos://lakehouse?purpose=rw

  # ── schedule (crons are derived from models) ────────────────────
  schedule:
    end_on:    <ISO-8601>            # optional; omit if none
    timezone: UTC                   # default UTC

  # ── workflow (service-scoped: knobs + resources + commands) ─────
  workflow:
    log_level: INFO                  # default INFO
    resources:
      requests: { cpu: 200m,  memory: 512Mi }
      limits:   { cpu: 1000m, memory: 1Gi }
    plan: { command: [vulcan], arguments: [plan, --auto-apply] }
    run:  { command: [vulcan], arguments: [run] }

  # ── api (service-scoped) ────────────────────────────────────────
  api:
    replicas: 1                     # default 1
    log_level: INFO                  # default INFO
    resources:
      requests: { cpu: 200m,  memory: 512Mi }
      limits:   { cpu: 1000m, memory: 1Gi }

  # ── use: secret / env projection, optional ─────────────────────
  use:
    secrets:
      - { id: <tenant>:<name>, context_alias: <alias> }
    env_vars:
      - { key: <ENV_NAME>, template: "{% raw %}{{ secrets[<alias>].<KEY> | base64_decode }}{% endraw %}" }

  # ── spark_conf, only when engine == spark ───────────────────────
  spark_conf:
    spark.sql.shuffle.partitions: "16"
    spark.sql.adaptive.coalescePartitions.enabled: "true"

  # ── trino, only when engine == trino ───────────────────────────
  trino:
    catalog:
      config: [ "<tenant>:<catalog-secret>" ]   # DataOS secret refs; each secret name becomes a Trino catalog
    coordinator:
      resource: { request: { cpu: "1", memory: 2Gi }, limit: { cpu: "2", memory: 4Gi } }
    workers:
      replicas: 3
      resource: { request: { cpu: "2", memory: 4Gi } }
      autoscale: { min_replicas: 2, max_replicas: 6, cpu_target: 70 }
```

{% hint style="warning" %}
**Use plural `resources`, `requests`, and `limits`**

Set `workflow` and `api` resources with the plural keys shown above. The singular form still loads, but it's deprecated: `vulcan create_deploy_yaml` prints a warning for it, and support will be removed in a future release.

```yaml
# Deprecated: still accepted, with a warning
workflow:
  resource:
    request: { cpu: 200m }
    limit:   { cpu: 1000m }
```

{% endhint %}

{% hint style="info" %}
**`spark_conf` and `trino` are engine-gated**

`spark_conf` is only valid when the resolved `engine` is `spark`; `trino` only when it's `trino`. Setting either under the wrong engine fails validation at `create_deploy_yaml` time, not at DataOS apply time. Depot cardinality is also engine-dependent: `spark` and `trino` allow one or more depots, every other engine requires exactly one.
{% endhint %}

{% hint style="info" %}
**Tenant can be set in config.yaml or via the environment**

`tenant` is a valid `config.yaml` key (`vulcan init` writes it directly), and `domain` becomes required once `tenant` is set. Alternatively, set `DATAOS_TENANT_ID` in your shell or `.env` instead of putting `tenant` in `config.yaml`.
{% endhint %}
{% endtab %}
{% endtabs %}

Once `config.yaml` is committed and pushed to your Git repository, move on to linting.

***

## Step 3: Lint the deploy manifest

Before deploying, render the manifest and validate it with `-l` (lint / dry-run) instead of applying it. This catches a missing required field, an unsupported `engine`, or an unknown key under `deployment:` before anything reaches DataOS:

```bash
vulcan create_deploy_yaml -o - | dataos-ctl resource apply --in -l
```

Fix any errors it reports in `config.yaml` and re-run this command until it passes cleanly.

***

## Step 4: Deploy

Once the lint check passes, render the manifest and apply it for real:

```bash
vulcan create_deploy_yaml -o - | dataos-ctl resource apply --in
```

***

## Step 5: Check status

```bash
dataos-ctl resource -t vulcan -n <name-of-dp> get -r
```

`-r` auto-refreshes the result. For the runtime status of each component (API, plan, and run), use `dataos-ctl resource get runtime -t vulcan -n <name-of-dp>`; see [Monitor & delete](https://v2.dataos.info/references/resources/vulcan/monitor-and-delete.md).

***

## Step 6: Update a deployed Data Product

Update a deployed Data Product in place. Don't delete the `vulcan` Resource and redeploy it to pick up a change: deleting also removes the API container groups, so every app, dashboard, and API client that queries the Data Product loses access until the redeploy finishes.

{% hint style="info" %}
In-place updates need `dataos-ctl` 3.24.0 or later. Check your version with `dataos-ctl version`.
{% endhint %}

First, commit and push the change to the Git ref the Data Product syncs from. Then pick the path that matches what changed:

{% tabs %}
{% tab title="config.yaml changed" %}
Use this when the change touches `config.yaml`, including the `deployment:` block, or when you aren't sure what changed. Render the manifest again and apply it with `--re-run`:

```bash
vulcan create_deploy_yaml -o - | dataos-ctl resource apply --in --re-run
```

`apply` updates the existing Resource in place, and `--re-run` runs it again straight after, so the change is picked up without waiting for the next scheduled run.
{% endtab %}

{% tab title="Only models changed" %}
Use this when only model files changed on the synced Git ref and the deploy manifest is unchanged. Re-run the plan workflow:

```bash
dataos-ctl resource runtime re-run -t vulcan -n <name-of-dp> -v v1alpha --sub-resource-id vulcan-plan
```

The plan workflow runs `vulcan plan --auto-apply` again against the latest commit on the synced ref.
{% endtab %}
{% endtabs %}

Then check the runtime:

```bash
dataos-ctl resource get runtime -t vulcan -n <name-of-dp>
```

The `vulcan-plan` container group should reach `completed`, and the `vulcan-api` container groups should stay `running` throughout. If the plan fails, [read its logs](https://v2.dataos.info/references/resources/vulcan/monitor-and-delete.md#step-4-read-logs).

***

## Verification

Confirm the deployment actually worked.

### Verify models in data warehouse

```sql
-- For Snowflake
SHOW TABLES IN SCHEMA <database>.<schema>;

-- Check specific table
SELECT * FROM <database>.<schema>.<table-name> LIMIT 10;
```

### Access Vulcan API

```bash
# Test API (if exposed)
curl --location 'https://<fqdn>/vulcan/tenants/<tenant>/data-products/<tenant>-<data-product-name>/livez' \
  --header 'Authorization: Bearer <your-token>'
```

***

## Related pages

* [Configurations overview](https://v2.dataos.info/references/resources/vulcan/configurations.md): full `config.yaml` key reference.
* [Usage guidance](https://v2.dataos.info/references/resources/vulcan/configurations/usage.md)
* [Agreement](https://v2.dataos.info/references/resources/vulcan/configurations/agreement.md)
* [Model Defaults](https://v2.dataos.info/references/resources/vulcan/configurations/model-defaults.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/references/resources/vulcan/deployment.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
