> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/dataos-resources/vulcan/deployment-checklist.md).

# Deployment checklist

This page walks you through deploying a Vulcan data product as a managed service in a DataOS environment. You develop and validate locally with the [Local Development Kit (LDK)](/references/dataos-resources/vulcan/ldk.md), then deploy by pushing your project to Git and applying a DataOS resource that points at it.

## Prerequisites

Configure these resources in your DataOS environment before you deploy.

1. **DataOS command-line interface (CLI).** Install and log in:

   ```bash
   ds version
   ds login
   ```
2. **Depot.** A depot connects to your engine (Snowflake, BigQuery, Databricks, and others). List depots with `ds resource -t depot get -a`. The depot needs read and write permissions for your schema.
3. **Engine stack.** A stack defines the execution environment. Supported engines: `snowflake`, `bigquery`, `databricks`, `postgres`, `redshift`, `trino`, `mysql`, `mssql`. List stacks with `ds resource -t stack get -a`.
4. **Compute resource.** Provides the execution environment for Vulcan workflows (for example, `cyclone-compute`). List with `ds resource -t compute get -a`.
5. **Git-sync secret.** Grants access to the private Git repository that holds your project:

   ```yaml
   name: git-sync
   version: v2alpha
   type: secret
   workspace: system
   layer: user
   description: "Secret for git-sync authentication"
   secret:
     type: key-value
     data:
       GITSYNC_USERNAME: "<your-git-username>"
       GITSYNC_PASSWORD: "<your-git-token-or-password>"
   ```

   Apply it with `ds resource apply -f git-sync-secret.yaml`.

## Configuration files

A deployment needs two files at the project root.

### config.yaml

Holds Vulcan runtime configuration: project identity, model defaults, gateways, notifications, and environment behavior. Business-facing guidance belongs in `usage.yaml`. See [Configuration](/references/dataos-resources/vulcan/configuration.md) for the full reference. For a deployment, use a depot-backed gateway and list owners:

```yaml
name: user-engagement
display_name: User Engagement Analytics
description: User Engagement Analytics delivers insights into user engagement patterns.

discoverable: true
version: 0.1.0
alignment: consumer_aligned
vde: false   # set to true to enable Virtual Data Environments; not supported on spark/trino

tags:
  - gold
  - analytics

terms:
  - glossary.revenue

model_defaults:
  dialect: snowflake
  start: '2025-01-01'
  cron: '@daily'

gateways:
  default:
    connection:
      type: depot
      address: dataos://snowflakevulcan2   # reference to your depot

notification_targets:
  - type: console
    notify_on:
      - apply_failure
      - run_failure
      - dq_failure

users:
  - username: <owner-username>
    github_username: <owner-gh-username>
    email: <owner-email@example.com>
    type: OWNER
```

`type: OWNER` marks a data product owner; list one entry per owner. `github_username` drives PR and CI bot interactions. Set `DATAOS_TENANT_ID` in your shell or `.env`; it is not a YAML key.

### usage.yaml

The `usage.yaml` file lives alongside `config.yaml` and documents how the data product should be used:

```yaml
good_for:
  - Reporting on daily revenue trends
  - Feeding BI dashboards
not_for:
  - Real-time operational decisions
caveats:
  - Revenue excludes refunds processed after month-end close
references:
  - title: Revenue definition
    url: https://wiki.internal/revenue
    type: documentation
```

| Field        | Required | Description                                                   |
| ------------ | -------- | ------------------------------------------------------------- |
| `good_for`   | optional | Recommended use cases for this data product                   |
| `not_for`    | optional | Use cases explicitly excluded or unsupported                  |
| `caveats`    | optional | Freshness, exclusions, or interpretation warnings             |
| `references` | optional | Supporting links; each entry takes `title`, `url`, and `type` |

### domain-resource.yaml

Defines the DataOS resource that deploys Vulcan as a managed service. Generate a starter manifest with `vulcan create_deploy_yaml`, or write it by hand. It declares the execution config, the repository, depots, the workflow schedule, the `plan` and `run` commands, and the API service:

```yaml
version: v1alpha
type: vulcan
name: user-engagement
tags:
  - gold
  - analytics
spec:
  runAsUser: "<dataos-username>"
  compute: cyclone-compute
  engine: snowflake
  repo:
    url: https://bitbucket.org/rubik_/vulcan-examples
    syncFlags:
      - '--ref=main'
      - '--submodules=off'
    baseDir: vulcan-examples/customer-usecase/usdk
    secretId: engineering:git-sync
  depots:
    - dataos://snowflakevulcan2?purpose=rw
  workflow:
    schedule:
      crons:
        - '*/45 * * * *'
      endOn: '2027-01-01T00:00:00-00:00'
      timezone: 'US/Pacific'
      concurrencyPolicy: Forbid
    logLevel: INFO
    resource:
      request:
        cpu: "200m"
        memory: "512Mi"
      limit:
        cpu: "1000m"
        memory: "1Gi"
    plan:
      command: [vulcan]
      arguments: [--log-to-stdout, plan, --auto-apply]
    run:
      command: [vulcan]
      arguments: [--log-to-stdout, run]
  api:
    replicas: 1
    logLevel: INFO
    resource:
      request:
        cpu: "200m"
        memory: "512Mi"
      limit:
        cpu: "5000m"
        memory: "4Gi"
```

## Deployment steps

1. **Prepare the repository.** Confirm your project layout includes `config.yaml`, `domain-resource.yaml`, `models/` (with `dq/`, `semantics/`, `metrics/`), `plugins/`, `seeds/`, and `audits/`. Push the project to Git.
2. **Create secrets.** `ds resource apply -f git-sync-secret.yaml`.
3. **Verify prerequisites.** Confirm the depot, compute, and stack exist:

   ```bash
   ds resource -t depot get -n <depot-name> -a
   ds resource -t compute get -n <compute-name> -a
   ds resource -t stack get -a
   ```
4. **Deploy.** Generate the manifest if needed, then apply it:

   ```bash
   vulcan create_deploy_yaml
   ds resource apply -f domain-resource.yaml
   ```
5. **Monitor.** Check status and logs:

   ```bash
   ds resource -t vulcan -n <data-product-name> get
   ds resource -t vulcan -n <data-product-name> logs
   ```

## Runtime entries

Vulcan does not run as a single container. DataOS splits a deployment into three components, each with its own runtime and logs:

* **plan** handles deployment preparation (`vulcan plan --auto-apply`).
* **run** executes models on schedule (`vulcan run`).
* **api** serves queries and exposes endpoints (a long-running service).

You will see entries for all three, plus more, because each scheduled run creates a new workflow pod and the API pod has sidecar containers. Use the right log for your question:

| What you are investigating      | Component | Runtime entry pattern                                                  |
| ------------------------------- | --------- | ---------------------------------------------------------------------- |
| Model execution results         | run       | `*-r-execute`, `workflow...run...`                                     |
| Migration, planning, auto-apply | plan      | `*-mgrt-execute`, `*-plan-execute`                                     |
| API availability, query issues  | api       | `*-api-*` (`*-main` for API, `*-sc-1` for GraphQL, `*-sc-2` for MySQL) |

The API pod runs multiple containers, each with its own log suffix:

| Container          | Log suffix | Use it for                                      |
| ------------------ | ---------- | ----------------------------------------------- |
| Main API container | `*-main`   | Core API/service behavior                       |
| GraphQL sidecar    | `*-sc-1`   | GraphQL-related investigation                   |
| MySQL sidecar      | `*-sc-2`   | MySQL wire protocol or client connection issues |

Fetch logs from a specific component and container:

```bash
dataos-ctl resource -t Vulcan -n <resource-name> logs \
  --container-group <name>-run-execute -c main
```

Map the component you want to the `--container-group` and `-c` flags:

| What you need             | `--container-group`   | `-c`   |
| ------------------------- | --------------------- | ------ |
| Planning / migration logs | `<name>-plan-execute` | `main` |
| Model execution logs      | `<name>-run-execute`  | `main` |
| API service logs          | `<name>-api`          | `main` |
| GraphQL sidecar logs      | `<name>-api`          | `sc-1` |
| MySQL sidecar logs        | `<name>-api`          | `sc-2` |

{% hint style="info" %}
**Spark engines have driver and executor logs.** For a Spark gateway, the `run` and `plan` pods are the Spark driver; the work runs on executors on your Spark cluster. Connection and scheduler errors appear in the DataOS `*-run-execute` logs, while task failures inside a UDF, out-of-memory errors on a worker, and shuffle-fetch failures live in the Spark executor logs. Read those from the Spark master UI at `http://<spark-master>:8080`, then drill into the application and its executors. When a `vulcan run` fails with a multi-line Java stack trace, the top frames are driver-side (visible in `*-run-execute`), but the root cause often sits in an executor; don't re-run the DataOS pod when the answer is in the executor logs. A common cause of a Spark run failing at startup is a driver/cluster version skew, which surfaces as `java.io.InvalidClassException`.

The `sc-1` (GraphQL) and `sc-2` (MySQL) sidecars are part of the `api` pod, not `run`. Spark workloads don't add new container groups to DataOS; the driver still runs inside the existing `*-run-execute` container.
{% endhint %}

## Verification

Confirm the models exist in your engine, and test the API:

```sql
SHOW TABLES IN SCHEMA <database>.<schema>;
SELECT * FROM <database>.<schema>.<table-name> LIMIT 10;
```

```bash
curl --location 'https://<env-fqn>/<tenant>/vulcan/<data-product-name>/livez' \
  --header 'Authorization: Bearer <your-token>'
```

## Continuous integration and continuous delivery (CI/CD)

CI/CD support for Vulcan is in progress. In the meantime, the building blocks are already available: run `vulcan test` and `vulcan plan` in your pipeline to validate logic and classify changes before merge, and use git-based model selection (`vulcan plan --select-model "git:feature"`) to scope a plan to what changed on a branch. See the [model selection guide](/references/dataos-resources/vulcan/incremental-by-time/model-selection.md).

## Related docs

* [Configuration](/references/dataos-resources/vulcan/configuration.md): the full `config.yaml` reference.
* [Plan](/references/dataos-resources/vulcan/core-concepts/plan.md): how `vulcan plan --auto-apply` classifies and applies changes.
* [Observability](/references/dataos-resources/vulcan/observability.md): monitoring runs and the query path after deployment.
* [Troubleshooting](/references/dataos-resources/vulcan/troubleshooting.md): validation and request errors you may hit against the deployed API.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://v2.dataos.info/references/dataos-resources/vulcan/deployment-checklist.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.