> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/resources/vulcan/monitor-and-delete.md).

# Monitor & delete

Monitor a deployed Data Product with the DataOS CLI: read its status and runtime, find the failing container group, read its logs, rerun after a fix, and delete the deployed Resource when it's

A deployed Data Product is represented as a single `vulcan` Resource in DataOS. There's no separate `data-product` resource type to monitor or delete: the same Resource you deployed in [Deployment steps](https://v2.dataos.info/references/resources/vulcan/deployment.md) is the one you inspect at runtime, and the one you delete when the Data Product is no longer needed.

That one Resource runs several container groups at once: the API, a plan workflow, a scheduled run workflow, and one historical workflow for every time the schedule has fired.

Monitoring follows the same order every time. Start with the Resource status, because the status decides the next command. Open container logs only after you know which container group is actually failing.

{% hint style="info" %}
`resource` and `rs` are the same command: `dataos-ctl resource get ...` and `dataos-ctl rs ... get` are interchangeable. Most examples on this page use `orders-analytics`. The [worked example](#worked-example-a-failing-scheduled-run) uses `customer-360`, a Data Product whose scheduled runs are failing while its API stays up.
{% endhint %}

## Step 1: Read the Resource status

```bash
dataos-ctl resource get -t vulcan -n <name>
```

The same command with the short alias:

```bash
dataos-ctl rs -t vulcan -n <name> get
```

Read the `STATUS` column:

| Status      | What it means                                                                                | Next step                                                                                               |
| ----------- | -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| `active`    | DataOS built the Resource and is reconciling it. A failed model run can still show `active`. | [Read the runtime](#step-2-read-the-runtime).                                                           |
| `error`     | The Resource failed while DataOS was building or reconciling it.                             | [Inspect the build](#when-the-status-is-error-or-errorqt) with `-d -b`.                                 |
| `error(QT)` | The same failure, quarantined. DataOS has stopped retrying the build.                        | [Inspect the build](#when-the-status-is-error-or-errorqt) with `-d -b`, then apply again after the fix. |

`error` and `error(QT)` are build failures. The Vulcan process often never started, so container logs are empty or missing until the build details show a container that ran.

### When the status is `error` or `error(QT)`

```bash
dataos-ctl resource get -t vulcan -n <name> -d -b
```

* `-d` (`--details`) prints the spec, the runtime state, and the error message.
* `-b` (`--buildResourceDetails`) prints the build objects DataOS created for this Data Product (workflows, services, and the other Kubernetes resources) and which one failed.

Read the failed build object before you open logs. Typical causes are a missing depot, a compute name that doesn't exist, a secret that didn't project, or a workflow DataOS couldn't create.

Fix that cause in `config.yaml` (or in the depot, secret, or compute it points at), then apply the Resource again:

```bash
vulcan create_deploy_yaml -o - | dataos-ctl resource apply --in
```

{% hint style="warning" %}
A [rerun](#step-5-rerun-after-the-cause-is-fixed) leaves a quarantined build in `error(QT)`. Only an apply clears it.
{% endhint %}

## Step 2: Read the runtime

Use this when `STATUS` is `active`, and again after you've fixed an `error` or `error(QT)` and applied.

```bash
dataos-ctl resource get runtime -t vulcan -n <name>
```

The table lists every container group. `SUB RESOURCE ID` says which part of the Data Product the row belongs to. `PHASE` says what that part is doing.

| Sub resource  | Container group type                     | Container groups                                              | Healthy phase                                                  |
| ------------- | ---------------------------------------- | ------------------------------------------------------------- | -------------------------------------------------------------- |
| `vulcan-api`  | `cgroup-stateful` and `cgroup-stateless` | `<name>-api-0` (stateful) and `servicev2alpha...` (stateless) | `running`                                                      |
| `vulcan-plan` | `cgroup-workflow`                        | `<name>-plan-execute`                                         | `completed`                                                    |
| `vulcan-run`  | `cgroup-workflow`                        | `<name>-r-execute`                                            | `completed`, or `running` while a scheduled run is in progress |
| `<name>-r`    | `cgroup-workflow`                        | `workflowv2alpha<name>run-...`, one row per schedule fire     | `completed`                                                    |

For `orders-analytics`, those rows are `orders-analytics-api-0` and `orders-analytics-api-1`, `servicev2alphaordersanalyticsapi-6rc9-...`, `orders-analytics-plan-execute`, `orders-analytics-r-execute`, and the `workflowv2alphaordersanalyticsrun-...` history.

A Data Product deployed with `engine: trino` also gets container groups for its dedicated Trino coordinator and workers. Data Products on any other engine don't create Trino container groups.

### Read the phase

* `running` means the API is live and serving the Data Product.
* `completed` means a workflow, such as plan or run, finished successfully.
* `failing` means that workflow started and the Vulcan process inside it failed. That's the moment to [pick the container](#step-3-pick-the-container) and read its logs.
* A pending or stuck phase means the corresponding component needs investigation.

For a healthy Data Product, the API container groups are `running`, while the plan and run workflow container groups are `completed` once they finish.

### Read the schedule history

Several `workflowv2alpha...` rows are normal: each cron tick adds one. The number embedded in the name increases with time, so the largest number is the latest fire. Read the newest row whose phase is `failing`. Older `completed` rows are earlier schedules that succeeded.

The stable names don't change between fires:

* `<name>-plan-execute` is the plan (`vulcan plan --auto-apply`).
* `<name>-r-execute` is the current scheduled run (`vulcan run`).
* `<name>-api-0` and `servicev2alpha...` are the API.

To keep the table refreshing while you watch:

```bash
dataos-ctl resource get runtime -t vulcan -n <name> -r
```

For one container group, add details before you open logs. This `-d` is on `get runtime`; it's separate from the `-d -b` used for `error` and `error(QT)`.

```bash
dataos-ctl resource get runtime -t vulcan -n <name> \
  --container-group <name>-r-execute \
  -d
```

## Step 3: Pick the container

Each container group can include multiple containers. The runtime output identifies the role of each one:

* `primary`: the main workload container. On run and plan workflows it's `main`; on the API it's `<name>-api` or the `*-main` container.
* `initSideCar`: initialization container that prepares the pod (git sync, secrets, and setup). Named `init` or `*-ic`.
* `sideCar`: helper container. `wait` on workflow groups, and `*-sc-*` on the API service group.

For `orders-analytics`, the runtime contains:

* API containers: `orders-analytics-api`, `ordersanal-6rc9-main`, and `ordersanal-sc-2`.
* Init sidecars: `orders-analytics-plan-6rc9-ic`, `orders-analytics-r-6rc9-ic`, and `ordersanal-6rc9-ic`.
* Workflow helper sidecars: `wait`.

Match the failing row, then open the container that runs Vulcan:

| What you see                                                                         | Container group                                  | Container to open                                                                                                   |
| ------------------------------------------------------------------------------------ | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------- |
| The latest `workflowv2alpha...` row is `failing`, or `<name>-r-execute` is `failing` | That workflow group                              | `main`                                                                                                              |
| `<name>-plan-execute` is `failing`                                                   | `<name>-plan-execute`                            | `main`                                                                                                              |
| The API phase isn't `running`, or queries fail while the run rows are `completed`    | `<name>-api-0` and the `servicev2alpha...` group | The primary container (`<name>-api` or the `*-main` container), then the `*-sc-*` sidecar if the primary is healthy |
| A workflow is `failing` and `main` has no logs                                       | The same group                                   | The init sidecar (`init` or `*-ic`)                                                                                 |

The Vulcan plan and run output is in `main`, not in `wait`. Open the init sidecar when `main` never started; open `main` when the phase is `failing` and the pod did start.

## Step 4: Read logs

Copy `CONTAINER GROUP NAME` from the runtime table, and pass the container with `-c`:

```bash
dataos-ctl resource log -t vulcan -n <name> --container-group <container-group-name> -c <container>
```

For example, to check the plan execution logs for `orders-analytics`:

```bash
dataos-ctl resource log -t vulcan -n orders-analytics --container-group orders-analytics-plan-execute -c main
```

On the API service group, pass the container name from the runtime table. For example, to inspect the sidecar container `ordersanal-sc-2`:

```bash
dataos-ctl resource log -t vulcan -n orders-analytics \
  --container-group servicev2alphaordersanalyticsapi-6rc9-748dc448d8-gnhlk \
  -c ordersanal-sc-2
```

The same pattern applies to any other sidecar: copy the `CONTAINER GROUP NAME` from the runtime output, then pass the container name with `-c`.

### Pull the full error

The command returns the last 40 lines. A long traceback, the SQL, or the Vulcan error often starts above that window, so the output looks trimmed. Pull a longer tail with `-l` (`--tailLines`):

```bash
dataos-ctl resource log -t vulcan -n <name> \
  --container-group <container-group-name> \
  -c main \
  -l 10000
```

Use `-l 10000` whenever the default output stops before the error.

### Follow a running container

Use `-f` to stream logs from a container that is still `running`, such as the API:

```bash
dataos-ctl resource log -t vulcan -n orders-analytics \
  --container-group orders-analytics-api-0 \
  -c orders-analytics-api \
  -f
```

## Step 5: Rerun after the cause is fixed

Rerun when the runtime shows a `failing` plan or run and you've already fixed the cause. That fix might be a model change on the Git ref the Data Product syncs, a depot grant, or a secret. Rerunning while the cause is still there only adds another `failing` workflow row.

A `vulcan` Resource runs several nested workloads, so every `runtime` action needs `--sub-resource-id` to say which one to act on:

| Sub-resource ID | Workload               | Supported runtime actions                             |
| --------------- | ---------------------- | ----------------------------------------------------- |
| `vulcan-plan`   | Plan workflow          | `stop`, `pause`, `resume`, `re-run`, `run`            |
| `vulcan-run`    | Scheduled run workflow | `stop`, `pause`, `resume`, `re-run`, `run`, `suspend` |
| `vulcan-api`    | API service            | `stop`, `resume`, `re-run`                            |

Re-run the workload that failed:

```bash
# The plan failed
dataos-ctl resource runtime re-run -t vulcan -n <name> -v v1alpha --sub-resource-id vulcan-plan

# The scheduled run failed
dataos-ctl resource runtime re-run -t vulcan -n <name> -v v1alpha --sub-resource-id vulcan-run
```

{% hint style="info" %}
Vulcan Resources are version `v1alpha`. Pass `-v v1alpha` on every `runtime` command; without it, the command targets version `v1`.
{% endhint %}

{% hint style="warning" %}
**Runtime actions follow an order**

* `resume` works only on a stopped workload. On a running one it fails with an error such as `service is not stopped, cannot resume`.
* `re-run` doesn't work on a stopped workload. It fails with `resource is stopped, cannot re-run; resume first`. Run `resume`, then `re-run`.
* An action the workload doesn't support fails and lists the supported ones, for example `service does not support directive command 'pause'; supported: stop|resume|re-run`.
  {% endhint %}

Then [read the runtime](#step-2-read-the-runtime) again. A new `workflowv2alpha...` row should appear and reach `completed`, and the API rows should stay `running`.

### Pause and resume the schedule

Pause the schedule while you're still investigating a failure that the next cron tick would fire again:

```bash
dataos-ctl resource runtime pause -t vulcan -n <name> -v v1alpha --sub-resource-id vulcan-run
```

Resume it after the rerun succeeds:

```bash
dataos-ctl resource runtime resume -t vulcan -n <name> -v v1alpha --sub-resource-id vulcan-run
```

## Worked example: a failing scheduled run

`customer-360` is `active`, and `get runtime` shows this shape:

| Container group                                               | Sub resource     | Phase       |
| ------------------------------------------------------------- | ---------------- | ----------- |
| `workflowv2alpha...1791885780...`, `...6980...`, `...7580...` | `customer-360-r` | `completed` |
| `workflowv2alpha...1791888780...`, `...1791889380...`         | `customer-360-r` | `failing`   |
| `customer-360-api-0`                                          | `vulcan-api`     | `running`   |
| `servicev2alphacustomer360api-...`                            | `vulcan-api`     | `running`   |
| `customer-360-plan-execute`                                   | `vulcan-plan`    | `completed` |
| `customer-360-r-execute`                                      | `vulcan-run`     | `failing`   |

The API is up and the plan succeeded. The scheduled run is what failed, and it failed on the two newest fires (`1791888780`, then `1791889380`). Earlier fires completed.

Open `main` on the newest failing group, with a long tail:

```bash
dataos-ctl resource log -t vulcan -n customer-360 \
  --container-group workflowv2alphacustomer360run-hjv0-1791889380-customer-360-r-t-2619376192 \
  -c main \
  -l 10000
```

If `main` is empty, repeat the command with `-c init` (the init sidecar on that group is `customer-360-r-hjv0-ic`). Once the model or config fix is on the synced ref, rerun:

```bash
dataos-ctl resource runtime re-run -t vulcan -n customer-360 -v v1alpha --sub-resource-id vulcan-run
```

If `get` had shown `error` or `error(QT)` instead of `active`, the first command would have been `dataos-ctl resource get -t vulcan -n customer-360 -d -b`, and the next step an apply, before any log or rerun.

## Delete the Data Product

Deleting a Data Product means deleting its deployed `vulcan` Resource. This removes every runtime resource created for it, including the API, plan, and run container groups. There's no delete action in the Data Product Hub; deletion only happens through the DataOS CLI.

{% hint style="warning" %}
**Updating the Data Product? Don't delete it.**

Deleting takes the Data Product's API down for every consumer until you redeploy. To pick up a model or `config.yaml` change, update the Resource in place instead. See [Update a deployed Data Product](https://v2.dataos.info/references/resources/vulcan/deployment.md#step-6-update-a-deployed-data-product).
{% endhint %}

{% hint style="danger" %}
Deletion is irreversible. If other Resources depend on this Data Product, check dependencies before deleting it.
{% endhint %}

Check dependencies first:

```bash
dataos-ctl resource deps -t vulcan -n <name>
```

Then delete the Resource:

{% tabs %}
{% tab title="Command 1" %}

```bash
dataos-ctl resource delete -t vulcan -n ${{name}}
```

{% endtab %}

{% tab title="Command 2" %}

```bash
dataos-ctl resource delete -i "${{name}}|v1alpha|vulcan"
```

{% endtab %}

{% tab title="Command 3" %}

```bash
dataos-ctl resource delete -f ${{manifest-file-path}}
```

{% endtab %}
{% endtabs %}

**Example:**

```bash
dataos-ctl resource delete -t vulcan -n orders-analytics
```

Add `--cascade` to also delete every Resource that depends on this product, or `--force` to delete past a dependency check that would otherwise block it.

## What gets removed

When the `vulcan` Resource is deleted, DataOS tears down the runtime components it created:

* `vulcan-api` container groups stop serving the Data Product API.
* `vulcan-plan` workflow resources are removed.
* `vulcan-run` workflow resources are removed.
* Completed historical workflow container groups are cleaned up as part of the same teardown.

These are stopped and removed one after another rather than all at once, so deletion takes a few minutes instead of finishing instantly. The Data Product Hub updates automatically once the Resource is gone; there's no separate manual step to remove it from the Hub.

## Verify deletion

```bash
dataos-ctl resource get -t vulcan -n <name>
```

You can also check the runtime again:

```bash
dataos-ctl resource get runtime -t vulcan -n <name> -r
```

If the Resource is still returned, deletion is still in progress. Wait a few minutes and check again.

## Related pages

* [Deployment steps](https://v2.dataos.info/references/resources/vulcan/deployment.md): how the `plan`, `run`, and `api` components get created in the first place.
* [Troubleshooting](https://v2.dataos.info/references/resources/vulcan/troubleshooting.md): validation and request errors, if a command on this page fails.
* [Build → Monitor & delete](https://v2.dataos.info/build/productize/monitor-and-delete): the same page, filed under the Build/Productize task guide.
* [Resource commands](https://v2.dataos.info/references/interfaces/command-line-interface/resource): full flag reference for `get`, `get runtime`, `log`, `runtime`, `deps`, and `delete`.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/references/resources/vulcan/monitor-and-delete.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
