> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/dataos-resources/vulcan/observability.md).

# Observability

Once a data product is applied, you keep it fresh with `vulcan run` and watch how it behaves: which intervals processed, which checks passed, and how serving queries resolve. This page covers running and scheduling, run activity and notifications, and the semantic query path that serves your data product.

## Running and scheduling

Use `vulcan run` to refresh data after a plan is applied. `vulcan plan` decides what the data product should look like; once that shape is applied, `vulcan run` keeps the data filled with fresh data.

```bash
vulcan run
```

`vulcan run` does not redesign the data product and does not pick up new local code changes. It works only with the version already applied (see [State](/references/dataos-resources/vulcan/core-concepts/state.md)). The rule of thumb: changed the shape, use `plan`; refreshing data, use `run`.

| Question                                                 | Use `plan` | Use `run` |
| -------------------------------------------------------- | :--------: | :-------: |
| Did model SQL or Python change?                          |     Yes    |     No    |
| Did a metric, semantic model, check, or metadata change? |     Yes    |     No    |
| Do you need Vulcan to review what changed?               |     Yes    |     No    |
| Do you only need to process new data?                    |     No     |    Yes    |
| Do you want scheduled refreshes?                         |     No     |    Yes    |

### What a run does

When you run `vulcan run`, Vulcan reads the applied state, finds the models in it, checks which time intervals have already been processed, and decides which missing intervals are ready to run now. It then runs models in dependency order, runs audits (through assertions) and DQ checks, marks intervals as processed, and records what happened.

Audits and assertions run during model evaluation and can block a bad result before it lands. DQ checks run after successful execution and record quality posture; they do not block.

A run can process new daily, hourly, weekly, or monthly data, catch up after a failed or delayed run, run a selected model and its dependencies, respect schedules, and wait for signals. It does not apply local code changes, decide whether a change is breaking, or promote new versions; those belong to `vulcan plan`.

#### Batches

When many intervals are ready, Vulcan groups them into batches for efficiency. Each batch processes `batch_size` intervals (configured per model). With `batch_size: 2` and 3 ready intervals, Vulcan runs 2 batches: the first processes intervals 1–2, the second processes interval 3. Batches run sequentially per model and across all models in dependency order.

#### DAG execution order

Vulcan always runs models in dependency order — upstream before downstream. Example: if `raw_orders → cleaned_orders → customer_revenue`, then `raw_orders` runs first, `cleaned_orders` second, and `customer_revenue` last. A failure in `raw_orders` skips `cleaned_orders` and `customer_revenue` for that run.

### Schedules and signals

A schedule (the model's `cron`) tells Vulcan when a missing interval is allowed to run. By default `vulcan run` respects schedules. A [signal](/references/dataos-resources/vulcan/core-concepts/signals.md) is a readiness check: it answers whether the source file arrived, the upstream table updated, or the partition is available, and Vulcan skips an interval until all signals say it is ready. This prevents incomplete data from being processed too early.

If `vulcan run` finishes with nothing to do, that is not a failure. A run produces no work for any of these reasons:

1. All scheduled intervals have already been processed and are up to date.
2. No models are scheduled to run at this time (cron has not triggered).
3. All selected models are paused or disabled.
4. All ready models are blocked by unresolved signals.
5. The selected model filter (`--select`) matched no models.

{% hint style="info" %}
A run that processes nothing is not a failure — it means all data is current.
{% endhint %}

### Useful commands

```bash
vulcan run                                          # run the applied shape
vulcan run --start 2026-05-01 --end 2026-05-07      # a specific window
vulcan run --ignore-cron                            # ignore schedules during a manual catch-up — see warning below
vulcan run --select-model analytics.customer_revenue              # one model and its upstreams
vulcan run --select-model analytics.customer_revenue --no-auto-upstream   # only that model
vulcan run --skip-janitor                           # skip cleanup
vulcan run --exit-on-env-update 75                  # exit with this code if applied state changes mid-run
```

By default, `--select-model` includes upstream dependencies. Use `--no-auto-upstream` to run only the selected model, but be careful: if upstream data is missing, the model runs with incomplete inputs.

{% hint style="warning" %}
`--ignore-cron` makes Vulcan run a model even if its scheduled time has not arrived yet. Use carefully — it can process data earlier than the normal schedule, which may produce partial-interval results if upstream data is not yet complete.
{% endhint %}

## Run activity and notifications

At the end of a run, Vulcan records run activity. This activity helps you debug, monitor freshness, and explain what happened during a scheduled refresh. What is recorded:

* Start and end time of the run
* Which models ran and which were skipped
* Which intervals were processed for each model
* Which assertions and DQ checks passed or failed
* Any errors or exceptions
* Final outcome: success, failure, or nothing-to-do

Configure notifications in `config.yaml` so the right people hear about lifecycle events. Targets fire on the events you choose:

| Event                            | `notify_on` value                           |
| -------------------------------- | ------------------------------------------- |
| Plan apply start, end, failure   | `apply_start`, `apply_end`, `apply_failure` |
| Run start, end, failure          | `run_start`, `run_end`, `run_failure`       |
| Data quality start, end, failure | `dq_start`, `dq_end`, `dq_failure`          |

Data quality failure notifications reach a model's owner in `prod` when the model has an `owner`, runs at least one assertion, and the owner has a target whose `notify_on` includes `dq_failure`. See [Configuration](/references/dataos-resources/vulcan/configuration.md#notifications) for target setup. For deployed runtime logs (plan, run, and API components), see the [Deployment checklist](/references/dataos-resources/vulcan/deployment-checklist.md#runtime-entries).

## The semantic query path

Consumers query a data product through the semantic APIs. Semantic queries are asynchronous: you submit a query, get a statement ID back immediately, and poll for the result. This lets Vulcan deduplicate identical queries, serve from cache, and scale execution across workers.

```
POST query ──► statement ID ──► poll status ──► GET result
```

The lifecycle has six stages: submit, transpile (convert semantic references to engine SQL), cache check, execute on a worker, store the result as Parquet in object storage, and fetch. A request resolves through one of three strategies, recorded on the statement:

| Strategy        | What happened                                            |
| --------------- | -------------------------------------------------------- |
| `execute`       | New execution; SQL sent to the engine.                   |
| `from_cache`    | Cache hit; result returned from a previous execution.    |
| `await_primary` | Piggyback; linked to an identical query already running. |

Statement status moves `ACCEPTED → QUEUED → IN_PROGRESS → SUCCESS` (or `FAILED` or `CANCELLED`). Cache hits skip straight to `SUCCESS`.

### Caching and invalidation

Before executing, Vulcan hashes the generated SQL into a fingerprint and checks whether an identical query ran before, whether one is already in flight (to piggyback on), and whether it failed recently. Cached results are invalidated when upstream models refresh: a run records the affected models, and Vulcan marks every cached fingerprint whose dependencies overlap as stale, so the next identical query executes fresh instead of returning stale data. This metadata lives in the state connection (see [State](/references/dataos-resources/vulcan/core-concepts/state.md#query-and-serving-state)).

### Result formats

The result endpoint supports Parquet (default; redirects to a presigned URL), JSON, CSV, and YAML. For JSON, CSV, and YAML you can paginate and project with `limit`, `offset`, and `columns`; Parquet ignores those and returns the full file.

For the full request and response shapes, endpoints, and state tables, see the [semantic query lifecycle guide](/references/dataos-resources/vulcan/incremental-by-time/semantic-query-lifecycle.md). To preview the SQL a semantic query generates before you send it, use `vulcan transpile` (see [Semantic model and components](/references/dataos-resources/vulcan/core-concepts/semantic-model-and-components.md#querying-the-semantic-layer)).

## Related docs

* [State](/references/dataos-resources/vulcan/core-concepts/state.md): what interval and query state Vulcan keeps.
* [Semantic query lifecycle guide](/references/dataos-resources/vulcan/incremental-by-time/semantic-query-lifecycle.md): the full asynchronous query path.
* [Troubleshooting](/references/dataos-resources/vulcan/troubleshooting.md): validation and request errors at query time.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://v2.dataos.info/references/dataos-resources/vulcan/observability.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.