> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/dataos-resources/vulcan/core-concepts/state.md). # State State is how Vulcan knows what has already been applied. It records which snapshots are promoted, which time intervals have been processed, which environments exist, and the metadata behind caching and query history. Without state, Vulcan could not tell a new model from an existing one, or a processed interval from a missing one. State lives in the **state connection**, configured per gateway in `config.yaml`. The warehouse is where Vulcan materializes data; the state connection is where Vulcan remembers what it did. ## The state connection Point `state_connection` at a database separate from your engine so state operations do not interfere with data processing. Use a lightweight store such as DuckDB for local development, and Postgres for production. ```yaml gateways: default: connection: type: postgres host: warehouse port: 5432 database: warehouse user: vulcan password: "{{ env_var('DB_PASSWORD') }}" state_connection: type: postgres host: statestore port: 5432 database: statestore user: vulcan password: "{{ env_var('STATE_DB_PASSWORD') }}" state_schema: vulcan # schema where state tables live (default: "vulcan") ``` If you do not configure `state_connection`, Vulcan falls back to the warehouse connection. State tables are created in `state_schema` (default `vulcan`). ## Applied state Applied state is the version of the data product Vulcan is allowed to run: an approved blueprint. When a developer changes a local file, that change is not automatically part of the blueprint. It must go through `vulcan plan` first. ``` Applied state: sales.orders uses approved version A Local file: sales.orders was edited and now represents version B vulcan run: still runs version A vulcan plan: reviews and applies version B ``` This protects teams from accidentally running unapproved changes. `vulcan plan` reads applied state to build a context diff and decide what changed; `vulcan run` reads applied state to find which intervals are missing. Neither command treats local edits as live until a plan applies them. ## What state tracks State records the information Vulcan needs to plan, run, and serve safely: * **Snapshots and fingerprints.** Each model version is fingerprinted by its data hash, metadata hash, and parent data hash. State stores these so a plan can detect direct, indirect, and metadata-only changes. See [Plan](/references/dataos-resources/vulcan/core-concepts/plan.md#snapshot-fingerprints). * **Promoted snapshots.** With a virtual layer, state records which physical snapshot each consumer-facing name points to, so promotion is a state and view update rather than a data rewrite. * **Intervals.** State records which time intervals each model has processed. This is what makes incremental processing work: `vulcan run` compares the calculated intervals against recorded intervals and processes only the missing ones. See the [incremental by time guide](/references/dataos-resources/vulcan/incremental-by-time.md). * **Environments.** State tracks which environments exist (such as `prod` and dev environments) so commands can target the right one. `vulcan environments` lists them. * **Plan and run activity.** State records plan applications, backfills, errors, and run history. ## How incremental state works When you first plan an incremental model, Vulcan calculates all intervals from the start date to now, backfills the missing ones, and records them. On the next run, it recalculates intervals, compares against what state already has, and processes only the new ones. ``` First plan (Jan 15): Calculates 3 weeks of intervals -> processes all 3 -> records weeks 1-3 Second run (Jan 22): Calculates 4 weeks -> state has weeks 1-3 -> processes only week 4 ``` This is why incremental models are fast and cheap: state lets Vulcan skip everything it has already done. ## Query and serving state When you serve a data product through the semantic APIs, Vulcan also keeps query metadata in the state connection. Four tables in the state schema power caching, deduplication, status tracking, and saved queries: | Table | Purpose | | --------------------- | ---------------------------------------------------------------------------------------------------- | | `_query_results` | Metadata about each stored result (path in object storage, row count, schema, lineage). | | `_query_fingerprints` | The cache index: maps a SQL fingerprint to a result, with the models it depends on for invalidation. | | `_query_requests` | An audit log of every query submission and its resolution path. | | `_query_perspective` | Saved queries (perspectives) that can auto-refresh when dependencies change. | When a pipeline run refreshes upstream models, Vulcan finds the fingerprints whose `depends_on` overlaps the affected models and marks them stale, so the next identical query executes fresh instead of returning stale data. DQ profiles are also written to a state-side table (`_check_profiles`). See [Observability](/references/dataos-resources/vulcan/observability.md) and the [semantic query lifecycle guide](/references/dataos-resources/vulcan/incremental-by-time/semantic-query-lifecycle.md) for the full query path. ## State and the two plan modes State works the same way with or without a virtual layer; the difference is what it points at: * **With a virtual layer (`vde: true`).** State records versioned physical snapshots and which one each consumer-facing view is promoted to. Applied states can safely reuse unchanged physical snapshots. * **Without a virtual layer (`vde: false`).** State still fingerprints snapshots and tracks intervals, but models materialize under their original names, and plans are forward-only. In both modes, `vulcan run` works from applied state: it finds missing intervals, respects schedules and signals, runs the DAG in order, and records intervals and activity. The virtual layer does not change how Vulcan detects code changes; that belongs to `vulcan plan`. ## Related docs * [Plan](/references/dataos-resources/vulcan/core-concepts/plan.md): how plans read state to classify changes and compute intervals. * [Observability](/references/dataos-resources/vulcan/observability.md): how `vulcan run` uses interval state to process new data. * [Incremental by time](/references/dataos-resources/vulcan/incremental-by-time.md): a worked example of interval tracking. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://v2.dataos.info/references/dataos-resources/vulcan/core-concepts/state.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.