> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/dataos-resources/vulcan/incremental-by-time/semantic-query-lifecycle.md).

# Semantic query lifecycle

This guide traces how a semantic query travels through Vulcan, from the POST request to the result download.

## Overview

Semantic queries are asynchronous. You submit a query, get a statement ID back immediately, and poll for the result. This design lets Vulcan deduplicate identical queries, serve results from cache, and scale execution across workers.

```
POST query ──► statement ID ──► poll status ──► GET result
```

The lifecycle has six stages:

1. **Submit:** the client sends a semantic query.
2. **Transpile:** Vulcan converts semantic references to engine SQL.
3. **Cache check:** Vulcan looks for an existing result before executing.
4. **Execute:** a worker runs the SQL on the engine.
5. **Store:** the result is saved as a Parquet file in object storage.
6. **Fetch:** the client retrieves the result using the statement ID.

## 1. Submit a query

Send a `POST` to `/api/v1/query/semantic/rest` with a JSON body that describes what you want. You reference semantic names defined in your `models/semantics/` files, such as `orders.total_revenue` and `orders.region`, and never write SQL directly.

```json
{
  "query": {
    "measures": ["<alias>.<measure_name>"],
    "dimensions": ["<alias>.<column_name>"],
    "timeDimensions": [
      {
        "dimension": "<alias>.<column_name>",
        "granularity": "month",
        "dateRange": ["<start_date>", "<end_date>"]
      }
    ],
    "filters": [],
    "segments": ["<alias>.<segment_name>"],
    "order": { "<field>": "asc" },
    "limit": 1000,
    "timezone": "UTC"
  }
}
```

| Field            | Description                                                                                                                                                |
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `measures`       | Measures to calculate.                                                                                                                                     |
| `dimensions`     | Columns to group by.                                                                                                                                       |
| `timeDimensions` | Time-based dimensions with granularity and date ranges. Valid `granularity` values: `second`, `minute`, `hour`, `day`, `week`, `month`, `quarter`, `year`. |
| `filters`        | Filter conditions.                                                                                                                                         |
| `segments`       | Predefined segment filters from semantic models.                                                                                                           |
| `order`          | Sort order.                                                                                                                                                |
| `limit`          | Maximum rows (1 to 50,000, default 10,000).                                                                                                                |
| `timezone`       | Timezone for time dimensions (default `UTC`).                                                                                                              |

## 2. Transpile to SQL

The engine does not understand semantic names. Vulcan loads the project's semantic catalog (all models, measures, dimensions, joins, and their physical mappings), sends it with your query to the transpiler service, and gets back engine-specific SQL. The transpiler handles joins, aggregations, grouping, and dialect-specific syntax automatically. You can preview this SQL locally with `vulcan transpile` (see [Semantic model and components](/references/dataos-resources/vulcan/core-concepts/semantic-model-and-components.md#querying-the-semantic-layer)).

## 3. Cache check

Before running anything, Vulcan hashes the generated SQL into a fingerprint and asks three questions:

* **Has this exact query run before?** If a cached result exists and the upstream data has not changed, Vulcan returns it immediately, with no engine call.
* **Is someone else already running this query?** If an identical query is in flight, Vulcan piggybacks on it; both requests share the result.
* **Did this query fail recently?** If so, Vulcan returns the error directly to avoid hammering a broken query. Override with `retry_on_recent_failure`.

Cached results are invalidated when upstream models refresh. If a pipeline run updates `orders`, all cached results that depend on `orders` expire, and the next identical query executes fresh.

## 4. Statement ID

Whichever path the request takes, the API responds immediately with HTTP 202 and a statement ID:

```json
{
  "id": "stmt-abc-123",
  "status": "QUEUED",
  "strategy": "execute",
  "_links": {
    "self": "/api/v1/query/statement/stmt-abc-123",
    "result": "/api/v1/query/statement/stmt-abc-123/result"
  }
}
```

Status moves `ACCEPTED → QUEUED → IN_PROGRESS → SUCCESS` (or `FAILED` or `CANCELLED`). Cache hits skip to `SUCCESS`; piggybacks mirror the primary query. The `strategy` records the path: `execute` (new execution), `from_cache` (cache hit), or `await_primary` (piggyback).

## 5. Execution

For new executions, the query goes through a PostgreSQL-backed job queue (PGQueuer). A worker picks it up, runs the SQL on the engine, saves the result as Parquet in object storage (S3, MinIO, or GCS), records metadata (row count, file size, schema, lineage), creates a cache entry, and updates the statement status.

## 6. Fetch the result

Poll for status:

```
GET /api/v1/query/statement/{id}
```

Keep polling until `status` is `SUCCESS` or `FAILED`. Then download:

```
GET /api/v1/query/statement/{id}/result
```

The result endpoint supports multiple formats:

| Format  | How to request                              | Behavior                                     |
| ------- | ------------------------------------------- | -------------------------------------------- |
| Parquet | Default or `format=parquet`                 | Redirects (307) to a presigned download URL. |
| JSON    | `Accept: application/json` or `format=json` | Returns data inline.                         |
| CSV     | `Accept: text/csv` or `format=csv`          | Returns data inline.                         |
| YAML    | `Accept: application/yaml` or `format=yaml` | Returns data inline.                         |

For JSON, CSV, and YAML, filter the result with `limit`, `offset`, and `columns`. Parquet ignores these and returns the full file.

## State tables

Vulcan persists query metadata in four tables in the state connection database (see [State](/references/dataos-resources/vulcan/core-concepts/state.md)). These power caching, deduplication, status tracking, and saved queries:

| Table                 | Purpose                                                                                  |
| --------------------- | ---------------------------------------------------------------------------------------- |
| `_query_results`      | Metadata about each result (object-store path, row count, schema, lineage).              |
| `_query_fingerprints` | The cache index: maps a SQL fingerprint to a result, with `depends_on` for invalidation. |
| `_query_requests`     | An audit log of every submission, its strategy, and its status.                          |
| `_query_perspective`  | Saved queries (perspectives) that can auto-refresh when dependencies change.             |

All `_query_*` tables live in the schema set by the `state_schema` config key (default: `"vulcan"`). The schema can be customized in `state_connection_config.yaml` under the `state_schema` key.

### `_query_fingerprints` column schema

| Column                  | Type      | Description                                               |
| ----------------------- | --------- | --------------------------------------------------------- |
| `fingerprint`           | string    | SHA-256 hash of the normalized query. Primary key.        |
| `normalized_sql`        | string    | Canonical form of the SQL (whitespace/casing normalized). |
| `query_type`            | string    | `SEMANTIC_REST` or `RAW_SQL`.                             |
| `created_ts`            | timestamp | When this fingerprint was first seen.                     |
| `expires_ts`            | timestamp | When the cached result expires (NULL = never).            |
| `invalidated_ts`        | timestamp | When the result was invalidated (NULL = still valid).     |
| `invalidated_by_run_id` | string    | ID of the run that invalidated this fingerprint.          |

### `_query_requests` column schema

| Column               | Type      | Description                                                  |
| -------------------- | --------- | ------------------------------------------------------------ |
| `request_id`         | string    | Unique ID for this request.                                  |
| `fingerprint`        | string    | FK → `_query_fingerprints.fingerprint`.                      |
| `primary_request_id` | string    | ID of the original request if this is a deduplication reuse. |
| `pgq_job_id`         | string    | PGQ job ID if processed asynchronously.                      |
| `ttl_minutes`        | int       | TTL in minutes (0 = never expire; range 5–43200).            |
| `created_ts`         | timestamp | Request submission time.                                     |
| `status`             | string    | `PENDING`, `RUNNING`, `SUCCESS`, `FAILURE`, `CANCELLED`.     |

### `_query_results` column schema

| Column        | Type      | Description                                               |
| ------------- | --------- | --------------------------------------------------------- |
| `fingerprint` | string    | FK → `_query_fingerprints.fingerprint`.                   |
| `result_path` | string    | Object-store path to the result file.                     |
| `row_count`   | int       | Number of rows in the result.                             |
| `created_ts`  | timestamp | When the result was written.                              |
| `references`  | array     | List of model FQNs whose data contributes to this result. |

### `_query_perspective` column schema

| Column             | Type   | Description                                                       |
| ------------------ | ------ | ----------------------------------------------------------------- |
| `slug`             | string | User-defined identifier. Must be unique per environment.          |
| `statement_id`     | string | FK → `_query_requests.request_id`.                                |
| `auto_refresh`     | bool   | Whether perspective results refresh automatically with the cache. |
| `is_public`        | bool   | Whether the perspective is accessible to all users.               |
| `tags`             | array  | User-defined tags.                                                |
| `owner`            | string | Owner's username.                                                 |
| `allowed_user_ids` | array  | Users allowed to access this perspective if `is_public=false`.    |

### How the tables relate

* `_query_requests` references `_query_fingerprints` via `fingerprint`
* `_query_perspective` references `_query_requests` via `statement_id`
* `_query_results` references `_query_fingerprints` via `fingerprint`

### Cache invalidation flow

1. A `vulcan run` or `vulcan plan --apply` completes; a record is written to the `_runs` activity table.
2. Vulcan finds all fingerprints whose `references` include any model updated in this run.
3. For each matching fingerprint, `expires_ts` is set to `now()` (immediate expiry).
4. `invalidated_ts` and `invalidated_by_run_id` are set on the fingerprint record.
5. The next request for the same normalized SQL sees an expired fingerprint and executes fresh.

Cached results stay fresh as long as the underlying data has not changed, and expire the moment it does.

## Quick reference

| Component            | Details                                                           |
| -------------------- | ----------------------------------------------------------------- |
| Submit endpoint      | `POST /api/v1/query/semantic/rest`                                |
| Poll endpoint        | `GET /api/v1/query/statement/{id}`                                |
| Result endpoint      | `GET /api/v1/query/statement/{id}/result`                         |
| Semantic definitions | `models/semantics/*.yml`                                          |
| Job queue            | PostgreSQL-backed (PGQueuer)                                      |
| Result storage       | Parquet in object storage (S3 / MinIO / GCS)                      |
| Caching              | Fingerprint-based, auto-invalidated on model refreshes            |
| Result formats       | Parquet, JSON, CSV, YAML                                          |
| State connection     | Configured via `state_connection` (default: warehouse connection) |

## Related docs

* [Observability](/references/dataos-resources/vulcan/observability.md): how runs invalidate the cache and where activity is recorded.
* [Semantic model and components](/references/dataos-resources/vulcan/core-concepts/semantic-model-and-components.md): the definitions queries resolve against.
* [Troubleshooting](/references/dataos-resources/vulcan/troubleshooting.md): semantic query and transpiler errors.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://v2.dataos.info/references/dataos-resources/vulcan/incremental-by-time/semantic-query-lifecycle.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.