> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/dataos-resources/vulcan/engine-overview.md).

# Engine overview

An engine is the warehouse or compute layer where Vulcan runs your transformations and stores your data. Vulcan does not have its own storage or compute. It generates the right SQL for your engine, executes it there, and reads and writes tables in place. Analytics can live in Snowflake, operations in Postgres, and machine learning on Spark, and the same data product contract reaches every consumer regardless of which engine sits underneath.

Vulcan works with Postgres, Snowflake, BigQuery, Databricks, Redshift, Spark, Trino, Microsoft Fabric, SQL Server, and MySQL. Availability differs per engine, so check the connection manual before you finalize a production setup.

## How Vulcan reaches an engine

You connect to an engine through a **gateway** in `config.yaml`. A gateway defines a `connection` (where Vulcan runs your data) and, optionally, a `state_connection` (where Vulcan stores its internal state). Every gateway needs a `connection.type`, and your project also needs `model_defaults.dialect`. In most projects, the two values name the same engine.

```yaml
gateways:
  default:
    connection:
      type: postgres
      host: warehouse
      port: 5432
      database: analytics
      user: vulcan
      password: "{{ env_var('POSTGRES_PASSWORD') }}"

model_defaults:
  dialect: postgres
```

The `connection.type` chooses the engine adapter that runs your queries. The `model_defaults.dialect` tells Vulcan which SQL dialect to write your models in, so it can transpile a model to the target engine. Keep the two aligned unless you have a specific transpilation need.

{% hint style="info" %}
Use environment variables for secrets such as passwords, tokens, and key files. Keep credentials out of `config.yaml` and inject them at runtime.
{% endhint %}

## Dialect, transpilation, and engine behavior

You write models in one dialect, and Vulcan rewrites them for the engine your gateway points at. For example, you write PostgreSQL-style casts (`x::INTEGER`), and Vulcan converts them to whatever your execution engine needs. This is also why the same model can move from a local DuckDB or Postgres engine to a production Snowflake engine without a rewrite.

Engines differ in what they support, and Vulcan handles those differences for you:

* **Materialization strategy** varies by engine. A `FULL` model runs `CREATE OR REPLACE TABLE` on Snowflake but `DROP` then `CREATE` on Postgres. An incremental model uses `MERGE` on engines that support it and `DELETE` then `INSERT` where they do not. See [Model and kinds](/references/dataos-resources/vulcan/core-concepts/model-and-kinds.md) for the per-kind tables.
* **Comment registration** (table and column descriptions) is supported on Postgres, Snowflake, and Spark, and skipped silently where the engine does not support it.
* **Feature support** such as `MERGE`, materialized views, partitioning, and clustering depends on the engine. The model property reference notes which engines support each option.

## Engine-specific connection manuals

This page explains the engine concept generically. The connection manual for each engine, including connection fields, prerequisites, permissions, authentication, and engine-specific behavior, lives in the Engine guide, not here.

See the [Engine guide](/references/engine-guide/engine-guide.md) for the per-engine manuals (Snowflake, Databricks, Spark, Trino, Postgres, and others). Use those pages when you need the exact connection fields and permissions for a specific engine.

## State connection

A gateway can keep its state separate from its data. Point `state_connection` at a lightweight store such as DuckDB for local development, and at Postgres for production. Separating state from your warehouse keeps state operations from interfering with data processing. See [State](/references/dataos-resources/vulcan/core-concepts/state.md) for what Vulcan stores and why.

## Related docs

* [Local Development Kit (LDK)](/references/dataos-resources/vulcan/ldk.md): install the CLI and create your first project against an engine.
* [Configuration](/references/dataos-resources/vulcan/configuration.md): the full `config.yaml` and gateway reference.
* [Engine guide](/references/engine-guide/engine-guide.md): per-engine connection manuals.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://v2.dataos.info/references/dataos-resources/vulcan/engine-overview.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.