> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/engine-guide/engine-guide.md).

# Engine Reference Guide

The full per-engine technical reference for building and running a Data Product on each supported Vulcan engine in DataOS.

This is the full technical reference for each engine that Vulcan runs against in DataOS. Where the Build section gives you the thin connection config you need to wire a gateway, this guide gives you the depth: authentication and grants, materialization behavior per model kind, local development setups, operational boundaries, performance ceilings, and the failure modes you hit in real projects.

Use this guide when you are building or operating a Data Product and you need to understand how a specific engine behaves, not just how to connect to it. Each page consolidates the engine setup, the runtime behavior, and the troubleshooting reference into one place.

## Who this is for

You are a data-product builder, a data engineer setting up an engine for the first time, or a developer debugging a Data Product. Each page leads with the common path (connect, materialize, deploy) and pushes deep edge cases into clearly marked sections, so you can read the top of a page and stop, or go all the way down when you need to.

## How Vulcan uses an engine

Vulcan transforms data that an ingestion path (such as Nilus) has already landed in your engine. Across every engine, the pattern is the same:

* You connect a gateway to the engine and set `model_defaults.dialect` to match.
* Vulcan compiles your SQL and semantic models to native engine SQL and materializes them as tables and views.
* A serving layer exposes the Data Product through REST, GraphQL, and (where supported) MySQL-wire and Postgres-wire endpoints.
* A scheduled workflow plans and runs the models; a Depot binds the engine credentials to the DataOS resource.

What changes per engine is the connection options, the authentication methods, the supported model kinds, the identifier casing rule, and the operational limits. That is what each page documents.

## Choose an engine

| Engine                                                           | Adapter type | Use it when                                                                    |
| ---------------------------------------------------------------- | ------------ | ------------------------------------------------------------------------------ |
| [Snowflake](/references/engine-guide/snowflake.md)               | `snowflake`  | You run on a cloud warehouse with separate storage and compute.                |
| [Databricks](/references/engine-guide/databricks.md)             | `databricks` | You use Databricks SQL Warehouse, Unity Catalog, and Delta Lake.               |
| [Spark](/references/engine-guide/spark-1.md)                     | `spark`      | You run an Iceberg lakehouse on Apache Spark and object storage.               |
| [Postgres](/references/engine-guide/postgres.md)                 | `postgres`   | You need a PostgreSQL server for smaller projects or database control.         |
| [Redshift](/references/engine-guide/redshift.md)                 | `redshift`   | You run analytics workloads on Amazon Redshift.                                |
| [Athena](/references/engine-guide/athena.md)                     | `athena`     | You query S3 data through Athena and AWS Glue without managing clusters.       |
| [Trino](/references/engine-guide/trino-1.md)                     | `trino`      | You query data lakes, databases, and object storage with a distributed engine. |
| [SQL Server](/references/engine-guide/sql-server.md)             | `mssql`      | You run an existing SQL Server estate or need on-premises T-SQL control.       |
| [Microsoft Fabric](/references/engine-guide/microsoft-fabric.md) | `fabric`     | You use Fabric Warehouse with OneLake and Microsoft Entra authentication.      |

## Common rules across engines

Apply these rules when configuring an engine for Vulcan. They cover engine settings, credentials, metadata scanning, and external source definitions.

* In `config.yaml`, set `connection.type` and `model_defaults.dialect` to match the target engine unless you have a specific transpilation need.
* Keep credentials out of `config.yaml`. Use `env_var()` for local environment variables and DataOS secrets for production.
* Set `timezone: UTC` on production schedules, set `endOn` 1 to 2 years out, and use `concurrencyPolicy: Forbid` to prevent overlapping runs.
* Declare externally managed source tables and views in `inputs.yaml`. This lets Vulcan resolve them during linting and lineage generation.
* Use a separate read-only role for the metadata scanning. Do not reuse the Vulcan service role for scanning.

## Related

For the thin connection config and the wiring steps, see the matching Build pages under [Connect engine](https://v2.dataos.info/build/productize/connect-engine).


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/references/engine-guide/engine-guide.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.