> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/engine-guide/engine-guide.md). # Overview This is the full technical reference for each engine that Vulcan runs against in DataOS. Where the Build section gives you the thin connection config you need to wire a gateway, this guide gives you the depth: authentication and grants, materialisation behaviour per model kind, local development setups, operational boundaries, performance ceilings, and the failure modes you hit in real projects. Use this guide when you are building or operating a Data Product and you need to understand how a specific engine behaves, not just how to connect to it. Each page consolidates the engine setup, the runtime behaviour, and the troubleshooting reference into one place. ## Who this is for You are a data-product builder, a data engineer setting up an engine for the first time, or a developer debugging a Data Product. Each page leads with the common path (connect, materialise, deploy) and pushes deep edge cases into clearly marked sections, so you can read the top of a page and stop, or go all the way down when you need to. ## How Vulcan uses an engine Vulcan transforms data that an ingestion path (such as Nilus) has already landed in your engine. Across every engine, the pattern is the same: * You connect a gateway to the engine and set `model_defaults.dialect` to match. * Vulcan compiles your SQL and semantic models to native engine SQL and materialises them as tables and views. * A serving layer exposes the Data Product through REST, GraphQL, and (where supported) MySQL-wire and Postgres-wire endpoints. * A scheduled workflow plans and runs the models; a depot binds the engine credentials to the DataOS resource. What changes per engine is the connection options, the authentication methods, the supported model kinds, the identifier casing rule, and the operational limits. That is what each page documents. ## Choose an engine

Engine	Adapter type	Use it when
Snowflake	`snowflake`	You run on a Snowflake cloud data warehouse with separated storage and compute.
Databricks	`databricks`	You run on a Databricks SQL warehouse with Unity Catalog and Delta Lake.
Spark	`spark`	You run a lakehouse on Apache Spark with Apache Iceberg tables over object storage.
Postgres	`postgres`	You run on a PostgreSQL server, for smaller projects or full control of the database.
Trino	`trino`	You query across data lakes, databases, and object storage with a distributed engine. Trino comes in three shapes, covered in its own section.

## Common rules across engines * Set `connection.type` and `model_defaults.dialect` to match the target engine unless you have a specific transpilation need. * Store credentials outside `config.yaml` and inject them at runtime with `env_var()` locally and DataOS secrets in production. * Use a separate read-only role for the metadata scanner. Do not reuse the Vulcan service role for scanning. * Set `timezone: UTC` on production schedules, set `endOn` 1 to 2 years out, and use `concurrencyPolicy: Forbid` to prevent overlapping runs. * Declare source tables that Vulcan does not own in `external_models.yaml` so the linter and lineage can resolve them. ## Related For the thin connection config and the wiring steps, see the matching Build pages under Connect engine. Each engine page below links back to its Build counterpart. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://v2.dataos.info/references/engine-guide/engine-guide.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.