> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/dataos-resources/vulcan/configuration.md). # Configuration Your Vulcan project needs a configuration file. It tells Vulcan how to connect to your engine, where to store state, and what defaults to use for your models. Without it, Vulcan does not know where your data lives or how to run your transformations. Create a configuration file in your project root: * `config.yaml`: YAML format. Use this for most projects. * `config.py`: Python format. Use this when you need dynamic configuration or want to generate settings programmatically. Business-facing usage guidance lives separately in `usage.yaml`. Keep connection and runtime settings in `config.yaml`, and keep guidance for consumers in `usage.yaml`. ## Example configuration A typical `config.yaml` for the `orders-analytics` example: ```yaml # Project identity name: orders-analytics display_name: Orders Analytics Platform description: Orders Analytics is a centralized data product delivering clean, trusted insights across the full order lifecycle. # Catalog metadata discoverable: true version: 0.1.2 alignment: consumer_aligned # Environment behaviour vde: false # set to true to enable Virtual Data Environments; not supported on spark/trino # Classification tags: - e-commerce - retail - sales_analytics terms: - glossary.data_product - glossary.analytics_platform # Gateway connection gateways: default: connection: type: postgres host: warehouse port: 5432 database: warehouse user: vulcan password: "{{ env_var('DB_PASSWORD') }}" state_connection: type: postgres host: statestore port: 5432 database: statestore user: vulcan password: "{{ env_var('STATE_DB_PASSWORD') }}" default_gateway: default # Model defaults (required) model_defaults: dialect: postgres start: 2024-01-01 cron: '@daily' # Linting rules linter: enabled: true rules: - ambiguousorinvalidcolumn - invalidselectstarexpansion ``` ## Minimal configuration The non-skippable parts of `config.yaml` are a non-empty `name`, a non-empty `description`, at least one working `gateways..connection`, `model_defaults.dialect`, and at least one entry in `users`. The runtime also needs `DATAOS_TENANT_ID` in the environment. ```yaml name: my-project description: My project description users: - username: jane email: jane@example.com type: OWNER gateways: default: connection: type: postgres host: localhost port: 5432 database: mydb user: myuser password: mypass model_defaults: dialect: postgres ``` ```bash # Required at runtime, not in YAML export DATAOS_TENANT_ID=my-tenant ``` Everything else has a default and you can omit it. {% hint style="info" %} **Tenant comes from the environment.** `tenant` is required by the stack, but it is not a YAML key. In production, the stack injects it through `DATAOS_TENANT_ID`. For local development, export it before running Vulcan. Without it, Vulcan refuses to load the project. {% endhint %} ## Project settings Project settings identify your data product. They do not change how Vulcan runs, but catalog tools rely on them for organization and discovery. | Option | Description | Required | | -------------- | ------------------------------------------------------------------------------- | :------: | | `name` | Project identifier. Can also be set via `DATAOS_RESOURCE_NAME`. | Yes | | `description` | Project description. Validated as non-empty. | Yes | | `display_name` | Human-readable name for UI and docs. | No | | `discoverable` | Whether the product appears in catalog search (default `true`). | No | | `version` | Release version, valid Semantic Versioning 2.0 (for example, `0.1.2`). | No | | `alignment` | Data Mesh orientation: `source_aligned` or `consumer_aligned`. | No | | `tags` | Labels for categorization. Merged with `DATAOS_RESOURCE_TAGS`. | No | | `terms` | Business glossary terms in dot notation (for example, `glossary.data_product`). | No | ## Users The `users` block is **required**: config load fails if the list is empty or the key is omitted. List the DataOS users associated with the project. Each `username` is validated against the current tenant at load time, so a mis-typed or removed username fails the load. ```yaml users: - username: jane email: jane@example.com type: OWNER - username: data_team email: data-team@example.com type: CONTRIBUTOR ``` | Key | Description | Required | | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------: | | `username` | DataOS username. Validated against the current tenant. | Yes | | `email` | Email address, used for notifications. | No | | `type` | Role in the project: `OWNER` or `CONTRIBUTOR`. | No | | `notification_targets` | Per-user notification targets. Same format as the root `notification_targets` key, but scoped to this user. When present, the user receives notifications via their personal targets instead of the global ones. | No | ```yaml users: - username: jane email: jane@example.com type: OWNER notification_targets: - type: slack_api token: "{{ env_var('SLACK_API_TOKEN') }}" channel: "UXXXXXXXXX" notify_on: - dq_failure - run_failure ``` ## Gateways Gateways define how Vulcan connects to your engine and state backend. Define multiple gateways for different environments (dev, staging, prod), each with its own connection settings. | Component | Description | Required | | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------- | :------: | | `connection` | Primary engine connection. | Yes | | `state_connection` | Where Vulcan stores internal state (defaults to `connection`). Point this at DuckDB for local testing and Postgres for production. | No | | `test_connection` | Connection for running tests (defaults to DuckDB). | No | | `scheduler` | Scheduler configuration. | No | | `state_schema` | Schema name for state tables (default `vulcan`). | No | | `default_gateway` | Which gateway to use when none is specified. | No | ```yaml gateways: default: connection: type: postgres host: warehouse port: 5432 database: warehouse user: vulcan password: "{{ env_var('DB_PASSWORD') }}" default_gateway: default ``` For the connection fields and authentication of a specific engine, see the [Engine guide](/references/engine-guide/engine-guide.md). For what Vulcan stores in the state connection, see [State](/references/dataos-resources/vulcan/core-concepts/state.md). {% hint style="success" %} Separate the state connection from your engine for better isolation. It keeps state operations from interfering with data processing. {% endhint %} ## Model defaults The `model_defaults` section is required, and you must at least set `dialect`. Other defaults apply to all models automatically, so you do not repeat them in every model file. The default model kind is `VIEW` unless you override it with `kind`. ```yaml model_defaults: dialect: postgres # required owner: data-team start: 2024-01-01 cron: '@daily' ``` Common keys: `dialect`, `owner`, `start`, `cron`, `kind`, `interval_unit`, `batch_concurrency`, `table_format`, `storage_format`, `on_destructive_change` (default `error`), `on_additive_change` (default `apply`), `physical_properties`, `virtual_properties`, `session_properties`, `audits`, `optimize_query`, `allow_partials`, `enabled`, `pre_statements`, and `post_statements`. See [Model and kinds](/references/dataos-resources/vulcan/core-concepts/model-and-kinds.md) for what each model property does. ### Owner resolution `model_defaults.owner` defaults to the `username` of the first entry in `config.users` when unset. Whether defaulted or set explicitly, the resolved owner must reference a username listed in `config.users`, or config load fails. Each model's own `owner` field is likewise validated against `config.users` at load time; a model naming an unlisted owner is rejected. ### Identifier resolution When an engine receives a query like `SELECT id FROM "some_table"`, it must resolve what `id` and `"some_table"` refer to. Dialects resolve identifiers differently: some are case-sensitive when quoted, and case-insensitive identifiers are usually lowercased or uppercased before lookup. Vulcan normalizes and quotes identifiers per dialect to extract column-level lineage. Configure the normalization strategy per dialect when you need to preserve casing: ```yaml model_defaults: dialect: "bigquery,normalization_strategy=case_sensitive" ``` ### Gateway-specific model defaults Override the global defaults for a single gateway by setting `model_defaults` under that gateway. This aligns identifier normalization across engines that resolve names differently: ```yaml gateways: redshift: connection: type: redshift model_defaults: dialect: "snowflake,normalization_strategy=case_insensitive" snowflake: connection: type: snowflake default_gateway: snowflake model_defaults: dialect: snowflake start: 2025-02-05 ``` Here the project default dialect is `snowflake`, and the `redshift` gateway transpiles Snowflake SQL to Redshift while treating identifiers as case-insensitive to match Snowflake's behavior. ### Name inference By default you set `name` in every model. If your directory structure matches your schema names, turn on name inference so a model at `models/sales/daily_sales.sql` becomes `sales.daily_sales`: ```yaml model_defaults: dialect: snowflake infer_names: true ``` ## Variables Store secrets and dynamic values outside your config files using environment variables, `.env` files, or overrides. Vulcan loads environment variables from a `.env` file in your project directory. Add `.env` to `.gitignore` so secrets stay out of source control. To use a custom `.env` location, pass the `--dotenv ` CLI flag or set `VULCAN_DOTENV_PATH`. The `--dotenv` flag must be placed **before** the subcommand: ```bash vulcan --dotenv /path/to/custom/.env plan export VULCAN_DOTENV_PATH=/path/to/custom/.env # alternative ``` ```bash # .env SNOWFLAKE_PW=my_secret_password S3_BUCKET=s3://my-data-bucket/warehouse # Override config values with the VULCAN__ prefix VULCAN__DEFAULT_GATEWAY=production VULCAN__MODEL_DEFAULTS__DIALECT=snowflake ``` Reference environment variables in YAML with `{{ env_var('VARIABLE_NAME') }}`: ```yaml gateways: my_gateway: connection: type: snowflake user: admin password: "{{ env_var('SNOWFLAKE_PW') }}" account: my_account ``` Environment variables with the `VULCAN__` prefix have the highest precedence and override config-file values. Use double underscores to navigate the hierarchy: ```bash export VULCAN__GATEWAYS__MY_GATEWAY__CONNECTION__PASSWORD="real_pw" ``` Set environment-variable overrides declaratively with the `env_vars` config key. You can also store reusable values under a `variables` block, optionally overridden per gateway, and give each user their own dev environment with `{{ user() }}`: ```yaml variables: warehouse_schema: analytics refresh_window_days: 7 default_target_environment: dev_{{ user() }} ``` ## Execution hooks Run SQL statements, SQL files, or macros automatically at the start and end of `vulcan plan` and `vulcan run`. Use `before_all` for setup (create schemas, validate prerequisites, log a run start) and `after_all` for cleanup or post-processing (grant privileges, refresh materialized views, send notifications). ```yaml before_all: - CREATE SCHEMA IF NOT EXISTS analytics - "@validate_source_data()" - file: ./statements/setup.sql after_all: - "@grant_select_privileges()" - ANALYZE analytics.daily_sales - ./statements/cleanup.sql ``` Each hook entry can be inline SQL, a macro call (`"@macro_name()"`), a file object (`{file: ./path.sql}`), or a file path string. Macros invoked in hooks can read `evaluator.views`, `evaluator.schemas`, `evaluator.this_env`, and `evaluator.gateway`, so a hook can adapt to the current environment. Keep hooks idempotent (use `IF NOT EXISTS` or `ON CONFLICT`), and gate production-only operations with `@IF(@this_env = 'prod', ...)`. Use `before_all` and `after_all` for pipeline-wide operations; use a model's `pre_statements` and `post_statements` for operations specific to one model. ## Linter The linter checks each model against your team's standards and catches common mistakes before they reach production. When you create a plan, every model's code is checked against the rules you configure, and violations stop the plan so you can fix them. Enable the linter and list the rules. The linter is off by default, so you must set `enabled: true`. ```yaml linter: enabled: true rules: - ambiguousorinvalidcolumn - invalidselectstarexpansion ``` Built-in rules are divided by scope. **Model-level rules** run once per model; **project-level rules** run once per project and validate project-wide configuration rather than individual model code. **Model-level rules** — run once per model: | Name | Check type | What it catches | | ---------------------------- | ----------- | -------------------------------------------------------------- | | `ambiguousorinvalidcolumn` | Correctness | Duplicate columns, or columns Vulcan cannot resolve. | | `invalidselectstarexpansion` | Correctness | A top-level `SELECT *` that Vulcan cannot expand into columns. | | `noselectstar` | Stylistic | A top-level `SELECT *`, even when it can be expanded. | | `nomissingaudits` | Governance | A model with no `audits` configured to test data quality. | **Project-level rules** — run once per project during `lint_models`, not per model: | Name | Check type | What it catches | | ------------------------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------- | | `nomissingdataosusername` | Governance | Every `config.users[].username` must exist in the current DataOS tenant. Reported once for the whole project, not per model. | Linter config keys: | Key | Description | Default | | ---------------------- | -------------------------------------------------------------------------------------- | ------- | | `linter.enabled` | Enable or disable linting. | `false` | | `linter.rules` | Rules to enforce at error level. Use `"ALL"` to enable every rule. | `[]` | | `linter.warn_rules` | Rules to report as warnings without stopping execution. | `[]` | | `linter.ignored_rules` | Rule names to suppress project-wide. Mutually exclusive with `rules` and `warn_rules`. | `[]` | Turn on every rule with `rules: "ALL"`, and exclude specific rules with `ignored_rules`. Put rules you want as suggestions in `warn_rules` so violations are reported without stopping execution. To exempt a single model, add `ignored_rules` to its `MODEL` block. You can also write custom rules in Python in the project's `linter/` directory by subclassing `Rule`. ```yaml linter: enabled: true rules: "ALL" warn_rules: ["invalidselectstarexpansion"] ignored_rules: ["noselectstar"] ``` ## Notifications Vulcan sends notifications when lifecycle events occur. Configure targets globally under `notification_targets`, or per user under each user's `notification_targets`. ```yaml notification_targets: - type: teams_webhook url: "{{ env_var('TEAMS_WEBHOOK_URL') }}" notify_on: - apply_failure - run_failure - dq_failure - type: console notify_on: - plan_change ``` Supported target types are `teams_webhook`, `slack_webhook`, `slack_api`, `smtp` (email), and `console`. The `notify_on` field selects which events fire a notification. | Event | `notify_on` value | Message sent | | -------------------- | ----------------- | --------------------------------------------------------------- | | Plan apply start | `apply_start` | "Plan apply started for environment `{environment}`." | | Plan apply end | `apply_end` | "Plan apply finished for environment `{environment}`." | | Plan apply failure | `apply_failure` | "Failed to apply plan.\n{exception}" | | Run start | `run_start` | "Vulcan run started for environment `{environment}`." | | Run end | `run_end` | "Vulcan run finished for environment `{environment}`." | | Run failure | `run_failure` | "Failed to run Vulcan.\n{exception}" | | Data quality start | `dq_start` | "Data quality checks started for environment `{environment}`." | | Data quality end | `dq_end` | "Data quality checks finished for environment `{environment}`." | | Data quality failure | `dq_failure` | "{dq\_error}" | Data quality failure notifications reach a model's owner when the model has an `owner`, runs at least one assertion, the owner has a target whose `notify_on` includes `dq_failure`, and the check fails in `prod`. During development, set the top-level `username` to a single user so only that user receives notifications. {% hint style="info" %} Use the `dq_*` event names. Older `check_start`, `check_end`, and `check_failure` values should be migrated to `dq_start`, `dq_end`, and `dq_failure`. {% endhint %} ### Custom notification classes To customize message text, subclass `BaseNotificationTarget` in a Python config file: ```python from vulcan.notification import BaseNotificationTarget class MyTarget(BaseNotificationTarget): def notify_apply_start(self, env): ... def notify_apply_end(self, env): ... def notify_apply_failure(self, env, exc): ... def notify_run_start(self, env): ... def notify_run_end(self, env): ... def notify_run_failure(self, env, exc): ... def notify_dq_failure(self, env, dq_error): ... ``` {% hint style="info" %} Override only the methods you need; the base class provides no-op defaults. {% endhint %} ## Environment and schema management These keys control how environments are named and where models land: | Key | Description | Default | | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | ------------------ | | `vde` | Turn on Virtual Data Environments (versioned physical tables plus a virtual layer). Rejected for `spark` and `trino` gateways. | `false` | | `default_target_environment` | Default environment for plan and run commands. | `prod` | | `environment_catalog_mapping` | Route each environment to a specific catalog. | `{}` | | `physical_schema_mapping` | Map model patterns to physical schema names. | `{}` | | `snapshot_ttl` / `environment_ttl` | Time before unused snapshots or dev environments are deleted. | `in 1 week` | | `pinned_environments` | Array of environment names excluded from janitor cleanup. | `[]` | | `environment_suffix_target` | Where the environment name is appended: `schema`, `table`, or `catalog`. | `schema` | | `physical_table_naming_convention` | How tables are named at the physical layer: `schema_and_table` or `table_only`. | `schema_and_table` | | `gateway_managed_virtual_layer` | When `true`, virtual-layer views are created by the model's own gateway connection rather than the state connection. | `false` | See [Plan](/references/dataos-resources/vulcan/core-concepts/plan.md) for how `vde` changes plan behavior. ## Environment variables A few values come from the shell or `.env`, not from YAML: | Variable | Effect | | ---------------------- | --------------------------------------------------------- | | `DATAOS_TENANT_ID` | Required at runtime. Supplies the tenant. Not a YAML key. | | `DATAOS_RESOURCE_NAME` | Overrides `name`. | | `DATAOS_RESOURCE_TAGS` | Merged into `tags`. | | `TEAMS_WEBHOOK_URL` | Recommended source for Teams webhook target URLs. | ## Migrating from the legacy schema If you have an older `config.yaml`, a few keys have moved: | Old key | Replacement | | ------------------------------------------- | ---------------------------------- | | `virtual_environment_mode: full` | `vde: true` | | `virtual_environment_mode: dev_only` | `vde: false` (or omit) | | `auto_categorize_changes` (top level) | `plan.auto_categorize_changes` | | `include_unmodified` (top level) | `plan.include_unmodified` | | `physical_schema_override` | `physical_schema_mapping` | | `disable_anonymized_analytics` | `analytics.enabled` | | `tenant` (in YAML) | `DATAOS_TENANT_ID` env var | | `heimdall.after_authorize` | `after_authorize` at root | | `check_start`, `check_end`, `check_failure` | `dq_start`, `dq_end`, `dq_failure` | | `metadata` (in `config.yaml`) | `usage.yaml` | ## Complete configuration reference The sections above cover the keys you set most often. This reference lists the remaining `config.yaml` keys by area. `model_defaults`, `linter`, `notifications`, and the environment keys are documented in their own sections above. ### Gateway and connection (root-level) Beyond the per-gateway keys in [Gateways](#gateways), these root-level keys set fallbacks and a shared state connection: | Key | Description | Default | | --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | | `gateways..variables` | Gateway-specific variables merged into the variable scope. | `{}` | | `default_gateway` | Name of the default gateway. | `""` | | `default_connection` | Root-level default connection used when a gateway omits one. | `null` | | `default_test_connection` | Root-level default connection for unit tests. | `null` | | `default_scheduler` | Root-level default scheduler. | Built-in | | `state` | Separate root-level state connection (alternative to per-gateway `state_connection`). Can also be loaded from `/etc/dataos/secret/state_connection_config.yaml`. With a root-level `state`, `state_schema` defaults to `{name}` (normalized). | `null` | ### Project management | Key | Description | Default | | --------------------------- | ------------------------------------------------- | --------------------- | | `ignore_patterns` | Glob patterns for files Vulcan should ignore. | Standard list | | `time_column_format` | Default format for model time columns. | `%Y-%m-%d` | | `infer_python_dependencies` | Auto-detect Python package requirements. | `true` | | `log_limit` | Default number of logs to keep. | `20` | | `cache_dir` | Directory for Vulcan's compiled project cache. | `.cache` | | `loader` / `loader_kwargs` | Loader class for project files and its arguments. | Default loader / `{}` | | `project` | Legacy alias of `name`; auto-filled. | — | ### Command configuration | Key | Description | Default | | ------------------------------ | -------------------------------------------------------------- | ------- | | `format` | SQL formatting options. | Default | | `ui` | UI server configuration. | Default | | `plan` | Plan command configuration. | Default | | `plan.auto_categorize_changes` | Auto-categorize changes as breaking/non-breaking. | Default | | `plan.include_unmodified` | Include unmodified models in plan output. | `false` | | `plan.use_finalized_state` | Use finalized state when creating plans. Requires `vde: true`. | `false` | | `migration` | Migration configuration. | Default | | `run` | Run command configuration. | Default | | `janitor` | Cleanup task configuration. | Default | | `model_naming` | Name-inference rules for models. | Default | | `cicd_bot` | CI/CD bot configuration. | `null` | ### Integrations and external services | Key | Description | Default | | ------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | --------------------------------------------------------------- | | `dbt` | dbt-specific configuration. | `null` | | `object_store` | Object storage for query results (MinIO/S3/GCS/Azure). | `null` | | `transpiler` | External transpiler service. | `{base_url: "http://127.0.0.1:8100", timeout: 30, token: null}` | | `graphql` | GraphQL proxy configuration. | `{base_url: "http://127.0.0.1:3000", timeout: 30}` | | `pgq` | PostgreSQL queue for async jobs. | Default | | `analytics` / `analytics.enabled` / `analytics.api_key` | CloudEvents telemetry. `api_key` is required when `enabled: true`. | `{enabled: false}` | | `openlineage` | OpenLineage data lineage integration. | `null` | | `after_authorize` | Auth extension hook called after Heimdall authorization, e.g. `plugins.auth_ext:resolve_user_groups`. | `null` | | `heimdall` / `heimdall.enabled` / `heimdall.base_url` | Heimdall authentication (Vulcan API only). `base_url` is required when `enabled: true`. | `{enabled: false}` | | `heimdall.timeout` | Request timeout in seconds for Heimdall auth calls. | `5` | | `hera` / `hera.enabled` / `hera.url` / `hera.token` | Hera/OpenMetadata sync. `url` and `token` are required when `enabled: true`. | `{enabled: false}` | ### Usage guidance (`usage.yaml`) Business-facing guidance lives in `usage.yaml`, not `config.yaml`. It is documentation for humans and catalog/AI experiences and does not affect runtime behavior. | Key | Description | Type | | ------------ | --------------------------------------------------------------------------------------------------------- | ---------------------- | | `good_for` | Use cases where the data product is a good fit. | array of string/object | | `not_for` | Use cases where the data product should not be used. | array of string/object | | `caveats` | Known limits, freshness notes, exclusions, or interpretation warnings. Each entry may carry a `severity`. | array of string/object | | `references` | Supporting links, each with `title`, `url`, and `type`. | array of object | ## Related docs * [Engine overview](/references/dataos-resources/vulcan/engine-overview.md): how gateways connect to engines. * [Model and kinds](/references/dataos-resources/vulcan/core-concepts/model-and-kinds.md): the model properties you can set as defaults. * [Deployment checklist](/references/dataos-resources/vulcan/deployment-checklist.md): the production deployment and CI/CD path. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://v2.dataos.info/references/dataos-resources/vulcan/configuration.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.