> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/v1/resources/vulcan/configurations.md). # Configurations Your Vulcan project needs a configuration file. It tells Vulcan how to connect to your data warehouse, where to store state, and what defaults to use for your models. Without it, Vulcan doesn't know where your data lives or how to run your transformations. ## Configuration file `vulcan init` creates the base configuration file in your project root. Update it for your project: * `config.yaml`: Configure connections, runtime behavior, and model defaults. * `usage.yaml`: Business-facing guidance about the data product itself: who it's for, who it isn't for, caveats, and reference links. See [Usage guidance](/references/v1/resources/vulcan/configurations/usage.md). * `agreement.md`: An optional, free-form data-sharing/usage agreement, read verbatim. See [Agreement](/references/v1/resources/vulcan/configurations/agreement.md). {% hint style="warning" %} **Key casing differs by file type** `config.yaml` keys are `snake_case`. The `gateways/**` subtree is preserved exactly as written, since gateway names are user identifiers, not schema keys. Asset YAML files (semantic, metric, dq, and policy models) are loaded **without** key conversion. Keys must already be `snake_case`: camelCase keys such as `dependsOn` or `timeDimension` are silently not recognized, not auto-converted. Rename them: `dependsOn` → `depends_on`, `timeDimension` → `time_dimension`. The document root of an asset file accepts either `kind:` or `type:` (case-insensitive): both `kind: dq` and `type: dq` load the same way. {% endhint %} ## Example configuration Here's what a typical configuration file looks like: {% code title="" expandable="true" %} ```yaml # Project identity name: orders-analytics display_name: Orders Analytics Platform description: Orders Analytics is a centralized data product delivering clean, trusted insights across the full order lifecycle. # Catalog metadata discoverable: true version: 0.1.2 alignment: consumer_aligned # Environment behavior vde: false # set to true to enable Virtual Data Environments; not supported on spark/trino # Classification tags: - e-commerce - retail - sales_analytics - customer_analytics - postgres terms: - glossary.data_product - glossary.analytics_platform - glossary.sales_operations # Gateway Connection gateways: default: connection: type: postgres host: warehouse port: 5432 database: warehouse user: vulcan password: "{{ env_var('DB_PASSWORD') }}" default_gateway: default # Model Defaults (required) model_defaults: dialect: postgres start: 2024-01-01 cron: '@daily' # Linting Rules linter: enabled: true rules: - ambiguousorinvalidcolumn - invalidselectstarexpansion ``` {% endcode %} ## Configuration structure ```mermaid %%{init: {"theme":"base","themeVariables":{"fontFamily":"PP Neue Montreal, Inter, Helvetica Neue, Arial, sans-serif","fontSize":"14px","primaryColor":"#EDE9E5","primaryTextColor":"#242422","primaryBorderColor":"#242422","lineColor":"#242422","secondaryColor":"#D6CDC6","tertiaryColor":"#FFFFFF","clusterBkg":"#EDE9E5","clusterBorder":"#54DED1","edgeLabelBackground":"#FFFFFF"},"flowchart":{"curve":"basis","padding":12,"nodeSpacing":40,"rankSpacing":50}}}%% graph TB Config[config.yaml] Usage[usage.yaml] Config --> Project[Project Settings] Config --> Gateways[Gateways] Config --> StateConn[State Connection] Config --> ModelDefaults[Model Defaults] Config --> Semantic[Semantic Layer] Config --> Options[Optional Features] Usage --> GoodFor[good_for] Usage --> NotFor[not_for] Usage --> Caveats[caveats] Usage --> References[references] Project --> Name[name, display_name] Project --> Desc[description] Project --> Tags[tags, terms] Gateways --> Connection[connection] Gateways --> Scheduler[scheduler] Semantic --> StrictValidation[strict_semantic_validation] Semantic --> Rollups[enable_rollup, rollup_schema] Options --> Linter[linter] Options --> Notifications[notifications] Options --> Variables[variables] Options --> ExecHooks[execution_hooks] classDef primary-teal fill:#54DED1,color:#202F36,stroke:#009293,stroke-width:1.5px,font-weight:600; classDef surface fill:#FFFFFF,color:#242422,stroke:#242422,stroke-width:1px; class Config,Usage primary-teal; class Project,Gateways,StateConn,ModelDefaults,Semantic,Options,GoodFor,NotFor,Caveats,References,Name,Desc,Tags,Connection,Scheduler,StrictValidation,Rollups,Linter,Notifications,Variables,ExecHooks surface; ``` Business-facing usage guidance (`good_for`, `not_for`, `caveats`, `references`) lives in a separate `usage.yaml` file (see [Usage guidance](/references/v1/resources/vulcan/configurations/usage.md)). An optional `agreement.md` holds free-form data-sharing agreement text, read verbatim (see [Agreement](/references/v1/resources/vulcan/configurations/agreement.md)). ## Configuration sections A walkthrough of each part of `config.yaml`, in the order you'll typically configure them. ### Project settings Project settings identify your project. They don't affect how Vulcan runs, but catalog tools rely on them for organization and discovery. Business-facing usage guidance belongs in `usage.yaml`, not in `config.yaml`. | Option | Description | Type | Required | | -------------- | ------------------------------------------------------------------------------------ | :-------------: | :------: | | `name` | Project identifier. Can also be set via `DATAOS_RESOURCE_NAME` env var. | string | Yes | | `description` | Project description. Has a placeholder default but is still validated as non-empty. | string | Yes | | `display_name` | Human-readable project name for UI/docs | string | No | | `discoverable` | Whether this product appears in catalog search | boolean | No | | `version` | Release version (SemVer 2.0, e.g. `0.1.2`) | string | No | | `alignment` | Data Mesh orientation: `source_aligned` or `consumer_aligned` | enum | No | | `tags` | Labels for categorization and filtering. Merged with `DATAOS_RESOURCE_TAGS` env var. | array of string | No | | `terms` | Business glossary terms using dot notation (e.g., `glossary.data_product`) | array of string | No | ```yaml # Project identity name: orders-analytics display_name: Orders Analytics Platform description: Orders Analytics delivers insights across the full order lifecycle. # Catalog metadata discoverable: true version: 0.1.2 alignment: consumer_aligned # Classification tags: - e-commerce - retail - sales_analytics terms: - glossary.data_product - glossary.analytics_platform - glossary.sales_operations ``` {% hint style="info" %} **Tenant comes from the environment** `tenant` is required by the platform, but it is not a YAML key in `config.yaml`. In production, the platform injects it through `DATAOS_TENANT_ID`. For local development, export it before running Vulcan: ```bash export DATAOS_TENANT_ID=marketing ``` Without `DATAOS_TENANT_ID`, Vulcan refuses to load the project. {% endhint %} ### Gateways Gateways define how Vulcan connects to your data warehouse and state backend. Define multiple gateways for different environments: dev, staging, prod. Each gateway has its own connection settings. `connection.type` accepts `postgres`, `snowflake`, `databricks`, `spark`, `trino`, `mssql` (SQL Server), or `fabric` (Microsoft Fabric, built on the `mssql` adapter). See [Engine Guides](/references/v1/engine-guide/engine-guide.md) for the complete technical reference. | Component | Description | Type | Required | | ----------------- | ------------------------------------------- | :----: | :------: | | `connection` | Primary data warehouse connection | object | Yes | | `scheduler` | Scheduler configuration | object | No | | `default_gateway` | Which gateway to use when none is specified | string | No | ```yaml # Gateway Connection gateways: default: connection: type: postgres host: warehouse port: 5432 database: warehouse user: vulcan password: "{{ env_var('DB_PASSWORD') }}" default_gateway: default ``` ### State connection Vulcan needs somewhere to persist plan/run state (applied model versions, environments, snapshots). Configure it with `state_connection`, using the same connection shape as a gateway `connection`: ```yaml state_connection: type: duckdb database: ./.state/vulcan.db ``` If `state_connection` is omitted, local runs default to DuckDB at `.vulcan/state.db`, resolved against the project root. `vulcan init` adds `.vulcan/` to `.gitignore`, since the default state database is local, disposable state rather than something to commit. A gateway can also override where its own state lives with `gateways..state_connection`, `state_schema`, or `rollup_schema` (see [Gateway and connection configuration](#gateway-and-connection-configuration) below). ### Model defaults The `model_defaults` section is required. At minimum, specify `dialect` to tell Vulcan what SQL dialect your models use. Other defaults are optional but apply to all models automatically, so you don't repeat the same settings in every model file. ```yaml model_defaults: dialect: postgres # Required owner: data-team start: 2024-01-01 cron: '@daily' ``` See [Model Defaults](/references/v1/resources/vulcan/configurations/model-defaults.md) for all available options. ### Semantic layer Two project-level flags control how strictly the semantic layer validates itself and whether it materializes rollups: ```yaml strict_semantic_validation: false # default enable_rollup: false # default rollup_schema: rollup # schema rollups are materialized into allow_reciprocal_joins: true # default metric_join_path: directed # default ``` * `strict_semantic_validation` defaults to `false`: shape violations in filter, segment, and calculated-measure expressions are logged, not blocked, so a project can load with warnings instead of failing. Set it to `true` to hard-fail on those violations instead. * `enable_rollup` defaults to `false`. Semantic-model `rollups:` blocks always validate, but only materialize a physical pre-aggregation table when this flag is `true`. * `rollup_schema` sets the physical schema rollups are materialized into; it factors into the physical-table naming budget (see [Rollups](/references/v1/resources/vulcan/models/semantic-models/rollups.md)). * `allow_reciprocal_joins` defaults to `true`: two semantic models may declare inverse joins to each other, as long as each reciprocal pair has inverse join types and exactly one side sets `skip_for_bi`. Set it to `false` to reject reciprocal pairs as cycles instead (see [Joins](/references/v1/resources/vulcan/models/semantic-models/joins.md#reciprocal-joins)). * `metric_join_path` sets the default join-path mode for metrics that reference multiple semantic models: `directed` (default) or `connected`. Override it per metric with `join_path:` (see [Join path mode](/references/v1/resources/vulcan/models/business-metrics.md#join-path-mode-connected-vs-directed)). ### Variables Store sensitive information like passwords and API keys without hardcoding them. Use environment variables, `.env` files, or configuration overrides. Variables also let you override configuration values dynamically. ```yaml variables: warehouse_schema: analytics refresh_window_days: 7 gateways: default: variables: warehouse_schema: analytics_dev # override per gateway ``` See [Variables](/references/v1/resources/vulcan/configurations/variables.md) for details. ### Execution hooks Run SQL statements automatically at the start and end of `vulcan plan` and `vulcan run` commands. Use `before_all` for setup tasks like creating temporary tables or granting permissions. Use `after_all` for cleanup or post-processing. A hook entry is inline SQL, a macro, or a file containing SQL statements. ```yaml before_all: - GRANT SELECT ON ALL TABLES IN SCHEMA analytics TO reporting_role - file: ./statements/setup.sql after_all: - ANALYZE analytics.daily_sales - ./statements/cleanup.sql ``` See [Execution Hooks](/references/v1/resources/vulcan/configurations/execution-hooks.md) for detailed examples and use cases. ### Linter Automatic code quality checks that run when you create a plan or run the lint command. Catches common mistakes and enforces coding standards. Use built-in rules or create custom ones. See [Linter](/references/v1/resources/vulcan/configurations/linter.md) for rules and custom linter configuration. ### Notifications Set up alerts via Slack, Teams webhook, email, or console targets. Get notified when plans start or finish, when runs complete, or when data quality checks fail. Data quality events use the `dq_*` names. ```yaml notification_targets: - type: teams_webhook url: "{{ env_var('TEAMS_WEBHOOK_URL') }}" notify_on: - apply_failure - run_failure - dq_failure - type: console notify_on: - plan_change ``` See [Notifications](/references/v1/resources/vulcan/configurations/notifications.md) for Teams webhook, Slack, API, and email setup. ### Auth extension hook Use the root-level `after_authorize` field when a data product needs to resolve user groups and claims after DataOS authorization. The hook points to a Python function in your project. Put plugin modules in a `plugins/` package at the project root: {% hint style="warning" %} **Required for auth-backed policies** If you are working with auth-backed policies or masking, make sure `config.yaml` includes this root-level hook: ```yaml after_authorize: "plugins.auth_ext:resolve_user_groups" ``` {% endhint %} ``` plugins/ ├── __init__.py └── auth_ext.py ``` ```yaml after_authorize: "plugins.auth_ext:resolve_user_groups" ``` ```python from __future__ import annotations from schema.auth import AuthExtensionContext, SecurityContext ROLE_ID_TAG_PREFIX = "roles:id:" GROUP_DELIMITER = "," POLICY_GROUP_PRIORITY = ("operator", "developer") async def resolve_user_groups(ctx: AuthExtensionContext) -> SecurityContext: """Derive policy groups from DataOS role tags.""" groups = [ tag.replace(ROLE_ID_TAG_PREFIX, "", 1) for tag in ctx.user_tags if tag.startswith(ROLE_ID_TAG_PREFIX) ] group = next( (policy_group for policy_group in POLICY_GROUP_PRIORITY if policy_group in groups), groups[0] if groups else "", ) return SecurityContext(group=group, groups=GROUP_DELIMITER.join(groups)) ``` ### Deployment The `deployment:` block drives `vulcan create_deploy_yaml`, which renders the DataOS `domain-resource.yaml` deterministically: every value in the output is either derived from your project, defaulted, or set explicitly here. There are no placeholders left to fill in by hand. ```yaml deployment: repo: url: git@bitbucket.org:org/repo.git base_dir: repo/vulcan-project compute: general-purpose-shared run_as_user: svc-user engine: mssql # required when the gateway connection type is `depot` ``` `repo.url`, `repo.base_dir`, `compute`, and `run_as_user` are required; everything else has a default or is derived (`engine` from the gateway connection type, crons from your models). {% hint style="warning" %} **`engine` is required for `depot` connections** When `gateways..connection.type` is `depot`, the engine can't be inferred from the depot address, so `deployment.engine` must be set explicitly. For direct connections (`mssql`, `postgres`, `fabric`, and so on), `engine` is derived automatically and should be omitted. Allowed values: `bigquery`, `databricks`, `fabric`, `mssql`, `mysql`, `postgres`, `snowflake`, `spark`, `trino`. {% endhint %} ## Complete configuration reference This section lists every key `config.yaml` supports, grouped by area, with type, requirement, default, and guidance on when to set it versus when to leave it out. ### Required for a valid project Vulcan refuses to load a project unless these are satisfied: | Key | Why it's required | | ------------- | ------------------------------------------------------------------------- | | `name` | Identifies the project/resource. Needed by the schema. | | `tenant` | Required unless the runtime environment already supplies it. | | `domain` | Required once `tenant` is present. | | `description` | Must be non-empty; used for discovery and documentation. | | `users` | Must contain at least one user; models and ownership validate against it. | ### Project identity | Configuration Key | Description | Type | Required | Default | When to use / skip | | ----------------- | ----------------------------------------------------------------------------------------------------------- | :-----: | :-------: | ----------------------------------------- | ------------------------------------------------------------------------------------- | | `name` | Project identifier (used for resource naming). | string | **Yes** | - | Always required. | | `tenant` | Tenant the project belongs to. | string | **Yes**\* | - | Set explicitly for local/dev work; often supplied by the runtime environment instead. | | `domain` | Business domain the project is grouped under. | string | **Yes**\* | - | Required once `tenant` is present. | | `description` | Project description and purpose. Validated as non-empty. | string | **Yes** | placeholder | Always required. | | `display_name` | Human-readable name for UI/docs | string | No | `null` | Use when `name` isn't human-friendly enough for UI/catalog display. | | `discoverable` | Whether the product is listed in catalog search | boolean | No | `true` | Turn off while a product is still internal-only or unfinished. | | `version` | Release version (SemVer 2.0) | string | No | `"0.0.0"` | Set once you track releases; must be valid SemVer. | | `alignment` | Data Mesh orientation (`source_aligned` or `consumer_aligned`) | enum | No | `consumer_aligned` | Use if your org tracks Data Mesh alignment. | | `project` | Legacy alias of `name`. Auto-filled from `name` if omitted. | string | No | `""` | Only for projects already using this legacy field. | | `tags` | Labels for categorization. Merged with `DATAOS_RESOURCE_TAGS`. | array | No | `[]` | Use when catalog search/filtering needs them. | | `terms` | Business glossary terms (e.g., `glossary.data_product`) | array | No | `[]` | Use if a business glossary exists to link into. | | `status` | Lifecycle status: `Draft` or `Active` | enum | No | - | Use to flag work-in-progress projects before they go live. | | `usage_path` | Path to the usage sidecar file. | string | No | `usage.yml`, falling back to `usage.yaml` | Use only if your usage file lives at a non-default path. | | `agreement_path` | Path to the optional data-sharing agreement sidecar file, read verbatim, if not the default `agreement.md`. | string | No | `agreement.md` | Use only if your agreement file lives at a non-default path. | \* `tenant` and `domain` become required together, per the rules above. ### Users | Configuration Key | Description | Type | Required | Default | When to use / skip | | ------------------------------ | ------------------------------------------------------------------------------------ | :----: | :------: | ------- | -------------------------------------------------------------------- | | `users` | List of DataOS users for the project. **At least one entry is required.** | array | **Yes** | - | Always required. | | `users[].username` | DataOS username. Must be a valid user in the current tenant. Validated at load time. | string | **Yes** | - | Always required per entry. | | `users[].github_username` | GitHub username for the user. | string | No | - | Use to link commits/PRs to this person. | | `users[].slack_username` | Slack username for the user. | string | No | - | Use to route Slack notifications to individuals. | | `users[].email` | Email address for notifications | string | No | - | Use when email-based notifications are configured. | | `users[].roles` | Fine-grained roles for the user on this project. | array | No | - | Use when `type` alone isn't granular enough. | | `users[].notification_targets` | Notification targets specific to this user. | array | No | - | Use when a specific user needs their own notification routing. | | `users[].type` | Role of the user in this project: `OWNER` or `CONTRIBUTOR` | string | No | - | Use to distinguish accountable owners from contributors. | | `username` | Single user to receive notifications | string | No | `""` | Use for a simple fallback recipient, separate from per-user targets. | ### Compliance, support, and agreements | Configuration Key | Description | Type | Required | Default | When to use / skip | | ----------------- | --------------------------------------------------------------------- | :----: | :------: | ------- | ------------------------------------------------------------------------------------------------------------------------------ | | `issue_tracker` | Link to where bugs/requests for this product are filed. | string | No | `null` | Use when a dedicated tracker exists. | | `compliance` | Compliance frameworks that apply (GDPR, SOC2, HIPAA, etc.). | array | No | `[]` | Use when the product is subject to specific frameworks. | | `support` | Support channel/contact information for consumers. | object | No | `null` | Use when consumers need a defined support path. | | `sla` | Uptime/response-time commitments. | object | No | `null` | Use when you've committed to a formal SLA. | | `license` | License terms (e.g., `proprietary`, `internal-use`). | string | No | `null` | Use when licensing needs to be declared. | | `data_holder` | Who is accountable for the underlying data. | string | No | `null` | Use when ownership isn't already clear from `users`. | | `usage` | Inline usage guidance, as an alternative to a sidecar `usage.yaml`. | object | No | `null` | Prefer the sidecar `usage.yaml` file instead (see [Usage guidance](/references/v1/resources/vulcan/configurations/usage.md)). | | `agreement` | Inline agreement text, as an alternative to a sidecar `agreement.md`. | string | No | `null` | Prefer the sidecar `agreement.md` file instead (see [Agreement](/references/v1/resources/vulcan/configurations/agreement.md)). | ### Gateway and connection configuration | Configuration Key | Description | Type | Required | Default | | ---------------------------------- | ---------------------------------------------------------------------------------------------------------- | :-----: | :-------: | ----------------------------------- | | `gateways` | Gateway configurations for different environments | object | **Yes**\* | `{"": {}}` | | `gateways..connection` | Primary data warehouse connection | object | **Yes** | - | | `gateways..state_connection` | Where this gateway's state is stored, if different from the main `state_connection`. | object | No | Uses `state_connection` | | `gateways..test_connection` | Lightweight connection for unit tests (e.g., DuckDB) instead of prod. | object | No | Uses `connection` | | `gateways..scheduler` | Scheduler configuration | object | No | Built-in (`BuiltInSchedulerConfig`) | | `gateways..state_schema` | Schema where state tables are stored for this gateway. | string | No | Default schema | | `gateways..rollup_schema` | Schema where metric rollups are materialized for this gateway. | string | No | Default schema | | `gateways..variables` | Gateway-specific variables | object | No | `{}` | | `gateways..model_defaults` | Model defaults scoped to this gateway, overriding the project-wide ones. | object | No | Project `model_defaults` | | `default_gateway` | Name of the default gateway | string | No | `""` | | `default_connection` | Root-level default connection | object | No | `null` | | `default_test_connection` | Root-level default test connection. | object | No | `null` | | `default_scheduler` | Root-level default scheduler | object | No | Built-in (`BuiltInSchedulerConfig`) | | `gateway_managed_virtual_layer` | Whether virtual-layer views are created by the model's own gateway | boolean | No | `false` | | `state_connection` | Root-level connection where Vulcan persists plan/run state, if you don't want to configure it per gateway. | object | No | DuckDB at `.vulcan/state.db` | \* At least one gateway with a `connection` is required. ### Model configuration | Configuration Key | Description | Type | Required | Default | | -------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-----------: | :-------: | -------------------------- | | `model_defaults` | Default values applied to all models | object | **Yes**\* | `{}` | | `model_defaults.dialect` | SQL dialect (postgres, snowflake, bigquery, etc.) | string | **Yes** | - | | `model_defaults.owner` | Default owner for all models. If not set, defaults to the first `config.users` entry. Must be a username listed in `config.users`; config load fails if the value does not match. | string | No | First `config.users` entry | | `model_defaults.stamp` | Arbitrary value used to force a version bump across models. | string | No | `null` | | `model_defaults.start` | Default start date for backfilling | string | No | Inferred | | `model_defaults.cron` | Default cron schedule (e.g., `@daily`) | string | No | `null` | | `model_defaults.kind` | Default model kind (FULL, INCREMENTAL, etc.) | string/object | No | `VIEW` | | `model_defaults.interval_unit` | Temporal granularity of data intervals | string | No | From cron | | `model_defaults.batch_concurrency` | Max concurrent batches for incremental models | integer | No | `1` | | `model_defaults.table_format` | Table format (iceberg, delta, hudi) | string | No | `null` | | `model_defaults.storage_format` | Storage format (parquet, orc) | string | No | `null` | | `model_defaults.on_destructive_change` | Action on destructive schema changes | string | No | `error` | | `model_defaults.on_additive_change` | Action on additive schema changes | string | No | `apply` | | `model_defaults.physical_properties` | Properties for physical tables/views | object | No | `{}` | | `model_defaults.virtual_properties` | Properties for virtual layer views | object | No | `{}` | | `model_defaults.session_properties` | Engine-specific session properties | object | No | `{}` | | `model_defaults.audits` | Assertion/assertion functions for all models (alias: `assertions`) | array | No | `[]` | | `model_defaults.optimize_query` | Whether to optimize SQL queries | boolean | No | `true` | | `model_defaults.allow_partials` | Whether models can process incomplete intervals | boolean | No | `false` | | `model_defaults.enabled` | Whether models are enabled by default | boolean | No | `true` | | `model_defaults.formatting` | Whether SQL is auto-formatted by default. | boolean | No | `true` | | `model_defaults.pre_statements` | SQL statements executed before model runs | array | No | `null` | | `model_defaults.post_statements` | SQL statements executed after model runs | array | No | `null` | | `model_defaults.on_virtual_update` | Statements run whenever the virtual layer updates. | array | No | `null` | \* The `model_defaults.dialect` field is required. ### Semantic layer | Configuration Key | Description | Type | Required | Default | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-----: | :------: | ---------- | | `strict_semantic_validation` | Hard-fail on semantic expression shape violations instead of only logging them | boolean | No | `false` | | `enable_rollup` | Materialize semantic-model `rollups:` blocks as physical pre-aggregation tables | boolean | No | `false` | | `rollup_schema` | Physical schema rollups are materialized into | string | No | `rollup` | | `allow_reciprocal_joins` | Whether two semantic models may declare inverse joins to each other. When enabled, each reciprocal pair must have inverse join types and exactly one `skip_for_bi`. | boolean | No | `true` | | `metric_join_path` | Default join-path mode for metrics that reference multiple semantic models: `directed` or `connected`. | string | No | `directed` | ### Variables and environment | Configuration Key | Description | Type | Required | Default | | ----------------- | -------------------------------------- | :----: | :------: | ------- | | `variables` | Root-level variables for models/macros | object | No | `{}` | | `env_vars` | Environment variable overrides | object | No | `{}` | ### Execution hooks | Configuration Key | Description | Type | Required | Default | | ----------------- | ------------------------------------------------------------------------ | :---: | :------: | ------- | | `before_all` | SQL statements, macros, or statement files executed at start of plan/run | array | No | `null` | | `after_all` | SQL statements, macros, or statement files executed at end of plan/run | array | No | `null` | ### Code quality and linting | Configuration Key | Description | Type | Required | Default | | ------------------- | -------------------------------------- | :-----: | :------: | ------------------ | | `linter` | Linting configuration | object | No | `{enabled: false}` | | `linter.enabled` | Enable or disable linting | boolean | No | `false` | | `linter.rules` | List of rules to enforce (error level) | array | No | `[]` | | `linter.warn_rules` | List of rules to warn about | array | No | `[]` | ### Notifications and users | Configuration Key | Description | Type | Required | Default | | ---------------------- | ------------------------------------------------------------------- | :---: | :------: | ------- | | `notification_targets` | List of notification targets (Teams webhook, Slack, email, console) | array | No | `[]` | See [Users](#users) above for `users` and `username`. ### Environment and schema management | Configuration Key | Description | Type | Required | Default | | ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-----: | :------: | ------------------ | | `vde` (alias: `virtual_data_environment`) | Turn full Virtual Data Environments on/off. `true` = versioned physical tables + virtual layer; `false` = simple mode. Defaults to `false`: enable it explicitly when you want VDE. Not supported for `spark` and `trino` gateways: validation rejects `vde: true` on those. Replaces the deprecated `virtual_environment_mode`. | boolean | No | `false` | | `default_target_environment` | Default environment for plan/run commands | string | No | `prod` | | `snapshot_ttl` | Time before unused snapshots are deleted | string | No | `in 1 week` | | `environment_ttl` | Time before dev environments are deleted | string | No | `in 1 week` | | `pinned_environments` | Environments not deleted by janitor | array | No | `[]` | | `physical_schema_mapping` | Map model patterns (regex) to physical schema names. Replaces the deprecated `physical_schema_override`, which is auto-converted with a warning. | object | No | `{}` | | `environment_suffix_target` | Where to append environment name (`schema`, `table`, `catalog`) | enum | No | `schema` | | `environment_catalog_mapping` | Route environments to specific catalogs (e.g., dev models go to `dev_catalog`, prod to `prod_catalog`). Useful in multi-catalog setups where each environment writes to a different database. | object | No | `{}` | | `physical_table_naming_convention` | How to name tables at the physical layer | enum | No | `schema_and_table` | | `gateway_managed_virtual_layer` | Whether virtual-layer views are created by the model's own gateway | boolean | No | `false` | | `ignore_patterns` | Glob patterns for files to ignore | array | No | Standard list | | `time_column_format` | Default format for model time columns | string | No | `%Y-%m-%d` | {% hint style="success" %} **Catalog in model names vs. environment catalog mapping** There are two ways to control which catalog your models land in: * **Model-level:** Use a three-part name (`catalog.schema.model`) in your MODEL definition to target a specific catalog for that model. See [model name property](/references/v1/resources/vulcan/models/data-models/properties.md#name). * **Environment-level:** Use `environment_catalog_mapping` to route all models in a given environment to a specific catalog, without changing individual model names. Model-level catalog takes precedence. If you set both, the catalog in the model name wins. ```yaml environment_catalog_mapping: dev: dev_catalog staging: staging_catalog prod: prod_catalog ``` {% endhint %} ### Project and runtime management | Configuration Key | Description | Type | Required | Default | | ------------------------------ | ----------------------------------------------------------------------------------------------- | :-----: | :------: | -------------- | | `log_limit` | Default number of logs to keep | integer | No | `20` | | `cache_dir` | Directory for Vulcan's compiled project cache | string | No | `.cache` | | `loader` | Loader class for loading project files | class | No | Default loader | | `loader_kwargs` | Arguments to pass to loader instance | object | No | `{}` | | `infer_python_dependencies` | Auto-detect Python package requirements | boolean | No | `true` | | `run` | Run command configuration | object | No | Default | | `plan` | Plan command configuration | object | No | Default | | `plan.auto_categorize_changes` | Auto-categorize changes as breaking/non-breaking. Replaces top-level `auto_categorize_changes`. | object | No | Default | | `plan.include_unmodified` | Include unmodified models in the plan output. Replaces top-level `include_unmodified`. | boolean | No | `false` | | `plan.use_finalized_state` | Use finalized state when creating plans. Requires `vde: true`. | boolean | No | `false` | | `format` | SQL formatting options | object | No | Default | | `ui` | UI server configuration | object | No | Default | | `migration` | Migration configuration | object | No | Default | | `janitor` | Cleanup task configuration | object | No | Default | | `model_naming` | Name inference rules for models | object | No | Default | ### Deployment Use `deployment` when the project is deployed as a DataOS resource rather than run locally/ad hoc; see [Deployment](#deployment) above for the walkthrough. | Configuration Key | Description | Type | Required | Default | | ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | :----: | :-------: | ---------------------------- | | `deployment` | Deployment configuration for the project. | object | No | `null` | | `deployment.repo.url` | Git repository URL for the deploy manifest. | string | **Yes**\* | - | | `deployment.repo.base_dir` | Base directory within the repo to load the project from. | string | **Yes**\* | - | | `deployment.repo.secret_id` | Reference to stored credentials for private repos. | string | No | `null` | | `deployment.repo.sync_flags` | Extra flags controlling how the repo is synced. | array | No | `[]` | | `deployment.compute` | Compute resource the deployment runs on. | string | **Yes**\* | - | | `deployment.run_as_user` | User the deployment runs as. | string | **Yes**\* | - | | `deployment.type` | Deployment resource type (e.g., `workflow`). | string | No | `null` | | `deployment.engine` | Compute engine used (e.g., `spark`). Required when the gateway connection is `type: depot`; derived automatically (and should be omitted) for direct connections. | string | No | Derived from connection type | | `deployment.state_store_connection` | State store connection reference for the deployment. | string | No | `null` | | `deployment.object_store_connection` | Object store connection reference for the deployment. | string | No | `null` | | `deployment.depots` | Depots the deployment has access to. | array | No | `[]` | | `deployment.schedule` | Cron schedule for the deployment. | string | No | `null` | | `deployment.workflow` | Workflow-specific settings. | object | No | `{}` | | `deployment.api` | API-specific settings. | object | No | `{}` | | `deployment.use` | Additional resource references to attach. | object | No | `{}` | | `deployment.spark_conf` | Spark configuration overrides. | object | No | `{}` | | `deployment.trino` | Trino-specific configuration. | object | No | `{}` | \* Required only when `deployment` is used (or when you run `vulcan create_deploy_yaml`). ### Integrations and external services | Configuration Key | Description | Type | Required | Default | | ---------------------------------------- | ---------------------------------------------------------------------------------------------------------- | :-----: | :------: | --------------------------------------------------------------- | | `dbt` | DBT-specific configuration | object | No | `null` | | `object_store` | Object storage for query results (MinIO/S3/GCS/Azure) | object | No | `null` | | `object_store.provider` | Object storage provider: `s3`, `gcs`, `azure`, or `minio`. | string | No | `null` | | `object_store.bucket` | Bucket name for storing query results. | string | No | `null` | | `object_store.base_path` | Base path within the bucket. | string | No | `null` | | `object_store.compression` | Compression codec: `snappy`, `gzip`, or `zstd`. | string | No | `null` | | `object_store.presigned_url_expiry_mins` | How long presigned URLs to results remain valid. | integer | No | `null` | | `object_store.credentials` | Credentials for the object store provider. | object | No | `null` | | `transpiler` | External transpiler service | object | No | `{base_url: "http://127.0.0.1:8100", timeout: 30, token: null}` | | `pgq` | PostgreSQL Queue for async jobs | object | No | Default | | `prometheus` | Prometheus metrics export configuration. | object | No | `null` | | `analytics` | CloudEvents telemetry configuration. Replaces the deprecated `disable_anonymized_analytics`. | object | No | `{enabled: false}` | | `analytics.enabled` | Enable telemetry publishing | boolean | No | `false` | | `analytics.api_key` | Telemetry API key. Required when `analytics.enabled: true`. | string | No | `null` | | `openlineage` | OpenLineage data lineage integration | object | No | `null` | | `after_authorize` | Auth extension hook called after DataOS authorization, for example `plugins.auth_ext:resolve_user_groups`. | string | No | `null` | | `hera` | Hera/OpenMetadata sync configuration | object | No | `{enabled: false}` | | `hera.enabled` | Enable Hera/OpenMetadata sync | boolean | No | `false` | | `hera.url` | Hera service URL. Required when `hera.enabled: true`. | string | No | `null` | | `hera.token` | Hera auth token. Required when `hera.enabled: true`. | string | No | `null` | {% hint style="info" %} **Object store: GCS and Azure supported alongside S3** `object_store`'s secret projection accepts `STORAGE_FORMAT: gcs` and `STORAGE_FORMAT: azure` in addition to `s3` (matching is case-insensitive), each with a `RELATIVE_PATH` key. No project change is needed if you're already on S3. See [State management](/references/v1/resources/vulcan/concepts/state-management.md) for why Vulcan needs an object store and what it keeps there. {% endhint %} {% hint style="info" %} **Fabric and Azure SQL now labeled correctly in the Datasets page** `hera`'s service-type mapping now identifies `fabric` and `azuresql` connection types, so datasets backed by those gateways show up as "Microsoft Fabric" in the Datasets catalogue view instead of a generic label. {% endhint %} {% hint style="info" %} **GraphQL is served in-process** GraphQL is served in-process at `/semantic/graphql`: there's no separate `graphql` config block, sidecar, or process to run. Execution tunables are set via environment variables instead: `GRAPHQL_POLL_*`, `RETRY_AFTER`, and `DEFAULT_TIMEZONE`. If your `config.yaml` still has a `graphql` (or `GraphQLConfig`) block, remove it and set the equivalent env vars. {% endhint %} ### Minimal valid configuration The non-skippable parts of `config.yaml` are: a non-empty `name`, a non-empty `description`, at least one working `gateways..connection`, `model_defaults.dialect`, and at least one entry in `users`. The runtime also needs `DATAOS_TENANT_ID` in the environment. ```yaml name: my-project description: My project description users: - username: jane email: jane@example.com type: OWNER gateways: default: connection: type: postgres host: localhost port: 5432 database: mydb user: myuser password: mypass model_defaults: dialect: postgres ``` ```bash # Required at runtime, not in YAML export DATAOS_TENANT_ID=my-tenant ``` Everything else has a default, so you can omit it. ## Validation rules Some fields become required only when another field is enabled: * `name` must be non-empty (or supplied via `DATAOS_RESOURCE_NAME`). * `description` must be non-empty. * `users` must contain at least one entry. Config load fails if the list is empty or the key is omitted entirely. * `users[].username` values must be valid DataOS usernames in the current tenant. Models referencing an unknown username are rejected at load time. * `model_defaults.owner`, when set explicitly, must be a username present in `config.users`. Config load fails if it references an unknown username. * Each model's `owner` field is validated against `config.users` at load time. Models referencing a username not listed in `config.users` are rejected. * `hera.url` and `hera.token` are required when `hera.enabled: true`. * `analytics.api_key` is required when `analytics.enabled: true`. * `vde: true` is rejected for `spark` and `trino` gateway types. * `version` must be valid SemVer 2.0 (e.g. `0.1.2`, `1.0.0-rc.1`). ## Environment variables A few values come from the shell or `.env`, not from YAML: | Variable | Effect | | ---------------------- | -------------------------------------------------------------------------------------- | | `DATAOS_TENANT_ID` | Required at runtime. Supplies the `tenant`. Not a YAML key. | | `DATAOS_RESOURCE_NAME` | Overrides `name` from `config.yaml`. | | `DATAOS_RESOURCE_TAGS` | Merged into `tags` from `config.yaml`. | | `TEAMS_WEBHOOK_URL` | Recommended source for Teams webhook notification target URLs. | | `GRAPHQL_POLL_*` | GraphQL execution polling tunables (replaces the old `graphql`/`GraphQLConfig` block). | | `RETRY_AFTER` | GraphQL retry-after interval. | | `DEFAULT_TIMEZONE` | Default timezone for GraphQL execution. | ## Migration from the legacy schema If you have an older `config.yaml`, these keys moved or changed: | Old key | Replacement | Notes | | ---------------------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------------ | | `virtual_environment_mode: full` | `vde: true` | Old string values fail validation. | | `virtual_environment_mode: dev_only` | `vde: false` (or omit) | `vde` defaults to `false`. | | `auto_categorize_changes` (top-level) | `plan.auto_categorize_changes` | Now nested under `plan`. | | `include_unmodified` (top-level) | `plan.include_unmodified` | Now nested under `plan`. | | `physical_schema_override` | `physical_schema_mapping` | Auto-converted with a warning. | | `disable_anonymized_analytics` | `analytics.enabled` | Move into the `analytics` block. | | `tenant` (in YAML) | `DATAOS_TENANT_ID` env var | No longer a YAML key. | | `check_start`, `check_end`, `check_failure` | `dq_start`, `dq_end`, `dq_failure` | Data quality notification events now use `dq_*` names. | | `metadata` (in `config.yaml`) | `usage.yaml` | Move business usage guidance out of runtime config. | | `graphql` / `GraphQLConfig` (in `config.yaml`) | `GRAPHQL_POLL_*`, `RETRY_AFTER`, `DEFAULT_TIMEZONE` env vars | GraphQL sidecar removed; served in-process at `/semantic/graphql`. | Quick migration checklist: 1. Replace `virtual_environment_mode: full` with `vde: true`. 2. Remove `virtual_environment_mode: dev_only` (or set `vde: false` explicitly). 3. Add `discoverable`, `version`, `alignment` near the top of the file if you want non-default values. 4. Make sure `version` is valid SemVer (`0.1.2`, not `0.1` or `v0.1.2`). 5. Move business usage guidance from `metadata:` into `usage.yaml`. See [Usage guidance](/references/v1/resources/vulcan/configurations/usage.md). 6. Move any DataOS auth extension hook to root-level `after_authorize`. 7. Replace `check_*` notification events with `dq_*` event names. 8. Remove any deprecated keys listed above. 9. Set `DATAOS_TENANT_ID` in your shell or `.env`. 10. Remove any `graphql`/`GraphQLConfig` block and set `GRAPHQL_POLL_*`, `RETRY_AFTER`, `DEFAULT_TIMEZONE` env vars if you relied on GraphQL execution tunables. ## Best practices Use environment variables for sensitive data like passwords and API keys. This keeps secrets out of your config files and makes it easier to manage different environments. Set meaningful defaults in `model_defaults` to reduce boilerplate. If most of your models use the same dialect, start date, or cron schedule, set it once here instead of repeating it everywhere. Enable linting to catch common errors early in development. Fix issues before they make it to production. Use multiple gateways for different environments: dev, staging, prod. Test changes safely before deploying to production, with a different database configuration for each environment. ## Related pages * [Usage guidance](/references/v1/resources/vulcan/configurations/usage.md): what `usage.yaml` is for and what its keys mean. * [Agreement](/references/v1/resources/vulcan/configurations/agreement.md): what `agreement.md` is for and how Vulcan reads and surfaces it. * [Model Defaults](/references/v1/resources/vulcan/configurations/model-defaults.md) * [Variables](/references/v1/resources/vulcan/configurations/variables.md) * [Execution Hooks](/references/v1/resources/vulcan/configurations/execution-hooks.md) * [Linter](/references/v1/resources/vulcan/configurations/linter.md) * [Notifications](/references/v1/resources/vulcan/configurations/notifications.md) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://v2.dataos.info/references/v1/resources/vulcan/configurations.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.