> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/engine-guide/snowflake.md). # Snowflake Snowflake is a cloud data warehouse with separated storage and compute. Vulcan runs against Snowflake to manage transformations with version control and gated deployments. This page is the full manual: it covers how you connect, what permissions you need, how each model kind materialises, how to set operational guardrails, and how to recover from the failures you are most likely to hit. Use the [Connect engine → Snowflake](https://v2.dataos.info/build/productize/connect-engine/snowflake) Build page for the quick connection steps. Use this page when you need depth. {% hint style="info" %} Engine adapter type: `snowflake`. Model dialect: `snowflake`. Recommended authentication: key-pair JSON Web Token (JWT). Tested Vulcan image: `tmdcio/vulcan-snowflake:0.228.1.23` on the Draco 1.38 series. {% endhint %} ## When to use Snowflake Choose Snowflake when your data already lands in a Snowflake account and you want elastic, per-query compute with minimal infrastructure to manage. Snowflake suits incremental warehouse builds, governed semantic serving, and mixed SQL plus Snowpark Python workloads. It is fully supported on Standard, Enterprise, and Business Critical editions, with a few features (Dynamic Tables, masking policies, row access policies, Semantic Views) requiring Enterprise or higher. ### Supported model kinds `FULL`, `SEED`, `VIEW`, `INCREMENTAL_BY_TIME_RANGE`, `INCREMENTAL_BY_UNIQUE_KEY`, `INCREMENTAL_BY_PARTITION`, and `MANAGED` (Dynamic Tables, Enterprise and above). ## Connection options Set these under `gateways..connection`. | Option | Description | Type | Required | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------- | ------ | ----------- | | `type` | Engine type name. Must be `snowflake`. | string | Yes | | `account` | The Snowflake account identifier in `-` format (for example, `myorg-myaccount`). Find it in your Snowflake console URL. | string | Yes | | `user` | The username Vulcan connects as. | string | Yes | | `password` | Password authentication. Use only where key-pair is not available. | string | Conditional | | `warehouse` | The warehouse used for running computations. | string | Yes | | `database` | The database to connect to. | string | Yes | | `role` | The role used for authentication. | string | No | | `authenticator` | The authenticator method (for example, `snowflake_jwt`, `externalbrowser`, `oauth`). | string | No | | `token` | The OAuth 2.0 access token. | string | No | | `private_key_path` | Path to the private key file for key-pair authentication. | string | No | | `private_key_passphrase` | Passphrase to decrypt the private key, if encrypted. | string | No | {% hint style="warning" %} Use `-` format for the account field. The legacy locator format fails. Always inject credentials from environment variables, for example `password: {{ env_var('SNOWFLAKE_PASSWORD') }}`. {% endhint %} ### Authentication methods * Key-pair JWT (recommended for production): set `authenticator: snowflake_jwt`, `private_key_path`, and `private_key_passphrase`. * Username and password: set `user` and `password`. * OAuth 2.0 token: set `token`. * External browser single sign-on (SSO): set `authenticator: externalbrowser`. * Role-based: set `role`. {% hint style="danger" %} OAuth and external-browser SSO are not the recommended path for the Vulcan gateway in production. Use key-pair JWT for service-account connections. {% endhint %} ## Permissions and grants Three roles are required, each with a distinct scope. Do not collapse them into one. | Role | Who holds it | Purpose | | ------------------- | ---------------------------- | ------------------------------------------------------------------------------------------ | | Admin role | Snowflake account admin | Provisions warehouses, creates resource monitors, manages masking and row access policies. | | Vulcan service role | The user Vulcan connects as | Runs models, creates and modifies tables and schemas in the target database. | | Consumer role | BI users, endpoint consumers | Reads data via Vulcan endpoints; no write access. | ### Admin role: one-time setup Run as `SYSADMIN` or `ACCOUNTADMIN`. ```sql -- Create warehouse CREATE WAREHOUSE vulcan_wh WITH WAREHOUSE_SIZE = 'MEDIUM' AUTO_SUSPEND = 60 AUTO_RESUME = TRUE; -- Attach resource monitor (required before production) ALTER WAREHOUSE vulcan_wh SET RESOURCE_MONITOR = vulcan_monthly_cap; ``` ### Vulcan service role: minimum grants ```sql -- Warehouse access GRANT USAGE ON WAREHOUSE TO ROLE ; -- Database access GRANT USAGE ON DATABASE TO ROLE ; -- Create objects (required for model materialisation) GRANT CREATE SCHEMA ON DATABASE TO ROLE ; GRANT USAGE, CREATE TABLE, CREATE VIEW ON FUTURE SCHEMAS IN DATABASE TO ROLE ; -- Key-pair auth on the user ALTER USER SET RSA_PUBLIC_KEY = ''; -- Bind role to user GRANT ROLE TO USER ; ``` ### Consumer role: read-only grants ```sql GRANT USAGE ON DATABASE TO ROLE ; GRANT USAGE ON SCHEMA . TO ROLE ; GRANT SELECT ON ALL TABLES IN SCHEMA . TO ROLE ; GRANT SELECT ON FUTURE TABLES IN SCHEMA . TO ROLE ; ``` ### Source schema versus target schema Snowflake access in DataOS splits into two areas. Grant read-only access on every source schema Vulcan reads from, and managed access on the schema where Vulcan materialises output. | Area | Purpose | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | Source schema (read-only) | Dataset discovery, profiling, data quality checks, semantic querying, reading upstream tables during model builds. | | Target schema (managed by Vulcan) | Tables and views created and updated by Vulcan during materialisation. | Source schema grants (read-only): ```sql GRANT USAGE ON WAREHOUSE TO ROLE ; GRANT USAGE ON DATABASE TO ROLE ; GRANT USAGE ON SCHEMA . TO ROLE ; GRANT SELECT ON ALL TABLES IN SCHEMA . TO ROLE ; GRANT SELECT ON FUTURE TABLES IN SCHEMA . TO ROLE ; GRANT SELECT ON ALL VIEWS IN SCHEMA . TO ROLE ; GRANT SELECT ON FUTURE VIEWS IN SCHEMA . TO ROLE ; ``` Target schema grants (managed by Vulcan): ```sql GRANT CREATE SCHEMA ON DATABASE TO ROLE ; GRANT USAGE ON SCHEMA . TO ROLE ; GRANT CREATE TABLE ON SCHEMA . TO ROLE ; GRANT CREATE VIEW ON SCHEMA . TO ROLE ; GRANT SELECT, INSERT, UPDATE, DELETE, TRUNCATE ON ALL TABLES IN SCHEMA . TO ROLE ; GRANT SELECT, INSERT, UPDATE, DELETE, TRUNCATE ON FUTURE TABLES IN SCHEMA . TO ROLE ; GRANT SELECT ON ALL VIEWS IN SCHEMA . TO ROLE ; GRANT SELECT ON FUTURE VIEWS IN SCHEMA . TO ROLE ; ``` Why each write privilege is needed: * `DELETE` is used for incremental rebuilds: Vulcan deletes existing rows for a partition or time slice, then inserts refreshed rows. It is required only on Vulcan-managed target tables, never on source schemas. * `TRUNCATE` is used for full refreshes, overwrite-style rebuilds, and complete table regeneration. * `OWNERSHIP` is needed only when DataOS must fully manage the target schema lifecycle, including `ALTER`, `REPLACE`, and `DROP`: ```sql CREATE SCHEMA IF NOT EXISTS .; GRANT OWNERSHIP ON SCHEMA . TO ROLE COPY CURRENT GRANTS; ``` ### Additional grants by feature | Feature | Additional grant | Edition | | -------------------------- | ---------------------------------------------------------------------------------------------- | ----------- | | Read Snowflake sample data | `GRANT IMPORTED PRIVILEGES ON DATABASE SNOWFLAKE_SAMPLE_DATA TO ROLE ` | Standard+ | | Masking policies | `GRANT CREATE MASKING POLICY ON SCHEMA . ...` plus `APPLY MASKING POLICY ON TABLE` | Enterprise+ | | Row access policies | `GRANT CREATE ROW ACCESS POLICY ON SCHEMA . ...` | Enterprise+ | | Snowpark external access | `GRANT USAGE ON INTEGRATION TO ROLE ` | Standard+ | | Dynamic Tables | `GRANT CREATE DYNAMIC TABLE ON SCHEMA . ...` | Enterprise+ | ### DataOS permissions Your DataOS operator must provision these before you deploy: | Permission | What it unlocks | | --------------------------------- | -------------------------------------------- | | `roles:id:data-dev` or equivalent | Create and apply Vulcan resources. | | Access to the target workspace | Apply secrets, depots, and domain resources. | | `depot:rw:` | Read/write access to the Snowflake depot. | | `depot:r:` | Read-only access (consumer). | | Git repository access | Vulcan pulls model code via git-sync. | Run `dataos-ctl get depot` to confirm your read access before you start. If the Snowflake depot is missing, request `depot:r` from your operator. ## Example gateway config Minimum local connection with key-pair JWT: ```yaml gateways: default: connection: type: snowflake account: - # from your Snowflake console URL user: authenticator: snowflake_jwt private_key_path: ./snowflake_key.p8 private_key_passphrase: warehouse: database: role: model_defaults: dialect: snowflake ``` Generate the key pair once per developer machine and register the public key on your Snowflake user. Keep the `.p8` file local and out of version control. ```bash openssl genrsa -out snowflake_key.pem 2048 openssl pkcs8 -topk8 -inform PEM -outform PEM \ -in snowflake_key.pem -out snowflake_key.p8 \ -v2 aes256cbc -passphrase pass: ``` ```sql ALTER USER SET RSA_PUBLIC_KEY = ''; ``` Validate the connection: ```bash vulcan migrate # initializes Vulcan state in Snowflake vulcan plan # dry run against your target DB; should succeed with no errors ``` ### Production deployment Snowflake has no local Docker setup. Production uses a depot instead of a direct connection. Apply resources in this order, because each step depends on the previous: ``` Snowflake key-pair Secret → Snowflake Depot → Git Secret → Vulcan config.yaml → Vulcan resource ``` | Manifest | Purpose | | ------------------------------- | ------------------------------------------------------------------------------------ | | `secret-snowflake-keypair.yaml` | Stores the Snowflake user, encrypted PEM, and passphrase. | | `depot-snowflake.yaml` | Registers URL, database, warehouse, account, and role; binds purposes to the secret. | | `secret-git-sync.yaml` | Repo credentials (`GITSYNC_USERNAME`, `GITSYNC_PASSWORD`) for git-sync. | | `config.yaml` | Gateway points at the depot (`type: depot`, `address: dataos://`). | | `domain-resource.yaml` | Workflow plus API; references `engine: snowflake`, depot, and repo. | A Snowflake depot is registered with one or more purposes. Bind the correct purpose to each secret: a `query` credential cannot create tables, and an `rw` credential handed to a consumer is an over-permission risk. | Purpose | Who uses it | What it allows | | ------- | ---------------------- | ---------------------------------------------------------------------------------- | | `rw` | Vulcan workflow | Read source tables, create and write model output, manage schemas. | | `scan` | Metadata scanner | Read `INFORMATION_SCHEMA` and `ACCOUNT_USAGE` to populate the catalog and lineage. | | `query` | Consumer direct access | Read-only access to Data Product tables. | {% hint style="warning" %} A misspelled depot name in the Vulcan resource passes `apply` but fails at every workflow run with `Depot not resolvable`. Verify that `dataos://?purpose=rw` matches the depot manifest's `name` field exactly. {% endhint %} ## Materialisation behaviour per model kind SQL models compile to native Snowflake SQL. Model kinds map to Snowflake operations as follows. | Model kind | Snowflake operation | Notes | | --------------------------- | ----------------------------------------------------- | --------------------------------------------------------------------------------- | | `FULL` | `CREATE OR REPLACE TABLE` | Rebuilds the table from scratch each run. | | `SEED` | Loads a file-backed reference dataset (commonly CSV). | For small reference tables. | | `VIEW` | `CREATE OR REPLACE VIEW` | Always a full rewrite; no partition or key tracking. | | `INCREMENTAL_BY_TIME_RANGE` | DELETE by time range, then INSERT | Deletes rows in the target window, then inserts refreshed rows. Re-runs are safe. | | `INCREMENTAL_BY_UNIQUE_KEY` | `MERGE` on the unique key | Updates existing rows by key or inserts new ones. | | `INCREMENTAL_BY_PARTITION` | DELETE by partitioning key, then INSERT | Partition-level consistency on reprocessing. | | `MANAGED` | `CREATE OR REPLACE DYNAMIC TABLE` | Requires Enterprise or higher. | ### Python (Snowpark) models Python models run via Snowpark. The model file declares its kind via `@model(...)` and the body returns a Snowpark DataFrame. {% hint style="warning" %} Import Snowpark functions (`col`, `lit`, `when`, and so on) inside `execute()`, not at module scope. Module-level imports raise `NameError: 'col' is not defined`. This is the most common Python-model error on Snowflake. {% endhint %} ```python def execute(context, start, end, execution_time, **kwargs): from snowflake.snowpark.functions import col, lit, when # INSIDE the function df = context.snowpark.table( context.resolve_table('"SNOWFLAKE_SAMPLE_DATA"."TPCH_SF1"."CUSTOMER"') ) return df.select(col("C_CUSTKEY"), col("C_NAME")) ``` Default to SQL. Reach for Snowpark when you need Python libraries, complex window expressions, or chained DataFrame operations. ### Semantic models, metrics, and identifier casing {% hint style="warning" %} Identifier casing is the rule that bites everyone once. Snowflake stores unquoted identifiers in UPPERCASE. Every column reference in a semantic model, every measure expression, segment filter, join clause, DQ SQL, and metric expression must be UPPERCASE. Lowercase causes `object does not exist`. {% endhint %} ```yaml # Snowflake: UPPERCASE column names everywhere dimensions: - USER_ID - SIGNUP_DATE - PLAN_TYPE measures: - name: active_users type: count filters: - "{users.STATUS} = 'active'" ``` `COUNT(DISTINCT …)` runs exactly on Snowflake (no automatic HyperLogLog downgrade) and is the single biggest warehouse-sizing driver on large facts. Cross-join-shaped semantic queries (joins without a foreign-key relation) run up to 30 times slower than grouped queries; model joins explicitly and pre-aggregate one side into a mart if the generated SQL contains a cross-join. ### Lifecycle DDL and macro rules {% hint style="warning" %} Wrap any DDL in `pre_statements` / `post_statements` that should only run during real execution. Without the guard, `vulcan plan` executes it. {% endhint %} ```sql @IF(@runtime_stage = 'evaluating', ALTER TABLE myproj.gold.orders CLUSTER BY (order_date, nation_name) ); ``` In `@macro()` functions, column arguments arrive as expression objects, not strings. Render them with `.sql(dialect=evaluator.dialect)`, and do not annotate parameters as `: str` (this raises `Coercion failed`). Keep an empty `macros/__init__.py` or macros are not discovered. ### Cross-database external models `vulcan create_external_models` inspects only the gateway database. For cross-database sources (for example, `SNOWFLAKE_SAMPLE_DATA.TPCH_SF1`), declare them manually with the triple-quoted identifier form: ```yaml - name: '"SNOWFLAKE_SAMPLE_DATA"."TPCH_SF1"."ORDERS"' columns: o_orderkey: NUMBER o_custkey: NUMBER o_orderdate: DATE o_totalprice: NUMBER ``` The triple quote tells Snowflake to treat the identifier as case-sensitive UPPERCASE. Without it, the linter throws `Table not found` and lineage stops at the model boundary. ## Endpoints Every Data Product on Snowflake gets REST, GraphQL, MySQL-wire, and Postgres-wire endpoints. All four push queries down to the same Snowflake gateway, so latency is identical across protocols. The endpoint layer adds about 1.5 to 2 seconds of overhead on standard queries and 4 to 5 seconds on complex queries. Result sets up to about 50,000 rows are buffered in API memory before streaming, so keep `api.limit.memory` at 3 GiB and never below 1.5 GiB. ## Metadata scanning and catalog DataOS scans two Snowflake surfaces: `INFORMATION_SCHEMA` (real-time structural metadata) and `SNOWFLAKE.ACCOUNT_USAGE` (lineage, tags, query history, with an approximately 3-hour lag that Snowflake imposes and that is not configurable). Use a read-only scanner role separate from the Vulcan service role: ```sql GRANT USAGE ON DATABASE TO ROLE ; GRANT USAGE ON ALL SCHEMAS IN DATABASE TO ROLE ; GRANT REFERENCES ON ALL TABLES IN DATABASE TO ROLE ; GRANT REFERENCES ON FUTURE TABLES IN DATABASE TO ROLE ; GRANT IMPORTED PRIVILEGES ON DATABASE SNOWFLAKE TO ROLE ; GRANT ROLE TO USER ; ``` {% hint style="warning" %} `IMPORTED PRIVILEGES ON DATABASE SNOWFLAKE` requires `ACCOUNTADMIN`. Without it, lineage edges, tags, procedures, functions, and Dynamic Table history are not populated. {% endhint %} Lineage is computed, not fetched: DataOS parses view definitions, write queries (`MERGE`, `INSERT INTO SELECT`, `CTAS`), stored-procedure child queries, and external-table locations from `ACCOUNT_USAGE.QUERY_HISTORY`. The scanner runs every 6 to 12 hours (configurable). Lineage edges appear up to 3 hours after a query runs; if lineage is missing after a `vulcan apply`, wait or trigger a manual scan. ## Engine-native feature support You can drive most Snowflake objects from a Vulcan Data Product through guarded lifecycle SQL. Highlights: | Feature | Vulcan pattern | Edition | | ---------------------------------------------------------- | ----------------------------------------------------------------------------- | --------------------------------------------- | | Transient / temporary table | kind `FULL` + `physical_properties (creatable_type = TRANSIENT \| TEMPORARY)` | Standard+ | | Dynamic Table | kind `MANAGED` + `physical_properties (target_lag, warehouse)` | Enterprise+ | | Secure view | kind `VIEW` + `virtual_properties (createable_type = SECURE)` | Standard+ | | Clustering | `ALTER TABLE … CLUSTER BY` in guarded `post_statements` | Standard+ | | Masking / row access policy | `CREATE … POLICY` in `before_all`; apply in `post_statements` | Enterprise+ | | Streams, stages, sequences, tasks, UDFs, stored procedures | `CREATE …` in `before_all` / `pre_statements` | Standard+ | | Iceberg table | `CREATE ICEBERG TABLE` in `before_all` (external volume required) | Standard+ | | Time Travel / Fail-safe | `DATA_RETENTION_TIME_IN_DAYS`; Fail-safe automatic on permanent tables | Standard (1 day), Enterprise+ (up to 90 days) | You can also import a Snowflake-native semantic view with `vulcan import_semantic_view` (requires Enterprise and `SELECT` on the view). The generated YAML follows the UPPERCASE rule. Not supported through Vulcan: warehouse create/resize/suspend (provision out of band), OAuth/SSO for the gateway, cross-database external auto-discovery, `MANAGED` on Standard edition, incremental processing on `VIEW` kind, and Snowpark external network access (the integration must be created and granted by a Snowflake admin). ## Operational boundaries These are concrete settings with hard thresholds. Tune from telemetry. ### Compute sizing | Workload | Size | Rationale | | -------------------------------- | ------------------------------------ | ----------------------------------------------------------------------------------- | | Dev / single-user plan iteration | X-SMALL or SMALL | Plan only builds a few models. | | First full backfill | LARGE or X-LARGE | Backfills parallelise across intervals; bigger finishes faster and cheaper overall. | | Daily incremental runs | MEDIUM | Touches only new partitions. | | Semantic API / concurrent BI | LARGE, raise `MAX_CONCURRENCY_LEVEL` | Each API query hits Snowflake directly. | | Snowpark / Python models | One size up versus equivalent SQL | The warehouse coordinates the Snowpark pool. | Always set `AUTO_SUSPEND = 60` and `AUTO_RESUME = TRUE` because Vulcan workloads are bursty. ### Concurrency `MAX_CONCURRENCY_LEVEL` controls how many queries run before queueing. Use the default (8) for model runs only, 12 to 16 with a live semantic API, and 20 or more for many BI sessions. Keep `concurrent_tasks` in `config.yaml` at or below `MAX_CONCURRENCY_LEVEL / 3` so model builds do not starve API queries. ### API replicas The API is stateless; add replicas before raising Snowflake concurrency. Use 1 to 2 for a single team, 3 to 5 for multi-team dashboards, 5 or more for high volume. Keep `api.limit.memory` at 3 GiB. ### Scheduling Schedule after upstream data lands, set `timezone: UTC`, set `endOn` 1 to 2 years out (an expired `endOn` halts the workflow silently), and use `concurrencyPolicy: Forbid`. ### Latency floor You cannot tune below these, regardless of warehouse size: cold start after `AUTO_SUSPEND` adds 1 to 3 seconds on the first query; the semantic API adds about 1.5 to 2 seconds (standard) or 4 to 5 seconds (cross-join/complex); query parse and plan is 50 to 200 milliseconds; a result-cache hit is under 1 second end to end. ### Cost guardrails Cap monthly spend with a resource monitor, set query and queued timeouts, tag every query for cost attribution, and keep `AUTO_SUSPEND` short. ```sql CREATE RESOURCE MONITOR vulcan_monthly_cap WITH CREDIT_QUOTA = 500 FREQUENCY = MONTHLY START_TIMESTAMP = IMMEDIATELY TRIGGERS ON 75 PERCENT DO NOTIFY ON 90 PERCENT DO SUSPEND ON 100 PERCENT DO SUSPEND_IMMEDIATE; ALTER WAREHOUSE vulcan_wh SET STATEMENT_TIMEOUT_IN_SECONDS = 3600 STATEMENT_QUEUED_TIMEOUT_IN_SECONDS = 300; ALTER SESSION SET QUERY_TAG = 'vulcan::'; ``` The queued timeout matters most on a high-concurrency API workload: it prevents endpoint requests from piling up behind a stuck job. Set Bronze/staging tables to `TRANSIENT` when you can re-derive them from sources. ## Performance reference Reference numbers from TPC-H SF1000, Medium warehouse, 100 queries simulating 10 users at 5 parallel requests. Treat these as planning baselines, not guarantees. | Metric | Value | | ----------------------------------------------- | ------------------ | | Standard query completion | 12 to 15 seconds | | Semantic API overhead, standard queries | about 1.84 seconds | | Cross-join heavy queries | up to 5.13 minutes | | Concurrent success rate (10 users × 5 parallel) | 82% | Cross-joins are the only outlier (about 30 times slower); the fix is always the SQL shape, never warehouse sizing. Doubling warehouse size halves engine time but does not change the API floor. When concurrency degrades, fix in order: cut cross-joins, add API replicas, raise `MAX_CONCURRENCY_LEVEL`, then size up the warehouse. ## Failure modes and troubleshooting The errors you are most likely to hit, with cause and fix. | Symptom | Likely cause | Fix | | -------------------------------------------------------------- | ---------------------------------------------------- | -------------------------------------------------------------------------- | | JWT token invalid on first run | Public key not registered or PEM/passphrase mismatch | `ALTER USER … SET RSA_PUBLIC_KEY`; verify passphrase. | | Account identifier not found | Wrong format | Use `-` from the console URL, not the legacy locator. | | `Object 'COL' does not exist` in semantic queries | Lowercase column in semantic model | Switch all identifiers to UPPERCASE. | | `Table not found: SNOWFLAKE_SAMPLE_DATA…` during plan | Cross-database source not declared | Add to `external_models.yaml` with a triple-quoted identifier. | | Insufficient privileges on schema during auto-apply | Role missing `CREATE SCHEMA` / `CREATE TABLE` | Grant per the service-role template. | | `IMPORTED PRIVILEGES` required reading sample data | Share not mounted to your role | `GRANT IMPORTED PRIVILEGES ON DATABASE SNOWFLAKE_SAMPLE_DATA TO ROLE `. | | Apply succeeds but every run fails with `Depot not resolvable` | Depot name mismatch | Match `dataos://?purpose=rw` to the depot's `name`. | | `vulcan plan` runs DDL on Snowflake | Lifecycle DDL not guarded | Wrap in `@IF(@runtime_stage = 'evaluating', …)`. | | `Coercion failed` evaluating a macro | Macro parameter annotated `: str` | Remove the annotation; render with `.sql(dialect=evaluator.dialect)`. | | `NameError: 'col' is not defined` in Snowpark | Functions imported at module level | Import inside `execute()`. | | Snowpark model runs 5 to 10 times slower than SQL | Warehouse undersized for the coordinator | Size up one step. | | Endpoint OOM on a 50,000-row result | `api.memory.limit` below 1.5 GiB | Raise to 3 GiB. | | First query after idle 1 to 3 seconds slower | Warehouse cold start | Expected; document in the SLO or pin a warming query. | | 18% concurrent failure rate on dashboards | Warehouse saturating `MAX_CONCURRENCY_LEVEL` | Raise concurrency, add API replicas, then size up. | | Audit/DQ checks suspiciously fast on re-run | Result-cache hit | Expected; tag DQ queries separately when benchmarking. | ### Health-check queries ```sql -- Recent Vulcan queries SELECT query_id, query_text, execution_status, total_elapsed_time/1000 AS seconds, warehouse_name FROM snowflake.account_usage.query_history WHERE query_tag LIKE 'vulcan:%' AND start_time > DATEADD(hour, -2, CURRENT_TIMESTAMP()) ORDER BY start_time DESC LIMIT 50; -- Queueing check SELECT warehouse_name, COUNT(*) AS queries, AVG(queued_provisioning_time + queued_overload_time)/1000 AS avg_queued_s FROM snowflake.account_usage.query_history WHERE start_time > DATEADD(hour, -1, CURRENT_TIMESTAMP()) GROUP BY warehouse_name ORDER BY avg_queued_s DESC; ``` ### Recovery procedures | Situation | Procedure | | ---------------------------------------- | ----------------------------------------------------------------------------------------------------------- | | Incremental run failed mid-way | Re-run the workflow; `INCREMENTAL_BY_TIME_RANGE` re-deletes and re-inserts the window, so re-runs are safe. | | State drifted (table dropped manually) | `vulcan plan` detects and recreates; or `vulcan run --rebuild `. | | Roll back a plan version | `vulcan rollback `; this re-runs prior materialisation and consumes credits. | | Workflow halted because `endOn` expired | Update the resource with a new `endOn` and re-apply. | | Resource monitor suspended the warehouse | Investigate spend; raise `CREDIT_QUOTA` or `ALTER WAREHOUSE … RESUME`. | ## Related * [Connect engine → Snowflake](https://v2.dataos.info/build/productize/connect-engine/snowflake) for the quick connection steps. * [Engine guide overview](/references/engine-guide/engine-guide.md) for the cross-engine rules. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://v2.dataos.info/references/engine-guide/snowflake.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.