> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/dataos-resources/vulcan/core-concepts/data-quality.md). # Data quality A data quality (DQ) check is a declarative rule pack that monitors a model's quality over time without blocking it. You define it in a YAML file under `dq/` with `kind: dq`. When a rule fails, the check warns you. It does not stop execution. DQ checks are one of three data quality artifacts in Vulcan, and they are easy to confuse. Keep them straight: **DQ checks watch, audits define, assertions attach.** This page covers DQ checks. Audits and assertions, which block bad rows at write time, are covered in [Assertions](/references/dataos-resources/vulcan/core-concepts/assertions.md). | Artifact | Where it lives | Blocks the model? | Job | | --------- | --------------------------------------------------- | :-----------------------: | -------------------------------------------------- | | DQ check | `dq/*.yml` with `kind: dq` | No | Monitor quality and detect anomalies over time. | | Audit | `audits/*.sql`, defined with `AUDIT (name ...)` | Yes, through an assertion | The reusable validation rule. | | Assertion | Inside a `MODEL (...)` block via `assertions (...)` | Yes | Attaches an audit (or a built-in rule) to a model. | ## What a DQ check is for DQ checks are configured in simple YAML files in `dq/` using `kind: dq`. Each file is one rule pack and targets exactly one model via `depends_on:`. They do not block your models (your models keep running even if a rule fails), they track historical patterns and trends, they support statistical analysis, and they integrate with the Activity API for monitoring and alerting. Use a DQ check for: * Monitoring quality trends over time (is completeness getting worse?). * Statistical anomaly detection (did revenue suddenly spike?). * Cross-model validation (do orders match customers?). * Non-critical validation: warnings, not blockers. * Building data quality dashboards. Use an [assertion](/references/dataos-resources/vulcan/core-concepts/assertions.md) instead for critical business rules that must pass and block bad data from flowing downstream. Use [profiles](#data-profiling) when you want to observe and track data characteristics rather than validate them. ## The three-layer strategy A layered approach keeps each artifact in its lane: * **Audits (blocking).** Stop bad rows at the door. Primary keys must be unique, revenue must be non-negative, foreign keys must be valid. * **DQ checks (non-blocking).** Watch for problems but never interfere. Row count within an expected range, anomaly detection on metrics, cross-table consistency. * **Profiles (observation).** Record what normal looks like. Track null percentages, monitor distributions, detect drift. For a revenue table, you might layer all three. The audit (attached as an assertion) blocks if revenue is invalid: ```sql MODEL ( name analytics.revenue, assertions ( not_null(columns := (customer_id, revenue)), accepted_range(column := revenue, min_v := 0, max_v := 100000000) ) ); ``` The DQ check monitors and profiles without blocking: ```yaml # dq/revenue.yml # Profiles watch and record. Rules warn if something looks unusual. Neither blocks the model. kind: dq name: revenue_dq depends_on: analytics.revenue profiles: - revenue - order_count - customer_tier rules: - anomaly detection for avg(revenue): name: revenue_anomaly_detection dimension: accuracy - change for row_count >= -30%: name: row_count_drop_alert dimension: timeliness ``` ## Quick start Create your first rule pack in `dq/customers.yml`: ```yaml kind: dq name: customers_dq depends_on: analytics.customers rules: - missing_count(email) = 0: name: no_missing_emails dimension: completeness description: "All customers must have an email address" ``` This rule ensures every customer has an email address. Rule packs and profiles run automatically when models execute, through a `plan` or `run`. The execution output looks like this: ```bash Check Executions (1 Models) └── hello.subscriptions ├── completeness (4/4) ├── uniqueness (1/1) └── validity (3/3) Profiled 1 model (3 columns): warehouse.hello.subscriptions: 3 columns ``` ## File structure DQ rule packs live in YAML files in `dq/`. Each file is one rule pack and targets exactly one model via `depends_on:`. Organize them however makes sense: ``` project/ ├── models/ │ ├── dq/ │ │ ├── users.yml # Rules for analytics.users │ │ ├── orders.yml # Rules for analytics.orders │ │ └── cross_model.yml # Rules anchored on one model, joining others via SQL │ └── *.sql └── config.yaml ``` Files must end in `.yml` or `.yaml`. The file name does not matter; Vulcan reads all files in `dq/`. By convention, name files after the model they target. ## Basic syntax A rule pack has a pack-level header followed by a list of `rules:`: ```yaml kind: dq # Required: file kind name: # Required: unique name for this pack depends_on: # Required: the model this pack validates # Optional pack-level fields: filter: "" # Applied to every rule in this pack profiles: # Columns to profile alongside the rules - - rules: - : # e.g. missing_count(col) = 0, failed rows, etc. name: dimension: # completeness | validity | accuracy | ... description: ``` `kind: dq` declares the file as a DQ rule pack, `name` identifies the pack, and `depends_on` ties every rule in the file to a single model. ### Rule forms A rule can be a bare expression (no metadata) or an expression with a metadata block. Shorthand: just the expression. Vulcan auto-names the rule: ```yaml rules: - missing_count(user_id) = 0 - duplicate_count(event_id) = 0 ``` Full form: the expression is a YAML key and the metadata is its value: ```yaml rules: - missing_count(event_type) = 0: name: no_missing_event_types dimension: completeness description: "All events must have an event type" ``` You can mix both in the same `rules:` list. ### Rule attributes All rule metadata is flat: keys sit directly under the rule expression, not nested under an `attributes:` block. | Field | Description | | --------------- | --------------------------------------------------------------------------- | | `name` | Stable identifier for the rule. Required to look it up in the Activity API. | | `dimension` | Data quality dimension (see below). | | `description` | Human-readable explanation. | | `filter` | SQL predicate applied to this rule only. Overlays the pack-level `filter:`. | | `warn` | Threshold expression for a warning (for example, `when < 10`). | | `fail` | Threshold expression for failing the rule (for example, `when > 100`). | | `warn_only` | `true` to downgrade any failure to a warning. | | `fail query` | SQL `SELECT` returning bad rows. Only used with `failed rows`. | | `samples limit` | Number of failed sample rows to capture (default `5`). | | `tags` | List of tags for filtering and organisation. | | `owner` | Team or person responsible. | | `severity` | `error` (default) or `warning`. | ```yaml kind: dq name: usage_events_dq depends_on: b2b_saas.usage_events rules: - row_count > 10: name: sufficient_events dimension: completeness warn: when < 10 fail: when > 100 - row_count > 0: name: events_in_run_window dimension: completeness filter: "event_date >= CAST('${execution_dt}' AS DATE) - INTERVAL '7 days'" warn_only: true ``` ## Data quality dimensions Rules are classified by the `dimension:` field. Vulcan supports 8 standard dimensions (based on ODPS v3.1): | Dimension | What it checks | Example rule | | -------------- | ---------------------------------- | ----------------------------------------------- | | `completeness` | No missing required data. | `missing_count(customer_id) = 0` | | `validity` | Data conforms to format or syntax. | `failed rows` with a `fail query` | | `accuracy` | Data matches reality. | `anomaly detection for avg(revenue)` | | `consistency` | Data agrees across sources. | A `failed rows` join across tables | | `uniqueness` | No duplicates. | `duplicate_count(email) = 0` | | `timeliness` | Data is current. | `change for row_count >= -30%` | | `conformity` | Follows standards. | `failed rows` checking `LENGTH(zip_code) != 5` | | `coverage` | All expected records are present. | `row_count >= 95% of historical_avg(row_count)` | ## Built-in rule types Vulcan provides several built-in rule types that cover most scenarios. ### Missing data ```yaml rules: - missing_count(email) = 0 # count of NULLs - missing_percent(email) < 5 # percentage of NULLs ``` ### Row count ```yaml rules: - row_count > 1000 - row_count between 1000 and 100000 - row_count > 500: filter: "status = 'active'" # count on filtered data ``` ### Duplicate count ```yaml rules: - duplicate_count(email) = 0 - duplicate_count(customer_id, order_date) = 0 # composite key ``` ### Failed rows The most flexible rule type. Write any SQL `SELECT` that returns the rows you consider invalid. An empty result passes; one or more rows fails, and Vulcan captures samples. ```yaml rules: - failed rows: name: invalid_revenue dimension: validity fail query: | SELECT customer_id, revenue, order_date FROM analytics.orders WHERE revenue < 0 OR revenue > 10000000 samples limit: 20 ``` When you need more control than anomaly detection provides, `failed rows` can implement a custom statistical outlier check: compute a z-score in a `stats` CTE and flag rows more than 3 standard deviations from the mean. ```yaml rules: - failed rows: name: revenue_outliers dimension: accuracy fail query: | WITH stats AS ( SELECT AVG(revenue) AS mean, STDDEV(revenue) AS stddev FROM analytics.revenue ) SELECT r.*, (r.revenue - s.mean) / s.stddev AS z_score FROM analytics.revenue r, stats s WHERE ABS((r.revenue - s.mean) / s.stddev) > 3 samples limit: 20 ``` ### Threshold and statistical rules Check aggregations against thresholds. You can use `avg`, `sum`, `min`, `max`, `count`, `distinct_count`, `stddev`, `percentile`, and similar: ```yaml rules: - avg(revenue) between 100 and 10000: dimension: accuracy - max(age) <= 120: dimension: validity - percentile(revenue, 95) < 50000: dimension: accuracy ``` {% hint style="info" %} A common starting approach: set thresholds at ±3× stddev from the historical mean. Example: `avg(revenue) between 2000 and 8000` when typical avg is 5000 and stddev is 1000. {% endhint %} ### Anomaly detection Anomaly detection rules learn what normal looks like from previous runs of the same rule, then flag the next value when it falls outside that range. There is no model to train and no threshold to set: ```yaml rules: - anomaly detection for row_count: name: row_count_anomaly dimension: accuracy - anomaly detection for avg(revenue): dimension: accuracy ``` Under the hood it builds a statistical model (mean, standard deviation, trends) from the collected history, compares each new value to the expected range, and flags deviations that are typically greater than 3 standard deviations. It works best with regular schedules (daily or hourly) and becomes more accurate after roughly 30 data points. Anomaly detection needs history before it is useful. Expect roughly the first 30 runs to be no-ops while the baseline fills in, then it catches drifts you would not have thought to write a threshold rule for. ### Change over time Compare the current value to the previous run, where `change = (current - previous) / previous * 100`: ```yaml rules: - change for row_count >= -50%: name: row_count_drop_alert dimension: timeliness - change for avg(revenue) >= -20%: dimension: timeliness ``` A one-sided bound alerts in one direction: `change >= -30%` alerts on a drop of more than 30%, `change >= 10%` alerts on growth of more than 10%. Use a bidirectional range to alert on movement either way: ```yaml rules: - change for row_count between -10% and 10%: dimension: timeliness ``` ## Filtering rules Apply a SQL predicate at the pack level (every rule in the file) or per rule (overlays the pack-level filter): ```yaml kind: dq name: completed_recent_orders_dq depends_on: analytics.orders filter: "status = 'completed' AND order_date >= CURRENT_DATE - INTERVAL '30 days'" rules: - missing_count(customer_id) = 0: dimension: completeness - row_count > 500: filter: "status = 'active'" # per-rule filter dimension: completeness ``` For different expectations on different slices of the same model (US vs EU customers), create a separate file per slice. Each pack has a single `depends_on:` and its own `filter:`. ## Cross-model validation Each rule pack targets a single model via `depends_on:`, but `failed rows` queries can reference any model in your warehouse. Anchor the pack on the model whose quality is at stake, then join to related models in SQL: ```yaml # dq/orders_cross_model.yml kind: dq name: orders_cross_model_dq depends_on: analytics.orders rules: - failed rows: name: orphaned_orders dimension: consistency fail query: | SELECT o.order_id, o.customer_id FROM analytics.orders o LEFT JOIN analytics.customers c ON o.customer_id = c.customer_id WHERE c.customer_id IS NULL samples limit: 10 ``` ## Data profiling Profiles automatically collect statistical metrics about your data over time. Unlike rules, which validate, profiles observe and track. Enable profiling with a top-level `profiles:` list in a `kind: dq` file. A pack can be profile-only, rules-only, or both. ```yaml # dq/customers.yml kind: dq name: customers_dq depends_on: analytics.customers profiles: - revenue - signup_date - customer_tier rules: - missing_count(email) = 0: dimension: completeness ``` Vulcan collects table-level metrics (row count), column-level metrics (null count and percentage, distinct count, duplicate count, uniqueness percentage), numeric metrics (min, max, avg, sum, standard deviation, variance, histogram buckets), and text metrics (length stats, most frequent values). ### Querying profiles Profiles are stored in the `_check_profiles` table, which has one row per metric: | Column | Meaning | | -------------- | ---------------------------------------------------------------------------------------------------------- | | `id` | Unique identifier for this metric row. | | `run_id` | Which profiling run this metric belongs to. | | `table_name` | Table being profiled. | | `column_name` | Column being profiled (NULL for table-level metrics like `row_count`). | | `profile_type` | Metric type, e.g. `row_count`, `distinct`, `missing_count`, `frequent_values`, `min`, `max`, `avg_length`. | | `value_number` | Numeric metric value (`row_count`, `distinct`, `min`, `max`, `avg`, etc.). | | `value_text` | Text metric value (rare). | | `value_json` | JSON-encoded metric; holds histograms and frequent values. | | `value_type` | Type of the stored value (`number`, `json`, etc.). | | `profiled_at` | When the profiling ran (epoch ms). | | `created_ts` | When the row was inserted. | Query it like any other table: ```sql SELECT to_timestamp(profiled_at/1000)::date AS date, value_number AS missing_count FROM _check_profiles WHERE table_name = 'warehouse.hello.subscriptions' AND column_name = 'mrr' AND profile_type = 'missing_count' ORDER BY profiled_at DESC LIMIT 30; ``` ### Detecting data drift Use `_check_profiles` to build a data-drift query that compares recent values against a historical baseline. ```sql WITH latest_profile AS ( SELECT column_name, mean, stddev FROM vulcan._check_profiles WHERE model_name = 'sales.orders' AND profile_ts = ( SELECT MAX(profile_ts) FROM vulcan._check_profiles WHERE model_name = 'sales.orders' ) ), current AS ( SELECT COUNT(DISTINCT customer_id) AS current_customers, SUM(mrr) AS current_mrr FROM sales.orders WHERE order_date >= CURRENT_DATE - INTERVAL '7 days' ), historical AS ( SELECT AVG(COUNT(DISTINCT customer_id)) OVER () AS avg_customers, AVG(SUM(mrr)) OVER () AS avg_mrr FROM vulcan._check_profiles WHERE model_name = 'sales.orders' AND profile_ts >= CURRENT_DATE - INTERVAL '30 days' ) SELECT (current_customers - avg_customers) / NULLIF(avg_customers, 0) * 100 AS distinct_change_pct, (current_mrr - avg_mrr) / NULLIF(avg_mrr, 0) * 100 AS mrr_change_pct FROM current, historical; ``` A common workflow: enable profiling on a new model, observe patterns for 30 or more days, query `_check_profiles` to find the typical range, then add rules informed by what you saw instead of guesses. ### Profiles-to-rules workflow 1. **Enable profiling on a pack** — add `profile: true` (or a `profiles:` list) under the pack for the model you want to observe: ```yaml profiles: - revenue - order_count ``` 2. **Query `_check_profiles` to learn typical ranges** — after enough history has accumulated: ```sql SELECT column_name, MIN(min_value) AS min_revenue, MAX(max_value) AS max_revenue, AVG(mean) AS typical_revenue, AVG(stddev) AS revenue_stddev FROM vulcan._check_profiles WHERE model_name = 'sales.orders' AND column_name = 'revenue' GROUP BY column_name; ``` 3. **Use the stats to write informed rules** — replace guesses with observed ranges: ```yaml rules: - avg(revenue) between 1000 and 5000: - stddev(revenue) < 3000: ``` {% hint style="info" %} Start conservative and tighten thresholds as you learn your data's normal range. {% endhint %} {% hint style="warning" %} Do not profile sensitive or personally identifiable information (PII) columns, every column, or very high-frequency models. Profiles add storage and overhead. Profile high-value production tables and the columns used downstream. {% endhint %} ### Profiling best practices **Do:** * Profile tables where you expect gradual drift (revenue, user counts, session durations). * Run profiles on a daily or weekly schedule. * Use profile stats to set `between` ranges rather than exact-match rules. **Avoid:** * Profiling columns with PII — profile values are stored in the state schema. * Profiling every column — focus on business-critical metrics. * Profiling high-frequency models on a tight schedule (adds query load). ## Running and troubleshooting DQ checks run automatically when models execute through `plan` or `run`. Run them on their own with `vulcan check`: ```bash vulcan check --select analytics.customers.invalid_emails --verbose ``` When a `failed rows` rule fails, query the captured samples: ```sql SELECT * FROM check_samples WHERE check_name = 'invalid_emails' AND status = 'failed' ORDER BY executed_at DESC LIMIT 10; ``` If a rule produces false positives during normal variance, widen the threshold (use `between` instead of an exact match) or switch to anomaly detection, which adapts to variance. If a rule is slow, add a pack-level `filter:` to scope it to recent data. For engines that support indexes, adding an index on the time column used in the DQ filter can speed up rule execution. Example: ```sql CREATE INDEX ON sales.orders (order_date); ``` ## Summary * DQ checks are YAML rule packs in `dq/` with `kind: dq`, one pack per model via `depends_on:`. They are non-blocking and track quality over time. * Rules can be shorthand or full form, classified by `dimension`, and filtered at the pack or rule level. * Profiles observe data characteristics over time and inform which rules to write. * DQ checks watch, audits define, assertions attach. For the blocking artifacts, see [Assertions](/references/dataos-resources/vulcan/core-concepts/assertions.md). ## Related docs * [Assertions](/references/dataos-resources/vulcan/core-concepts/assertions.md): audits and assertions that block bad rows at write time. * [Test](/references/dataos-resources/vulcan/core-concepts/unit-test.md): validate model logic against fixtures before deployment. * [Observability](/references/dataos-resources/vulcan/observability.md): how checks run and where their activity is recorded. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://v2.dataos.info/references/dataos-resources/vulcan/core-concepts/data-quality.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.