> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/build/productize/data-quality.md).

# Data quality

Add DQ checks that monitor quality over time and profile columns.

A data product makes guarantees about its data. You enforce those guarantees three ways, and the difference between them is worth holding onto: unit tests validate logic; audits define a rule that assertions attach to a model to block bad data; and DQ (data quality) checks watch data over time and warn without blocking.

This page covers DQ checks. [Assertions](/build/productize/assertions.md) covers the blocking layer; [Unit Test](/build/productize/unit-test.md) covers logic verification.

A DQ check is a promise about a model's data that Vulcan scores over time. Unlike an assertion, a failing DQ check doesn't stop the run: the model still materializes. Instead, the check records the failure and builds a historical record of the data's health. Use DQ checks for trend monitoring and any validation where a failure should warn, not block.

| Use a DQ check for                                       | Use an [assertion](/build/productize/assertions.md) instead for |
| -------------------------------------------------------- | --------------------------------------------------------------- |
| Monitoring quality trends over time                      | A critical rule that must block bad data                        |
| Row count or value-range checks that don't need to block | Null or uniqueness checks on key columns                        |
| Cross-table consistency monitoring                       | Any rule where failure should stop the run                      |

DQ checks live in `dq/*.yml`, one file per model, with `kind: dq`. `orders-analytics` has 7 packs covering row counts, null rates, value validity, and referential integrity across the bronze layer.

## The shape

```yaml
kind: dq
name: fct_daily_sales_dq
depends_on: silver.fct_daily_sales

profiles:
  - order_date
  - total_revenue
  - shipment_rate

expectations:
  - expectation: expect_table_row_count_to_be_between
    min_value: 20
    meta:
      name: minimum_daily_sales_rows
      dimension: completeness
      description: Daily sales should contain at least 20 rows
  - expectation: expect_column_values_to_not_be_null
    column: order_date
    meta:
      name: no_missing_order_date
      dimension: completeness
  - expectation: expect_column_values_to_be_between
    column: total_revenue
    min_value: 0
    meta:
      name: total_revenue_non_negative
      dimension: validity
  - expectation: expect_column_values_to_be_between
    column: shipment_rate
    min_value: 0
    max_value: 1
    meta:
      name: shipment_rate_between_zero_and_one
      dimension: validity
```

Every pack needs `kind: dq`, a unique `name`, and `depends_on` (the model the pack validates). `expectations` and `profiles` are both optional (a pack can be profile-only, checks-only, or both), and so is a pack-level `filter` (a SQL condition that limits the rows every check in the file judges).

{% hint style="warning" %}
**Still using `rules:`?**

Packs that list checks under `rules:` still load. Vulcan converts each rule into an expectation when the project loads, but the `rules:` syntax will be deprecated in December 2026. Write new checks as expectations, and move older packs to `expectations:` before then. A pack contains one of the two lists, not both.
{% endhint %}

{% hint style="info" %}
On Snowflake, write column names in uppercase in your DQ files, such as `REVENUE`, `EMAIL`, and `CUSTOMER_ID`.
{% endhint %}

{% hint style="warning" %}
DQ YAML is loaded without key conversion, so keys must already be `snake_case`: a camelCase key like `dependsOn` is silently unrecognized rather than converted.
{% endhint %}

{% hint style="info" %}
Column profiling used to be configurable via a `profiles` property inside a `MODEL(...)` block. That's deprecated. Profiling now lives exclusively under `profiles:` in a `kind: dq` file.
{% endhint %}

## When checks run

DQ checks run after the data is written, and only when that write succeeded. When a scheduled run or a plan apply finishes without errors, Vulcan scores the packs for the models that were just built. A model that failed to build is left unscored, and if the run or apply hits any error, scoring for that job is skipped. A failed check is a recorded result: it doesn't roll the table back.

## Two things a pack does

**Profiles** collect statistics for the listed columns on every run: null count, distinct count, min, max, distribution. They observe rather than validate. Because Vulcan stores profiles over time, you can see what's normal for your data and set informed thresholds. Query them first, then write expectations.

**Expectations** validate. Each one names a check and the limit it enforces, with optional metadata under `meta`:

```yaml
expectations:
  - expectation: expect_column_values_to_be_unique
    column: order_id
  - expectation: expect_column_values_to_not_be_null
    column: email
    meta:
      name: no_missing_emails
      dimension: completeness
      description: Every customer has an email
```

`meta` takes `name`, `dimension`, and `description`. If you omit `name` or `dimension`, Vulcan fills them in. Set `name` on any check you want to follow over time.

## Check types

* **Missing data**: `expect_column_values_to_not_be_null`.
* **Row count**: `expect_table_row_count_to_be_between` with `min_value`, `max_value`, or both.
* **Uniqueness**: `expect_column_values_to_be_unique`, `expect_compound_columns_to_be_unique`.
* **Validity**: `expect_column_values_to_be_in_set`, `expect_column_values_to_match_regex`, `expect_column_values_to_be_between`.
* **Numeric**: `expect_column_mean_to_be_between`, `expect_column_min_to_be_between`, `expect_column_max_to_be_between`, `expect_column_sum_to_be_between`.
* **Freshness**: `vulcan_freshness`.
* **Custom SQL**: an `unexpected_rows_expectation` with an `unexpected_rows_query` that returns the invalid rows. It fails if the query returns any rows.

```yaml
expectations:
  - expectation: unexpected_rows_expectation
    unexpected_rows_query: |
      SELECT user_id, email FROM analytics.users
      WHERE email NOT LIKE '%@%'
    meta:
      name: invalid_emails
      dimension: validity
```

See [References → Vulcan → Data quality](https://v2.dataos.info/references/resources/vulcan/quality/data-quality) for every check, its parameters, and the `rules:` line each one replaces.

## Dimensions

Classify each check with `meta.dimension` so quality reporting groups it correctly. Options: `completeness`, `validity`, `accuracy`, `consistency`, `uniqueness`, `timeliness`, `conformity`, `coverage`.

A DQ check tells you something drifted; it doesn't stop the drift from shipping. When a check must block bad data before it reaches a consumer, write an [assertion](/build/productize/assertions.md) instead.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/build/productize/data-quality.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.