> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/build/july2026/productize/data-quality.md).

# Data quality

Add DQ checks that monitor quality over time and profile columns.

A data product makes guarantees about its data. You enforce those guarantees three ways, and the difference between them is worth holding onto: unit tests validate logic, audits define a rule, assertions attach that rule to a model and block bad data, and DQ (data quality) checks watch data over time and warn without blocking.

This page covers DQ checks. [Assertions](https://v2.dataos.info/build/july2026/productize/assertions.md) covers the blocking layer; [Unit Test](https://v2.dataos.info/build/july2026/productize/unit-test.md) covers logic verification.

A DQ check is a rule that monitors a model over time. Unlike an assertion, a failing DQ check doesn't stop the run: the model still materializes. Instead, the check surfaces a warning and builds a historical record of the data's health. Use DQ checks for trend monitoring and any validation where a failure should warn, not block.

| Use a DQ check for                                       | Use an [assertion](https://v2.dataos.info/build/july2026/productize/assertions.md) instead for |
| -------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| Monitoring quality trends over time                      | A critical rule that must block bad data                                                       |
| Row count or value-range checks that don't need to block | Null or uniqueness checks on key columns                                                       |
| Cross-table consistency monitoring                       | Any rule where failure should stop the run                                                     |

DQ checks live in `dq/*.yml`, one file per model, with `kind: dq`. `orders-analytics` has 7 packs covering row counts, null rates, value validity, and referential integrity across the bronze layer.

## The shape

```yaml
kind: dq
name: fct_daily_sales_dq
depends_on: silver.fct_daily_sales

profiles:
  - order_date
  - total_revenue
  - shipment_rate

rules:
  - row_count >= 20:
      name: minimum_daily_sales_rows
      dimension: completeness
      description: Daily sales should contain at least 20 rows
  - missing_count(order_date) = 0:
      name: no_missing_order_date
      dimension: completeness
  - invalid_count(total_revenue) = 0:
      valid min: 0
      name: total_revenue_non_negative
      dimension: validity
  - invalid_count(shipment_rate) = 0:
      valid min: 0
      valid max: 1
      name: shipment_rate_between_zero_and_one
      dimension: validity
```

Every pack needs `kind: dq`, a unique `name`, `depends_on` (the model the pack validates), and `rules`. `profiles` is optional, and so is a pack-level `filter` (a SQL predicate applied to every rule in the file, overlaid by any per-rule `filter`).

{% hint style="warning" %}
DQ YAML is loaded without key conversion, so keys must already be `snake_case`: a camelCase key like `dependsOn` is silently unrecognized rather than converted.
{% endhint %}

{% hint style="info" %}
Column profiling used to be configurable via a `profiles` property inside a `MODEL(...)` block. That's deprecated. Profiling now lives exclusively under `profiles:` in a `kind: dq` file.
{% endhint %}

## Two things a pack does

**Profiles** collect statistics for the listed columns on every run: null count, distinct count, min, max, distribution. They observe rather than validate. Because Vulcan stores profiles over time, you can see what's normal for your data and set informed thresholds. Query them first, then write rules.

**Rules** validate. Write them in shorthand for quick checks, or full form with metadata:

```yaml
rules:
  - duplicate_count(order_id) = 0          # shorthand
  - missing_count(email) = 0:              # full form
      name: no_missing_emails
      dimension: completeness
      severity: error
```

Full-form rule metadata is flat: `name`, `dimension`, `description`, `filter`, `tags`, `owner`, `severity` (`error`, default, or `warning`), plus two threshold fields for tuning how strict a rule is: `warn` (emit a warning at this threshold, e.g. `when < 10`) and `fail` (fail at this threshold), and `warn_only: true` to downgrade any failure on that rule to a warning.

## Rule types

* **Missing data** - `missing_count(col) = 0`, `missing_percent(col) < 5`.
* **Row count** - `row_count > 1000`, `row_count between 5000 and 15000`.
* **Uniqueness** - `duplicate_count(col) = 0`, `duplicate_count(col1, col2) = 0`.
* **Numeric** - `avg(revenue) between 100 and 10000`, `min(price) >= 0`.
* **Custom SQL** - a `failed rows` rule with a `fail query` that returns the invalid rows. It fails if the query returns any rows.

```yaml
rules:
  - failed rows:
      name: invalid_emails
      dimension: validity
      fail query: |
        SELECT user_id, email FROM analytics.users
        WHERE email NOT LIKE '%@%'
      samples limit: 10
```

See References → Vulcan → Data quality for the full list of built-in rule types.

## Dimensions

Classify each rule with a `dimension` value so quality reporting groups it correctly. Options: `completeness`, `validity`, `accuracy`, `consistency`, `uniqueness`, `timeliness`, `conformity`, `coverage`.

A DQ check tells you something drifted; it doesn't stop the drift from shipping. When a rule must block bad data before it reaches a consumer, write an [assertion](https://v2.dataos.info/build/july2026/productize/assertions.md) instead.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/build/july2026/productize/data-quality.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
