> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/build/productize/data-quality.md).

# Data quality

A data product makes guarantees about its data. You enforce those guarantees three ways, and the difference between them is worth holding onto: unit tests validate logic, audits define a rule, assertions attach that rule to a model and block bad data, and DQ (data quality) checks watch data over time and warn without blocking.

This page covers DQ checks. [Assertions](/build/productize/assertions.md) covers the blocking layer; [Unit Test](/build/productize/unit-test.md) covers logic verification.

A DQ check is a rule that monitors a model over time. Unlike an assertion, a failing DQ check doesn't stop the run: the model still materializes. Instead, the check surfaces a warning and builds a historical record of the data's health. Use DQ checks for trend monitoring, anomaly detection, and any validation where a failure should warn, not block.

| Use a DQ check for                       | Use an [assertion](/build/productize/assertions.md) instead for |
| ---------------------------------------- | --------------------------------------------------------------- |
| Monitoring quality trends over time      | A critical rule that must block bad data                        |
| Anomaly detection (sudden drops, spikes) | Null or uniqueness checks on key columns                        |
| Cross-table consistency monitoring       | Any rule where failure should stop the run                      |

DQ checks live in `dq/*.yml`, one file per model, with `kind: dq`. `orders-analytics` has 7 packs covering row counts, null rates, value validity, and referential integrity across the bronze layer.

## The shape

```yaml
kind: dq
name: fct_daily_sales_dq
depends_on: silver.fct_daily_sales

profiles:
  - order_date
  - total_revenue
  - shipment_rate

rules:
  - row_count >= 20:
      name: minimum_daily_sales_rows
      dimension: completeness
      description: Daily sales should contain at least 20 rows
  - missing_count(order_date) = 0:
      name: no_missing_order_date
      dimension: completeness
  - invalid_count(total_revenue) = 0:
      valid min: 0
      name: total_revenue_non_negative
      dimension: validity
  - invalid_count(shipment_rate) = 0:
      valid min: 0
      valid max: 1
      name: shipment_rate_between_zero_and_one
      dimension: validity
```

Every pack needs `kind: dq`, a unique `name`, `depends_on` (the model the pack validates), and `rules`. `profiles` is optional, and so is a pack-level `filter` (a SQL predicate applied to every rule in the file, overlaid by any per-rule `filter`).

{% hint style="warning" %}
DQ YAML is loaded without key conversion, so keys must already be `snake_case`: a camelCase key like `dependsOn` is silently unrecognized rather than converted.
{% endhint %}

{% hint style="info" %}
Column profiling used to be configurable via a `profiles` property inside a `MODEL(...)` block. That's deprecated. Profiling now lives exclusively under `profiles:` in a `kind: dq` file.
{% endhint %}

## Two things a pack does

**Profiles** collect statistics for the listed columns on every run: null count, distinct count, min, max, distribution. They observe rather than validate. Because Vulcan stores profiles over time, you can see what's normal for your data and set informed thresholds. Query them first, then write rules.

**Rules** validate. Write them in shorthand for quick checks, or full form with metadata:

```yaml
rules:
  - duplicate_count(order_id) = 0          # shorthand
  - missing_count(email) = 0:              # full form
      name: no_missing_emails
      dimension: completeness
      severity: error
```

Full-form rule metadata is flat: `name`, `dimension`, `description`, `filter`, `tags`, `owner`, `severity` (`error`, default, or `warning`), plus two threshold fields for tuning how strict a rule is: `warn` (emit a warning at this threshold, e.g. `when < 10`) and `fail` (fail at this threshold), and `warn_only: true` to downgrade any failure on that rule to a warning.

## Rule types

* **Missing data** - `missing_count(col) = 0`, `missing_percent(col) < 5`.
* **Row count** - `row_count > 1000`, `row_count between 5000 and 15000`.
* **Uniqueness** - `duplicate_count(col) = 0`, `duplicate_count(col1, col2) = 0`.
* **Numeric** - `avg(revenue) between 100 and 10000`, `min(price) >= 0`.
* **Custom SQL** - a `failed rows` rule with a `fail query` that returns the invalid rows. It fails if the query returns any rows.

```yaml
rules:
  - failed rows:
      name: invalid_emails
      dimension: validity
      fail query: |
        SELECT user_id, email FROM analytics.users
        WHERE email NOT LIKE '%@%'
      samples limit: 10
```

Vulcan also supports anomaly detection (learning a baseline and flagging deviations) and change monitoring (drops or spikes between runs). `orders-analytics` doesn't use these; see References → Vulcan → Data quality for the full reference.

## Dimensions

Classify each rule with a `dimension` value so quality reporting groups it correctly. Options: `completeness`, `validity`, `accuracy`, `consistency`, `uniqueness`, `timeliness`, `conformity`, `coverage`.

A DQ check tells you something drifted; it doesn't stop the drift from shipping. When a rule must block bad data before it reaches a consumer, write an [assertion](/build/productize/assertions.md) instead.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://v2.dataos.info/build/productize/data-quality.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.