Before you use this reference
This page is a lookup, not a tutorial. The explanations live in Data Quality Fundamentals, Data Contracts, Expectation-Based Testing, The Testing Pyramid for Data, Statistical Methods for Anomaly Detection, Data Quality as an SLA, and Data Quality Testing Anti-Patterns.
The six dimensions#
Accuracy does the value reflect reality?Completeness is anything missing?Consistency do related values agree with each other?Timeliness is the data recent enough to be useful?Uniqueness does the same entity appear more than once?Validity does a value conform to its defined format/range?Data contract checklist#
Schema: field names, types, nullabilitySemantics: what each field actually means (units, encoding)Guarantees: freshness, completeness, ordering SLAsChange policy: how a breaking change gets communicated before it landsOwner: a named team on the producer sideExpectation-based check#
validator.expect_column_values_to_not_be_null("customer_id")validator.expect_column_values_to_be_between("amount", min_value=0)validator.expect_column_values_to_be_in_set("status", ["pending", "shipped"])The four-layer testing pyramid#
Unit many, fast — pure transformation logicIntegration fewer, slower — real/realistic systems, one stageEnd-to-end few, slow — full pipeline, small realistic datasetData quality continuous — real production data, ongoing (not pre-deploy)Statistical anomaly check#
recent = metric_history.tail(30)mean, std = recent.mean(), recent.std()is_anomalous = not (mean - 3*std <= today_value <= mean + 3*std)Needs enough history (weeks, not days) before the bounds mean anything.
Severity#
| Tier | Response | Example |
|---|---|---|
| Hard failure | Stop the pipeline | Row count of zero, duplicate primary keys |
| Soft failure | Log and alert, continue | Elevated but still-low null rate, a statistical anomaly |
Checklist before trusting a check#
- Does it fire when fed deliberately bad data? (test the check itself)- Is it placed at the stage where a failure would actually help localize the problem, not only at the pipeline's final output?- Is its threshold based on enough real history to be meaningful?Further reading#
- Great Expectations documentation — a full reference implementation of the expectation-based pattern above.
- The Test Pyramid — Martin Fowler — the original three-layer pyramid this module's fourth layer extends.
- Statistical process control — the discipline behind the rolling-window anomaly check above.
- dbt data tests — a SQL-based alternative to the Python expectation syntax shown above.
- Data quality — the general concept the six dimensions above are drawn from.