Data Quality with Pandas covers detecting bad rows with boolean masks — one implementation of the broader discipline of data validation. This module is about what sits above that technique: what "data quality" actually means, formally, and the methodology for verifying it systematically rather than one ad-hoc check at a time.
Quality is measured against an expectation, not observed directly#
Data quality is not a
property a dataset either has or lacks in isolation — it is a measurement
against an explicit expectation someone stated in advance. A status
column containing "shipped" is high quality if the expected values are
{"pending", "shipped", "delivered"}, and is a quality failure if the
expected set was actually {"pending", "completed", "cancelled"} and
"shipped" was never a valid value at all. The data did not change
between those two judgments — the expectation did. Every technique in this
module assumes an expectation has already been written down somewhere;
without one, "is this correct" has no answer to check against.
Six dimensions, not one#
Data quality decomposes into distinct dimensions, each checked differently and each capable of failing independently of the others:
- Accuracy — does the value reflect reality? A
priceof29.99that should be39.99is accurate-looking but wrong. - Completeness — is anything missing? A null
customer_id, a table with fewer rows than the source system reports. - Consistency — do related values agree? An order marked
"shipped"with noshipped_attimestamp is internally inconsistent. - Timeliness — is the data recent enough to be useful? A dashboard reading data that stopped updating six hours ago is stale, even if every value it does have is accurate.
- Uniqueness — does the same real-world entity appear more than once where it shouldn't? A duplicated order row double-counts revenue.
- Validity — does a value conform to its defined format or range? An
email column with a value containing no
@character, a negativequantity.
A dataset can score well on some of these and badly on others at the same time — accurate values arriving late, or complete rows that duplicate freely. Naming which dimension actually failed is what turns "the data looks wrong" into a check that can be automated and monitored.
This module assumes correct code, and asks about the data anyway
Testing Ingestion Code Without Hitting the Network covers verifying that code behaves correctly against known inputs. Code that is perfectly correct can still produce bad output from bad input — a retry loop working exactly as designed still faithfully ingests a source's malformed row. Data quality is the discipline of checking the data, a distinct question from whether the code that produced it has bugs — and a narrower one than data governance, which additionally covers who is allowed to access and change data in the first place.
What the rest of this module covers#
Data Contracts covers writing expectations down formally, before data ever flows, rather than discovering them after a failure. Expectation-Based Testing covers checking those expectations declaratively instead of scattering ad hoc assertions through pipeline code. The Testing Pyramid for Data places quality checks alongside code tests in one coherent strategy, and Statistical Methods for Anomaly Detection covers catching problems no fixed rule was written to anticipate.
Common mistakes#
Checking quality without a written expectation#
A check with no explicit stated expectation behind it cannot be reviewed, disagreed with, or maintained by anyone other than whoever wrote it from memory. Writing the expectation down — even briefly — is what makes a quality check a shared, durable artifact instead of one person's tribal knowledge.
Treating quality as a single pass/fail signal#
Collapsing accuracy, completeness, consistency, timeliness, uniqueness, and validity into one "is the data good" check hides which dimension actually failed, making root-cause diagnosis slower every time something breaks.
Assuming correct code implies correct data#
The two are related but independent claims. Testing Ingestion Code verifies the first; this module is about verifying the second, which requires checking the data itself, not just the logic that moved it.
Further reading#
- Data quality — the general concept this module builds a full methodology around.
- Data validation — the broader discipline row-level checks are one implementation of.
- Data lineage — tracing where a value actually originated, relevant once a quality dimension fails and the cause needs tracking down.
- dbt data tests — a widely used, SQL-based implementation of the same dimension-checking idea, alongside Great Expectations' Python-based one.
- Data Quality with Pandas — the row-level technique this module's methodology sits above.
- Testing Ingestion Code Without Hitting the Network — verifying code correctness, the complementary discipline to verifying data correctness.