Data Quality Testing Anti-Patterns

Every mechanism covered earlier in this module — contracts, expectations, statistical checks, tiered severity — can still be undermined by how it is actually deployed. These three patterns are common enough to check for by name before assuming a quality program that technically exists is actually working.

Alert fatigue: the over-sensitive check nobody trusts anymore#

A soft-failure alert that fires on every minor, harmless fluctuation trains its recipients to ignore it — and once ignored, it stays ignored even the one time it fires for something real:

Week 1: alert fires (false alarm) -> investigated, dismissedWeek 2: alert fires (false alarm) -> investigated, dismissedWeek 3: alert fires (false alarm) -> glanced at, dismissedWeek 4: alert fires (real problem) -> dismissed without investigation,                                       same as the last three

Data Quality as an SLA covers choosing severity deliberately; this is what happens when that choice defaults to "alert on everything" instead, manufacturing false positives until the alert channel itself stops functioning as a signal, well before anyone decides to formally ignore it.

Checking only at the end of the pipeline#

A single quality check on the final output confirms something is wrong without saying where it went wrong — the root cause could be at ingestion, during any transformation stage, or in a join against another table:

Source -> Stage 1 -> Stage 2 -> Stage 3 -> [only check here]                                              "something is wrong" —                                              but which stage?

Checks placed at each meaningful stage boundary — not just the end — narrow a failure to a specific stage immediately, doing part of the job a full data lineage system would otherwise have to do after the fact, and the same "shift the detection earlier" principle Data Contracts applies to catching a breaking schema change at deploy time instead of at a downstream consumer.

Never testing that a check actually catches what it claims to#

An expectation that has never been deliberately triggered by bad data is an unverified assumption, not a confirmed safeguard. Software testing has a name for verifying this deliberately — mutation testing: introduce a deliberate fault and confirm the test suite actually catches it, rather than trusting that it would. The same discipline Testing Ingestion Code Without Hitting the Network applies to deliberately exercising a retry loop's failure path instead of only ever seeing it pass:

# Confirms the expectation actually fires on bad data,# not just that it passes on good databad_batch = good_batch.copy()bad_batch.loc[0, "amount"] = -1result = validator.validate(bad_batch)assert not result["success"], "Expectation should have caught the negative amount"

Without this, an expectation with a typo in its column name, or a threshold accidentally set to always pass, looks identical to a correctly working one — both show green, right up until the day a real violation sails through uncaught.

Common mistakes#

Instrumenting every table and column before any evidence of a real problem#

Broad, undifferentiated quality coverage applied everywhere at once dilutes attention across low-stakes checks and high-stakes ones equally — the data-quality equivalent of premature program optimization. Starting with the tables and dimensions that would actually cause harm if wrong — and expanding coverage from evidence, not by default — keeps the checks that exist worth paying attention to.

Treating a quality program as "done" once checks exist#

Checks that exist, run, and are never revisited drift out of sync with what the data and its consumers actually need over time — the same staleness risk Data Contracts warns about for an unmaintained contract.

Assuming an alert that fires often is working correctly#

A high firing rate is at least as likely to mean the threshold is wrong as it is to mean real problems are common. Statistical Methods for Anomaly Detection covers one common cause — a threshold calibrated on too little history — worth checking before assuming the frequent alerts reflect frequent real problems.

Further reading#

• Mutation testing — the software-testing discipline of deliberately verifying a test suite actually catches the faults it claims to.
• Data Quality as an SLA — the severity-tiering decision that prevents alert fatigue when made deliberately.
• Testing Ingestion Code Without Hitting the Network — the same "test the failure path on purpose" discipline, applied to code instead of quality expectations.
• Data Contracts — the shift-left principle behind checking at each pipeline stage instead of only the end.

See also#