Software testing has a well-known shape: Martin Fowler's Test Pyramid, building on Mike Cohn's original formulation — many fast unit tests at the base, fewer integration tests in the middle, a handful of slow end-to-end tests at the top. A data pipeline needs the same discipline, plus one layer the classic pyramid does not have at all.
The classic pyramid, applied to pipeline code#
/\ /E2E\ Few, slow: full pipeline against real-ish systems /------\ /Integr. \ More, moderate speed: one stage against a real DB /----------\ / Unit \ Many, fast: pure transformation logic, small fixtures /--------------\- Unit tests check pure transformation functions against small, known inputs — the same style Testing Ingestion Code Without Hitting the Network covers for an API client, applied to a data transformation instead.
- Integration tests check that a pipeline stage actually moves data correctly between real or realistic systems — a test database, a real (non-mocked) file format round trip.
- End-to-end tests run the full pipeline against a small, realistic dataset, checking the whole chain works together.
All three layers share one property: they run before the code ships, against data the test itself controls.
The layer the classic pyramid does not have#
None of the three layers above can catch a problem that only exists in production data — an upstream API that silently changed a field's meaning, a partner's file that started arriving with a new encoding, a distribution shift no fixture ever exercised. This needs a genuinely different kind of check: expectation-based validation running continuously against the real, live data, not a pre-deployment test against data the test itself constructed:
/\ /E2E\ /------\ /Integr. \ /----------\ / Unit \ /--------------\ / Data Quality \ Continuous, against real production data —/ (ongoing) \ not a pre-deployment test at all--------------------This layer is monitoring, not testing
The three classic layers ask "does the code work, given data I control?" The data quality layer asks a different question entirely: "is the real data, arriving right now, still what everything downstream assumes it is?" No amount of passing unit tests answers that question — only checking the actual data, continuously, does.
Why all four layers are necessary, not redundant#
Each layer catches a category of problem the others structurally cannot:
- A unit test catches a logic bug in a transformation function, using data the test wrote itself — it says nothing about whether real upstream data still matches what the function expects.
- A production data quality check catches an upstream change no fixture anticipated — but it fires after the bad data has already arrived, and it does not localize which function is wrong the way a failing unit test does.
Relying on only one layer leaves a real, specific gap: unit tests alone miss upstream drift entirely; production checks alone catch problems too late to prevent bad data from ever reaching a pipeline stage, and diagnose slowly once they do fire.
Common mistakes#
Treating production data quality checks as a substitute for unit tests#
A production check firing tells you something is wrong right now, in real data — it does not tell you which function has a bug, the way a focused, fast-running unit test does. Skipping unit tests because production checks "would catch it eventually" trades a fast, localized signal for a slow, generic one.
Treating unit tests as sufficient on their own#
Perfectly passing unit tests say nothing about whether today's real upstream data still looks like the fixtures those tests were written against. Data Quality Fundamentals makes the same point from the other direction: correct code does not imply correct data.
Collapsing every layer into one slow end-to-end check#
A single giant test covering unit-level logic, integration behavior, and data quality all at once is slow to run and unhelpful to debug — a failure could mean any of a dozen things. Keeping the layers distinct keeps each failure specific enough to act on quickly.
Further reading#
- The Test Pyramid — Martin Fowler — the original software testing pyramid this article extends with a data-specific fourth layer.
- Test fixture — the small, known inputs a unit test runs a transformation function against.
- Testing Ingestion Code Without Hitting the Network — the unit-testing layer applied to an API client specifically.
- Expectation-Based Testing — the mechanism behind this pyramid's continuous top layer.