Statistical Methods for Anomaly Detection

Catching the problem no fixed rule was written to anticipate — a row count that quietly dropped, an average that drifted outside its normal range — by comparing today's data against its own recent history instead of a hardcoded threshold.

AdvancedJuly 1, 2026

Every expectation covered so far in this module checks against a fixed rule someone wrote down in advance — amount >= 0, status IN (...). This article covers a different class of check: comparing today's data against its own recent history, to catch a problem no one thought to write a fixed rule for.

A fixed rule cannot catch a shift in what "normal" looks like#

expect_row_count_to_be_between(min=900, max=1100) catches a count of 50 or 5,000. It does not catch a count of 700 on a day that should have had 1,000 — a real drop, inside the "allowed" range, invisible to a rule that was never tuned to today's actual typical volume. Detecting that requires comparing today's value against recent values, not a number fixed once and left unchanged.

Control charts: the discipline this borrows from#

Statistical process control, developed for manufacturing quality control, formalizes exactly this comparison with a control chart: track a metric's mean and standard deviation over a rolling recent window, and flag any new value that falls unusually far from that recent mean — conventionally more than three standard deviations, a threshold chosen to balance catching real problems against flagging normal variation:

Python

recent = daily_row_counts.tail(30)mean, std = recent.mean(), recent.std()
lower_bound = mean - 3 * stdupper_bound = mean + 3 * std
today_is_anomalous = not (lower_bound <= today_count <= upper_bound)

The bounds move with the data — a metric that has been trending upward for weeks gets a correspondingly higher expected range, rather than being compared against a number fixed on the day someone first wrote the check.

Applying this to more than row counts#

The same rolling-window comparison applies to any metric with a meaningful recent history, not just volume:

Text

Row count:        today's count vs. the last 30 days' countColumn mean:       today's average `amount` vs. its 30-day rolling averageNull rate:         today's % nulls in a column vs. its recent typical rateDistinct count:    today's distinct `customer_id` count vs. recent history

A sudden jump in a column's null rate — even one still technically inside a fixed expect_column_values_to_not_be_null threshold if that threshold were set loosely — is exactly the kind of anomaly this comparison-against-recent-history approach is built to surface.

This needs enough history to mean anything

A rolling 30-day window computed from three days of actual history produces a mean and standard deviation that do not yet reflect real typical behavior — the check will fire constantly, or never, essentially at random, until enough genuine history has accumulated. A statistical anomaly check is not usable from day one the way a fixed-threshold expectation is.

Statistical checks complement fixed rules; they do not replace them#

A fixed expectation (amount >= 0) catches an impossible value instantly, with no history required and no ambiguity about whether it is "normal." A statistical check catches a value that is individually plausible but represents a real shift from recent behavior. Neither substitutes for the other — The Testing Pyramid for Data covers why a pipeline typically needs more than one kind of check, and this is a second axis of that same principle: not just when a check runs, but what kind of wrongness it is actually capable of catching.

Common mistakes#

Using a fixed threshold for a metric that naturally trends or seasons#

A metric with genuine week-over-week growth, or a strong day-of-week pattern, makes a fixed threshold either too loose to catch real problems or too tight to survive normal variation. A rolling comparison against recent history — ideally accounting for known seasonality — handles both cases a static number cannot.

Trusting a statistical check before enough history exists#

A brand-new anomaly check on a rolling window with only a few data points is not yet meaningful. Treat it as informational until the window has accumulated enough genuine history to represent real typical behavior.

Reaching for statistical detection before a fixed rule would do#

Not every check needs a rolling window and a standard deviation — an amount that can never legitimately be negative needs a fixed rule, not statistics. Save the added complexity of a statistical check for metrics that genuinely vary over time in ways a fixed threshold cannot express.