Statistical Pitfalls in Everyday Data Analysis

None of the pitfalls in this article are exotic statistics — each one is a short, well-documented result that nonetheless surfaces constantly in ordinary dashboards and A/B test readouts, usually because an aggregation hid the detail that would have explained it.

Simpson's paradox: the aggregate can reverse the detail#

A well-known real dataset — two kidney stone treatments, first published by Charig et al. and widely used since to teach this exact effect — makes the point better than a constructed example could:

data = pd.DataFrame([    {"treatment": "A", "stone_size": "small", "successes": 81, "total": 87},    {"treatment": "A", "stone_size": "large", "successes": 192, "total": 263},    {"treatment": "B", "stone_size": "small", "successes": 234, "total": 270},    {"treatment": "B", "stone_size": "large", "successes": 55, "total": 80},])data["rate"] = data["successes"] / data["total"]

Grouped by stone_size, treatment A has the higher success rate for both small stones (93% vs 87%) and large stones (73% vs 69%). Yet summed across both groups:

overall = data.groupby("treatment")[["successes", "total"]].sum()overall["rate"] = overall["successes"] / overall["total"]

Treatment B comes out ahead overall — 83% vs 78% — because A was used on far more of the harder large-stone cases, dragging its combined average down even though it won every individual comparison. Simpson's paradox is this exact reversal: whenever a metric is compared across two groups that also differ in composition — device type, region, cohort size — the aggregate number can point the opposite direction from every underlying segment. Before trusting an overall comparison, check whether the two groups being compared actually have a similar mix underneath it.

Sampling bias: the filter that shapes the answer#

A metric is only as representative as the rows that made it into the calculation. A common, easy-to-miss version in a data pipeline: computing average session duration only from sessions that reached a "purchase" event, because that is the table the report happens to join against:

avg_duration = df[df["reached_purchase"]]["duration_seconds"].mean()

This measures the average duration of sessions that converted, not "average session duration" — every browsing session that left before converting is excluded, and those sessions are systematically different (often much shorter) from the ones counted. The result is not wrong arithmetic; it is sampling bias — a correct calculation over a subset that silently stopped representing the population the report claims to describe.

Correlation, confounding, and a third variable#

Ice cream sales and drowning incidents rise and fall together across the year, and neither causes the other — both follow summer temperature. A confounding variable that drives two measured quantities produces a correlation between them with no causal link at all, which is what correlation does not imply causation actually means in practice: not "correlation is meaningless," but "a correlation alone does not tell you which variable, if any, is driving the other, or whether a third one is driving both." Before treating a correlation in a dataset as actionable, ask what else changes alongside both variables — seasonality, a marketing campaign, a product launch — that could be the real driver of each independently.

Percentiles: the method changes the number#

A percentile that does not fall exactly on a data point has to be interpolated somehow, and there is more than one standard way to do it:

import numpy as np
values = list(range(1, 11))  # 1 through 10np.percentile(values, 90, method="linear")   # 9.1  (NumPy's default)np.percentile(values, 90, method="higher")   # 10np.percentile(values, 90, method="nearest")  # 9

All three are legitimate, standard interpolation methods — the p90 of the same ten numbers is 9.1, 10, or 9 depending only on which one is used. Two tools disagreeing on a reported p90 — a BI dashboard, a spreadsheet, and a Python script, say — are rarely both wrong; they usually just made different, individually defensible, and undocumented choices about interpolation.

Common mistakes#

Comparing an aggregate metric across groups with different composition#

Simpson's paradox does not require anyone to make an error — the arithmetic in each step is correct. The mistake is trusting the aggregate comparison without checking whether the underlying mix is actually comparable.

Treating a filtered average as if it described the whole population#

df[condition]["metric"].mean() describes the rows that satisfy condition, full stop. Naming that scope explicitly in a report or a column name costs nothing and prevents the number from being misread later by someone who did not write the filter.

Reporting a percentile without naming the interpolation method#

A p90 without a stated method is not fully specified — it is one of several numbers depending on convention. Naming the method, or matching whatever convention the destination dashboard already uses, avoids a mismatch that looks like a bug but is really just two unstated defaults disagreeing.

Further reading#

• Simpson's paradox — the full effect, with the classic Berkeley admissions example alongside the medical one used here.
• Sampling bias — the general category of error behind the filtered-average example.
• Confounding and Correlation does not imply causation — the reasoning behind the ice-cream-and-drowning example.
• numpy.percentile — the full list of interpolation methods, including the three compared above.

See also#