Comparing Predictions to Actuals

Whenever a system produces two measurements of the same underlying thing — a forecast and what actually happened, a sensor reading and a reference instrument, a model's estimate and a survey — the next question is always the same: how far apart are they, and in what way? This article covers the standard toolkit for answering that once the two measurements are lined up in one table.

Lining the two measurements up#

compared = predicted.merge(actual, on="entity_id", how="inner")

An inner merge keeps only rows present on both sides — an entity with a prediction but no matching actual (or the reverse) simply drops out. That is worth noticing explicitly: on a small or partial dataset, an inner join can leave very few rows, which makes the metrics below a check that the pipeline works end to end rather than a statistically meaningful verdict. The row count surviving the merge is itself worth printing before trusting anything computed from it.

Per-row error#

compared["error"] = (compared["predicted"] - compared["actual"]).abs()compared["pct_error"] = compared["error"] / compared["actual"] * 100

Every summary metric below is built from these two columns: the raw gap, and that gap as a percentage of the actual value.

Four ways to summarize the gap#

import numpy as np
mae = compared["error"].mean()rmse = np.sqrt(((compared["predicted"] - compared["actual"]) ** 2).mean())mape = compared["pct_error"].mean()correlation = compared[["actual", "predicted"]].corr().iloc[0, 1]

Each answers a different question, and reading only one of them tends to hide something:

• MAE (mean absolute error) is the average gap, in the original units — easy to explain to anyone: "off by about this much, on average."
• RMSE (root mean squared error) squares each error before averaging, which weights large misses more heavily. RMSE noticeably above MAE signals a few large errors dragging the average up, not error spread evenly across every row.
• MAPE (mean absolute percentage error) makes errors on small and large values comparable — but it becomes unstable, even exploding toward infinity, when actual is close to zero, so it is worth distrusting on near-zero rows.
• Correlation asks whether the two series move together, independent of scale. High correlation with a large MAE often means a fixable, consistent offset; low correlation means the prediction is not tracking the underlying pattern at all.

Finding the worst-matched rows#

worst = compared.sort_values("error", ascending=False).head(10)

The four summary numbers say whether predictions are trustworthy on average; they say nothing about where they are wrong. Sorting by error and taking the top rows turns a validation pass into a concrete list of cases worth a closer look before anyone relies on the results downstream.

Common mistakes#

Reading one metric in isolation#

MAE, RMSE, MAPE, and correlation each surface a different kind of problem. A report built on just one of them is usually hiding something the other three would have caught.

Trusting a small inner-join result the same as a large one#

A handful of surviving rows after the merge is a smoke test, not a statistically meaningful comparison. Treat the row count as part of the result, not a detail to skip past.

Averaging MAPE across near-zero actuals#

A single row with actual near zero can produce a percentage error in the thousands and distort the whole average. Exclude near-zero rows from MAPE specifically, or report MAE alongside it as a sanity check.

Further reading#

• scikit-learn: Regression metrics — precise definitions of MAE, RMSE, and MAPE.
• Pearson correlation coefficient — what .corr() computes and how to read its value.
• DataFrame.merge — the join type options behind lining up the two measurements.

See also#