Shadow run vs ground truth: how calibration actually works

The names are precise. A shadow run is a forecast run that happened before the outcome was known — it ran in the shadow of the future, producing a probability distribution over scenarios. Ground truth is the realized timeline — what the world actually did, recorded after the fact. The calibration loop compares the two: the shadow run said these scenarios with these probabilities; the ground truth is this outcome; the score (Brier, log-loss via deino-eval) is the distance. Run the loop enough times and you have a calibration record — the evidence that the probabilities are probabilities, not decorations.

The shadow run is a forecast, not a backtest

A backtest runs a model against historical data it has already seen — which is useful for engineering but not for calibration, because the model was designed with knowledge of that data. A shadow run is different: it is a forecast made before the outcome, against inputs the model did not get to train on, scored after the fact. The shadow run is honest because it was genuinely ignorant when it ran. The ground truth is honest because it is what happened, not what was expected. The comparison is the measurement a backtest cannot give you.

A backtest is a model graded on data it already saw. A shadow run is a forecast graded on a future it did not know. Only the second calibrates.

What the score tells you

The score is the distance between the probability the shadow run assigned and the outcome that realized. A sharp, well-calibrated cone — high probability on the scenario that happened — scores well. A spread, high-entropy cone that assigned low probability to the realized outcome scores poorly — and that is correct, because a spread cone that missed is a cone that did not know. The score does not punish uncertainty per se; it punishes uncertainty that was wrong. A bimodal cone that put mass on the realized regime scores better than a sharp cone that bet on the wrong one. The score reads the shape, not just the peak.

The loop, not the run

One shadow run is one data point. Calibration is the accumulation: a record of scored shadow runs across series and time, building a picture of whether the cones are trustworthy. A single good score could be luck; a loop of good scores against a disclosed baseline is evidence. Gate 1b requires the shadow run to beat a disclosed baseline on one forecastable series before any forecast side is positioned as accurate — and beat means the loop, measured, on record, not one run that looked good.

• Shadow run (gnl_shadow_run): a forecast made before the outcome was known. Honest because it was ignorant.
• Ground truth (gnl_ground_truth): the realized timeline. Honest because it is what happened.
• Score with Brier and log-loss (deino-eval): the distance between the forecast and the realization. The shape matters, not just the peak.
• Calibration is the loop: one run is a data point; a scored record across time is the evidence.

What this earns

What the loop earns is the word probability. A number called a probability without a calibration loop is a decoration — a confidence dressed up as a measurement. A number called a probability with a scored shadow-run loop behind it is a probability: it has been measured against reality, it has been found calibrated or not, and the record is there to read. We are pre-revenue, building the POC; the shadow-run machinery runs, and the loop is inert until it is filled with enough scored runs to beat a baseline. The word probability is what the loop earns, and it is not earned yet.

A probability without a calibration loop is a decoration. A probability with a scored shadow-run loop behind it is a probability. The loop is what earns the word, and the word is not earned until the loop is filled.

— calibration design note