Calibration earns the word 'probability'
A number without a loop is decoration — the loop is what makes it a probability.
Any model can print a percentage. A percentage becomes a probability only after a calibration loop has scored it against a realized timeline and it has beaten a disclosed baseline. Before that, the number is decoration. After that, it is a probability. The gate between the two is the entire honesty of the product.
A probability is not a number; it is a number that has been measured. The distinction is the one most often flattened in AI marketing, where a model outputs "72%" and the word "probability" is attached to it as if the output earned the label by existing. It did not. A probability is a number whose calibration has been scored against outcomes that actually happened. Before that scoring, the number is a guess with a decimal point. The calibration loop is what converts the one into the other.
The loop, not the number
The calibration loop in DEINO is deino-eval: shadow runs against a realized timeline, scored with Brier and log-loss. A shadow run is a forecast made at time T over a window that has since closed — so the ground truth is known, not estimated. The Brier score measures how close the probabilities were to what happened; log-loss penalizes confident wrongness harder than uncertain wrongness. Together they answer the only question that earns the word "probability": when this system said 70%, did it happen about 70% of the time? Without that answer, the 70% is decoration.
The gate that protects the word
Gate 1b is the calibration gate: a shadow run must beat a disclosed baseline on one forecastable series before any forecast side is positioned as accurate. "Disclosed baseline" matters — the comparison is not against an aspiration, it is against a stated alternative the reader can evaluate. Beating a disclosed baseline on one series is a deliberately low bar; it is the first honest claim, not the last. We do not claim the system is calibrated everywhere; we claim it is calibrated somewhere, measured, and the rest is gated. The gate is what keeps the word "probability" honest.
A probability is a number that has been measured. Before the loop, the number is decoration with a decimal point. The loop is what converts one into the other.
- Shadow run: a forecast at time T over a window that has closed — ground truth is known, not estimated.
- Brier + log-loss via deino-eval: the scores that measure whether 70% meant 70%.
- Gate 1b: beat a disclosed baseline on one series before any accuracy claim — the first honest claim, not the last.
What we do not claim, and when
We are pre-revenue, building the POC. The sisters run; the distributions are produced; the normalization is guaranteed (INV-1, in deino-oracle). The calibration loop is inert until it is filled — meaning the shadow-run harness exists, but no baseline has been beaten and disclosed yet. We do not call the outputs calibrated. We call them produced, and we name the gate that the word "calibrated" is gated behind. That is the honest ordering: the mechanism is shipped, the measurement is pending, the claim is gated. Reverse the ordering and you have the marketing version of probability — the one that fails procurement.
The word "probability" is earned by a loop, not asserted by a model. The loop is the difference between a defensible forecast and a confident guess, and only one of those is something a regulator will let you keep.
— calibration note