deino-eval: Brier and log-loss, not accuracy

deino-eval is the regression and calibration harness. Its job is to take a forecast — a probability distribution over scenarios — and score it against what actually happened, once the timeline realizes. The scores are Brier (for binary and multi-category outcomes) and log-loss (for the probability assigned to the realized outcome). Both are proper scoring rules: they reward honesty, not confidence. A model that says 70% when it is 70% right scores better than a model that says 90% when it is 70% right. The score is the calibration signal.

Why Brier and log-loss, not accuracy

Accuracy is a coarse metric that hides miscalibration. The model was right 70% of the time says nothing about whether the 70% forecasts were made at 70% confidence or 90% confidence — and the difference is the whole point. A model that is right 70% of the time at 90% confidence is miscalibrated: it is overconfident, and a user hedging on its probabilities would over-hedge. Brier and log-loss penalize that: they measure the distance between the probability assigned and the outcome, so a 90% forecast on a 70%-base-rate event scores worse than a 70% forecast. Proper scoring rules are how you measure calibration, not just correctness.

Accuracy rewards overconfidence as long as the guess lands right. Brier and log-loss penalize it. The scoring rule is the honesty rule.

Scoring against a realized timeline

The harness scores against gnl_ground_truth — the realized timeline — and the forecasts come from gnl_shadow_run: a shadow run that produced a cone against the same inputs, before the outcome was known. The comparison is forecast-then-realization: the shadow run said this, the ground truth is that, the score is the distance. The loop is the calibration: each scored shadow run is a measurement, and the measurements accumulate into a calibration record. One run does not calibrate a model; a loop does.

Proper scoring rewards honesty

A proper scoring rule is one where the optimal strategy is to report your true probability. If you believe the outcome is 70%, the score that minimizes your expected penalty is the one that reports 70% — not 90%, not 50%. The scoring rule is engineered so that hedging — reporting a probability you do not believe — costs you. That is why Brier and log-loss are the right tools for a platform whose product is a probability: they make honesty the optimal behavior, and they make overconfidence visible as a bad score.

• Brier and log-loss are proper scoring rules — the optimal report is the true probability. Hedging costs.
• Accuracy hides miscalibration — 70% right is silent on whether the confidence was 70% or 90%.
• Scoring is against a realized timeline (gnl_ground_truth) from a shadow run (gnl_shadow_run) — forecast-then-realization.
• One run does not calibrate; a loop does. The calibration record is the accumulation of scored runs.

Accuracy is a loop, not a claim

The harness exists to make accuracy a measurement, not a marketing claim. Is the model accurate is a question that, in a defensible platform, is answered by here is the calibration loop and its scores against a disclosed baseline — not by yes. A yes is a promise; a loop is evidence. Gate 1b requires the shadow run to beat a disclosed baseline on one forecastable series before any forecast side is positioned as accurate — and beat means the Brier and log-loss scores are better, measured, on record. We are pre-revenue, building the POC; deino-eval runs, and the loop is inert until it is filled with scored shadow runs.

A yes to is it accurate is a promise no honest system can keep. A calibration loop with proper scoring rules against a disclosed baseline is the answer a regulator can read. The loop is the evidence; the word is not.

— deino-eval design note