Accuracy is a loop, not a property
Calibration in progress against a disclosed baseline — the honest version of "is it accurate".
"Is your model accurate?" is the question every vendor wants to answer with a yes. The honest answer is not a yes; it is a loop, and a loop is what an examiner can actually check.
Accuracy is not a property a model has; it is a measurement a loop produces. A model that is "accurate" without a disclosed baseline, a scoring method, and a realized timeline to score against is a model with a claim and no check. The honest version of the answer is "we measure accuracy against a disclosed baseline, via Brier and log-loss, on shadow runs against a realized timeline, and here is the loop that produced the number." That is an answer an examiner can verify; "yes" is an answer they cannot.
The loop, not the claim
In DEINO the loop is concrete: gnl_shadow_run against gnl_ground_truth, scored with Brier and log-loss via deino-eval. The shadow run is a forecast made as if it were live, held aside, and later compared to what actually happened. The baseline is disclosed — not a number we picked to look good, a number the buyer can see and challenge. Gate 1b requires a shadow run to beat that baseline on one forecastable series before any forecast side is positioned as accurate. Until Gate 1b passes, the honest statement is "calibration in progress," not "accurate."
Why "accuracy" is the wrong word
A point estimate is "accurate" or it is not, and you only know after the fact. A distribution is calibrated or it is not, and calibration is measurable on a run of outputs — Brier scores the squared error of probabilistic forecasts, log-loss scores the surprise. The Oracle normalizes into an Ensemble (sum ≈ 1.0, scenarios sorted, entropy in nats) so the shape is the thing scored, not just the mode. A model that is "accurate" on the mode but mis-calibrated on the spread is not a model a CRO can hedge with; the loop scores the spread, because the spread is the decision.
"Is it accurate?" is a question with a yes. "Is it calibrated, against what, and who scored it?" is a question with a loop. Only the second is checkable.
The tiers and the accuracy claim
The tiers treat accuracy differently. The EU AI Act (binding law) does not require accuracy; it requires transparency, logging, and oversight — it asks you to show how you got an output, not to promise the output was right. SR 11-7 (supervisory guidance) expects model risk management, which includes ongoing performance monitoring — closer to the loop than the Act, but still guidance, not statute. NIST AI RMF and ISO/IEC 42001 (voluntary) document measurement. None of them let you say "accurate" without the loop; all of them read a "yes" without a baseline as a claim that was never checked.
- Accuracy is a loop: shadow run vs ground truth, Brier and log-loss, disclosed baseline, Gate 1b.
- Calibration scores the spread, not just the mode — the spread is the decision a CRO hedges.
- The EU AI Act does not require accuracy; it requires the transparency that lets accuracy be checked.
- A "yes" without a baseline is a claim; a loop with a disclosed baseline is evidence.
What the loop does not promise yet
The loop does not promise the forecasts are accurate today; it promises they are measurable, and the gate (Gate 1b) is what converts "measurable" into "beats a baseline on one series." We are pre-revenue, building the POC; the sisters run, the ensemble normalizes, and the calibration loop is inert until it is filled. The honest posture is "calibration in progress against a disclosed baseline," and that posture is more defensible than an unmeasured "yes" — because the examiner can check the first and can only trust the second.
A model with a loop and a disclosed baseline is a model under measurement. A model with a yes and no baseline is a model under a claim. Examiners check the first and trust the second only until they ask.
— calibration note