Onboarding into a regulator's sandbox

This is an anonymized design-partner scenario; the institution is a fintech entering a regulator sandbox, the figures are illustrative, and no actor has agreed to anything. We are pre-revenue, building the POC. With that label on it, here is what onboarding into a sandbox looks like when the regulator is a calibration partner, not a gatekeeper with a checklist.

The sandbox is a calibration partner

A regulator sandbox is often read as a waiver — a supervised space where a model can run before it is approved. The honest reading is different: the sandbox is a calibration partner. The regulator watches the loop — shadow runs against realized outcomes, scored with Brier, the gap measured — and decides whether the loop is honest before it lets the model into production. The deliverable in the sandbox is not the model; it is the loop, and the regulator is the one reading whether the loop closes.

What the regulator sees

The regulator in the sandbox sees the calibration loop, not a dashboard. They see the shadow run against realized outcomes, the Brier and log-loss scores via deino-eval, the disclosed baseline, and the gap. They see the manifest — the NodeVersions, the model version, the agent trace — so the shadow run is reproducible by them, not just by the firm. The regulator is not asked to trust the model; they are asked to trust the loop, and the loop is the thing they can inspect.

The sandbox is not a waiver. It is a calibration partner that watches the loop and decides whether the loop is honest before it lets the model out.

• The sandbox deliverable is the loop, not the model — the regulator reads whether the loop closes.
• The regulator sees the shadow run, the Brier gap, and the manifest — the loop is reproducible by them.
• Gate 1b is the gate inside the sandbox: beat a disclosed baseline on one series before any accuracy claim.

Why the loop is the honest exit

A model exits a sandbox when the regulator is satisfied it is safe — and "safe" in a sandbox that reads the loop means "the loop is honest, the gap is measured, and the firm is not claiming accuracy it has not earned." The firm that exits is the one that showed the Brier trend, the disclosed baseline, and the manifest. The firm that does not exit is the one that brought a dashboard and a point estimate and asked the regulator to trust it. The loop is the honest exit because the loop is the thing the regulator can verify. Gate 1b has not passed in this scenario; the sandbox is the loop in motion.

A sandbox is a calibration partner that watches the loop. The model exits when the loop is trusted, not when the model is right — and the loop is the only thing the regulator can verify.

— design-partner sandbox note