Traceability across a model swap
Pin the model by hash, not by name — swap the live model, the chain still verifies.
Every model-risk team swaps models. The traceability question is whether the decision made by the old model is still reconstructable after the swap. Most teams discover the answer is no — at the audit.
A common compliance failure: the log records that model v2 produced output X, then the team swaps to model v3, and v2 is deprecated, archived, or deleted. An examiner asks about output X eighteen months later, and the log says "model v2" — but v2 is gone. The log is now uncheckable. The decision is traceable in name and untraceable in fact. This is the failure mode every model-risk team hits eventually, and the one most teams discover at the audit, not before it. The fix is structural, not procedural: pin the model by hash, not by name, and make the provenance record provider-agnostic so the chain still verifies after the live model has changed.
We are pre-revenue and building the POC. This post is about the traceability-across-swaps property the manifest (BR-842) is designed for, and why pinning by hash is the structural fix.
A name is not a reference
A model name — "gpt-4o-2024-08-06", "claude-3.5-sonnet", "llama-3.1-70b" — is a label, not a reference. The label can be reused, the underlying weights can be updated, and the provider can deprecate the label entirely. A log that records the label has recorded a pointer to something that may no longer exist. A hash — the cryptographic hash of the model weights, or a provider-issued content hash — is a reference. The hash either matches the weights that produced the output or it does not, independent of whether the label still exists. The manifest pins the model by hash; the label is metadata, not the identity.
Provider-agnostic provenance
The manifest is provider-agnostic: the model_version field carries a hash, not a provider-specific identifier. DEINO uses OpenAI-compatible providers (OpenAI, vLLM, OpenRouter, Together) with no direct Anthropic SDK dependency — and the manifest does not care which one produced the output, because the hash is the identity. Swap the live model from one provider to another; the historical manifest still verifies against the hash of the model that produced it. Bring your own keys, own your history: the chain survives the vendor, because the identity is the hash, not the vendor.
A name is a label; a hash is a reference. Pin the model by hash and the chain still verifies after the vendor is swapped, the weights are updated, or the label is deprecated.
- Pin the model by hash, not by name — the hash is the identity; the label is metadata.
- Provider-agnostic provenance: the manifest carries a hash, not a vendor identifier, so the chain survives a swap.
- Swap the live model; the historical manifest still verifies against the hash of the model that produced the output.
- Bring your own keys, own your history — vendor lock-in is a compliance risk, and the hash is the structural fix.
What this costs and what it buys
Pinning by hash costs a discipline: the team has to record the hash at the time of the run, not reconstruct it later. That discipline is what the manifest enforces by construction — the model_version field is written when the output is produced, into the hash chain, and it cannot be retroactively edited without breaking the chain. What it buys is traceability across swaps: the decision made by the old model is reconstructable after the swap because the hash still resolves, even if the name does not. The eighteen-month-old decision is defensible because the bytes and their hash are still there.
The log that records a name is a log that expires when the name does. The log that records a hash is a log that survives the swap — and the swap is inevitable.
— traceability-across-swaps note
The honest scope
The manifest attests provenance and reproducibility — never truth. A hash-pinned record of a wrong decision is still a record of a wrong decision; pinning by hash is not a claim of accuracy. Calibration is a separate, gated loop (Gate 1b). What hash-pinning gives you is the ability to answer the examiner after a swap, not the ability to answer "was the decision right." The first is a property of the substrate; the second is a property of the loop, and the honesty is in keeping them separate.