deino-ingest: the signal pipeline

A raw external feed is bytes with a shape only the feed knows. A typed NodeVersion is a versioned, hash-linked record on the GNL. Between them is deino-ingest — the pipeline that turns one into the other without lying about the transformation.

The pipeline, in one line

deino-ingest takes a raw external feed, normalizes it through a Strategy mapper into a GeoSignal (for atlas sources) or a typed event (for GNL ingestion), validates the fields against the schema, and writes a NodeVersion to the ledger. The raw bytes never reach the GNL; the typed, validated, versioned record does. The pipeline is the boundary where untrusted external data becomes a typed internal record, and the type discipline is the whole point.

Validate at the boundary

The rule, restated from the architecture: validate at system boundaries, fail fast with clear errors, never trust external data. A raw feed is external data; it gets validated the moment it enters the platform. A field that does not match the schema is a rejection, not a best-effort parse. The alternative — parse loosely, store whatever came in, hope the consumer handles it — is how a world model fills with garbage that surfaces at query time. deino-ingest rejects early, so the GNL never holds a malformed node.

From GeoSignal to NodeVersion

For the atlas sources, the path is feed to Strategy mapper to GeoSignal to cache. That is the World Monitor gateway, and it lives in deino-atlas. For the GNL ingestion path, the step beyond the cache is NodeVersion creation: the typed event becomes a new version of a node on L1, L2, L3, or L4, linked to its predecessor, hash-chained under BR-842. The ingest pipeline does not decide which layer; the typed event carries its layer, and the ledger appends the version. That is the append-only, immutable property: ingest creates, it never overwrites.

The raw bytes do not survive the crossing; the type does. The pipeline is the boundary where untrusted external data becomes a typed internal record.

What the manifest records

Every NodeVersion the pipeline produces is part of the provenance chain. The audit manifest for an output that consumed an ingested node records the NodeVersion refs — so an auditor can see exactly which version of which feed the output read. Ingest is not a side channel; it is part of the reproducibility record. A replay of the output replays the ingest, because the NodeVersions are immutable and the hash chain points at them.

• Validate at the boundary — a malformed field is a rejection, not a best-effort parse.
• Ingest creates NodeVersions; it never overwrites — append-only, hash-linked (BR-842).
• The manifest records the ingested NodeVersion refs — ingest is part of reproducibility, not a side channel.

The honesty of rejection

A pipeline that accepts everything is not a pipeline; it is a funnel. The value of deino-ingest is partly in what it rejects: the malformed feed, the field that changed shape, the identifier that does not resolve. Each rejection is a signal that the upstream changed, which is the kind of thing you want to know at ingest time, not at query time. A pipeline that fails loud at the boundary is a pipeline that keeps the world model clean, even when it means a feed is dark for a day while someone fixes the mapper.

A pipeline that accepts everything is a funnel, not a pipeline. The rejections are the signal that the upstream changed — and you want that signal at ingest, not at query.

— deino-ingest design note

What it is not

deino-ingest does not certify that the external feed is accurate. It certifies that the typed record is a faithful, validated, versioned copy of what the feed said. Accuracy of the feed is a property of the source; calibration of any forecast that consumes it is gated (Gate 1b). We are pre-revenue and building the POC; the pipeline is built, and its honesty is that it types and validates, it does not verify.