deino-atlas: a gateway, not a scraper

The Facade pattern is the shape. deino-atlas runs one background poller per source on a fixed schedule with a token-bucket budget. Each poller pulls an external feed, normalizes the result to a GeoSignal (Strategy mappers in sources.rs), upserts the signal into a durable Postgres cache, and hands the delta to a GeoPublisher (Observer). Clients read the cache. They never touch the upstream. The upstream call volume is bounded by our schedule, not by the number of clients — one poller, one rate; ten thousand clients, same rate.

One poller per source, on a schedule

Each source has a SourceDescriptor in a registry — a data record, not a code change. The descriptor names the source, the endpoint, the env var for the key, the schedule, the mapper. The poller runs on that schedule: fetch, map, upsert, publish. The schedule is a token-bucket, so a burst does not exceed the upstream’s tolerance and a lull does not starve the cache. The poller is the only thing that talks to the upstream; everything else talks to the cache. That is the Facade: one front, many readers, one upstream.

A scraper’s upstream call volume is the clients’ to multiply. A gateway’s is ours to bound. One poller, one schedule, ten thousand clients — same upstream rate.

Clients read the cache, never the upstream

The cache is a durable Postgres table — the geo cache, in the same one database the whole platform runs on. A client asking for a snapshot reads from the cache; a client on the realtime WS reads the deltas the GeoPublisher broadcasts. Neither goes upstream. The property this gives you is the one a scraper cannot: the upstream does not care how many clients you have. A scraper calls the upstream once per client per request; a gateway calls it once per poll, regardless of clients.

Sources are data, not code

Add a feed by adding a SourceDescriptor to the registry. The engine does not change; the mapper is the only code, and the descriptor is the wiring. A source whose key_env is unset self-disables — it returns Ok(None), the registry marks it Disabled — so a missing key never breaks the platform. The feed is absent, not broken. A missing key degrades gracefully, an unreachable upstream is a poll that produced nothing, and the cache keeps the last known state.

• One poller per source on a fixed schedule (token-bucket) — upstream call volume is ours to bound, not the clients’.
• Clients read the durable Postgres cache, never the upstream. The Facade is the boundary.
• Sources are data (SourceDescriptor) — add a feed without touching the engine; a missing key self-disables.
• Realtime per tile: one broadcast channel per geohash, viewport-only deltas, gzip when asked.

Realtime, per tile

The realtime endpoint is GET /api/v1/atlas/ws — a WebSocket, user-JWT-authed via query token, with per-user connection caps. The deltas stream through GeoHub, which holds one broadcast channel per geohash tile. A client receives only the deltas for its viewport’s tiles — not the whole world. The tiling is geohash prefixes, computed in deino-core::geo, no PostGIS. Clients that send gzip:true get gzip Binary frames. The realtime surface is authed but not per-request rate-limited; the limit is the connection, because a stream is not a request.

Why a gateway, not a scraper

A scraper fetches what a client asks for, when the client asks — and every client is an upstream call. A gateway inverts the relationship. The upstream is polled on our schedule; the clients read our cache; the upstream’s rate limit is a problem between us and the upstream, not between the client and the data. For a planetary-scale monitor, that is the only shape that survives — one schedule, one cache, many readers. We are pre-revenue, building the POC; the gateway runs, the cache holds, and the calibration of anything that forecasts from this data is gated (Gate 1b) until it is measured.

A scraper makes the upstream’s limits the client’s problem. A gateway makes the upstream’s limits our problem and gives the client a cache. For a planetary-scale monitor, only one of those shapes survives.

— deino-atlas design note