The page says visible_couriers_stale > 8% for 5m. Dispatchers see drivers frozen two streets behind their actual position, while the ingest graph still looks healthy. That combination is the first warning that scaling realtime fan-out publishing for event delivery has failed at the map: accepting location events is not the same as delivering a current view to every viewer.

The answer is to scale realtime fan-out publishing around bounded-staleness presence, not raw connection count. Partition by delivery area, coalesce replaceable positions, attach monotonic sequence numbers, and give reconnecting clients a snapshot-plus-replay path. Page on user-visible age and recovery failure. Queue depth belongs in the diagnosis, but it is a poor primary signal because a deep queue can contain obsolete coordinates that should never reach a screen.

This distinction matters outside logistics too. A game chat room that survives reconnects has the same underlying presence problem: the service must distinguish a live member from an old socket and restore the room view without replaying duplicate messages. The payload differs; the state-machine discipline doesn't.

What should page first when realtime fan-out event delivery makes a tracking map stale?

Start with the screen's promise. For a delivery map, that promise is usually closer to “show the latest accepted courier position within an agreed age” than “deliver every coordinate.” Those are different contracts. If a courier sends points faster than a browser can render them, delivering all intermediate points increases lag while making the map less current.

The first page should therefore combine impact and duration: the proportion of actively viewed couriers whose displayed position age exceeds the service objective, sustained long enough to ignore a brief deploy or network handoff. The 8% for 5m values above are example policy values, not universal defaults. A dense urban dispatch desk and a customer watching one doorstep need separate objectives. I'm not sure one global threshold can represent both; per-view objectives and replay traces would settle that decision.

Work backwards from the alert in a fixed order. First, verify the browser's last applied sequence and snapshot timestamp. Next, compare them with the latest sequence accepted for that courier. Then inspect the subscriber's partition, pending bytes, coalesced updates, disconnect reason, and reconnect attempt. Only after that should the runbook inspect aggregate publish rate or host saturation. This order keeps the investigation tied to the incorrect map rather than whichever infrastructure chart happens to be red.

The useful signal is an age decomposition:

Interval Begins Ends What it isolates
ingest age device timestamp server acceptance device or upstream delay
fan-out age server acceptance subscriber write routing and backpressure
apply age subscriber write client application transport and client work
recovery age reconnect start current view restored snapshot and replay behavior

Do not merge these into one average. Averages conceal the tail, and the tail is where a courier marker becomes operationally misleading. Report distributions by area partition and client class, with cardinality controlled at collection time; a raw courier ID belongs in sampled traces, not as a permanent metric label.

One more trap: connection presence is not application presence. A transport can remain open while the consumer falls behind, and a new socket can overlap an old one during a mobile handoff. Presence should be a leased application record with an epoch, an expiry, and a last-applied sequence. The newest epoch owns the visible session. Older epochs may finish in-flight work, but their writes cannot move state backwards.

Build a fan-out path that discards obsolete work

Treat courier coordinates as replaceable state and delivery milestones as durable events. A position at sequence 418 supersedes position 417 for a live map; a “delivered” transition must not disappear merely because a newer coordinate arrived. Putting both classes into one undifferentiated FIFO either wastes capacity on old positions or risks dropping business events under pressure.

A practical pipeline has four boundaries. Ingest validates identity and assigns a monotonic sequence within the courier stream. A partition owner maintains the latest position and appends durable transitions. The fan-out layer routes only to authorized viewers of that delivery area. Each client tracks an epoch and last-applied sequence, so duplicates are harmless and regression is rejected. This is the idempotency reflex: assume redelivery before an incident proves that assumption necessary.

Keep the partition key aligned with the unit of viewing and failure containment. An area or route partition lets one busy district shed intermediate positions without delaying unrelated districts. A courier-only partition simplifies ordering but scatters one map subscription across many owners. Neither choice is free — area partitions can become hot around major events, while courier partitions increase subscription coordination. Measure skew with representative traffic before committing to either.

The following Go sketch captures the state transition at the subscriber boundary. It isn't a broker client or a full protocol. The important behavior is that an old epoch or sequence cannot overwrite newer state.

package presence import "sync" type Update struct { CourierID string Epoch uint64 Sequence uint64 Latitude float64 Longitude float64 } type Cursor struct { Epoch uint64 Sequence uint64 } type Store struct { mu sync.Mutex cursors map[string]Cursor } func NewStore() *Store { return &Store{cursors: make(map[string]Cursor)} } func (s *Store) Apply(u Update) bool { s.mu.Lock() defer s.mu.Unlock() current := s.cursors[u.CourierID] if u.Epoch < current.Epoch { return false } if u.Epoch == current.Epoch && u.Sequence <= current.Sequence { return false } s.cursors[u.CourierID] = Cursor{Epoch: u.Epoch, Sequence: u.Sequence} return true }

Short code. Important invariant.

Backpressure policy should be explicit at every hop. For replaceable positions, retain the newest update per courier and count how many were coalesced. For durable transitions, reject unbounded buffering: apply a finite admission limit, preserve the event in durable storage, and let the consumer resume from its cursor. Never claim success merely because bytes entered a queue.

Reconnect with a snapshot, a cursor, and one owner

Reconnect is where apparently healthy realtime systems duplicate or regress state. Walk through a concrete, hypothetical trace: a client has applied sequence 418 when its mobile transport drops, the partition owner accepts 419 through 423 during the gap, and a replacement connection opens with epoch 12 while the old epoch 11 still appears alive. The new connection asks to recover from 418. If the server joins it to live traffic immediately, sequence 424 can reach the browser before a snapshot containing 423; applying that snapshot without a guard then moves the marker backwards. If both epochs can acknowledge, the owner may also mistake progress on the abandoned connection for progress on the visible one. The correct trace records the recovery boundary, makes epoch 12 authoritative, sends a snapshot at sequence 423, ignores later writes from epoch 11, and admits live sequence 424 only after the snapshot is installed. The browser still checks every epoch and sequence because network delivery can be duplicated or reordered around the handoff. This example is deliberately small enough for a deterministic test, yet it exercises the same ownership race that becomes hard to see across thousands of connections.

Use a small recovery protocol. The client presents its last cursor and session epoch. The server establishes a recovery boundary, returns a current snapshot at that boundary, and replays durable events after the boundary before exposing live updates. Replaceable coordinates need only the current value; durable delivery transitions need ordered replay. Every applied item carries enough ordering metadata for the client to reject duplicates.

Make ownership unambiguous. When a reconnect creates epoch 12, epoch 11 is stale even if its socket has not closed. This prevents two live transports from alternately updating presence or acknowledging the same transition. It also gives the on-call a concrete question: which epoch owns this courier, and what cursor has each viewer applied?

For game chat, the same method restores a room roster and message cursor: snapshot membership, replay messages after the boundary, then join the live stream. WebRTC can carry peer media and data, but transport choice doesn't remove the need for application-level epochs, authorization, ordering, and recovery. A server-mediated map will often favor a different transport topology because one courier update must reach many independently authorized viewers. The selection rule is the state contract, not the novelty of the wire protocol.

Test the race, not just the happy path. In a deterministic harness, pause delivery after the recovery boundary, publish two updates, expire the old epoch, and release messages in reverse order. Assert that the displayed sequence never decreases, a durable transition appears once in the materialized view, and the reconnect duration is observable. Then repeat with a hot partition and a slow client. These tests turn “reconnect seems fine” into invariants a deployment can enforce.

Tune the alert without training the team to ignore it

After adding client-applied sequence, epoch, and recovery timing, the earlier page becomes actionable. The responder can separate slow ingest from fan-out lag, distinguish a stale session from a current one, and see whether recovery restores the promised view. A deploy gate can compare the tail of display age and recovery age between the candidate and baseline, while synthetic clients continuously force reconnects across multiple partitions.

The catch is false-positive cost. A threshold that pages on every brief radio handoff teaches the team to distrust the only signal tied to customer impact. A threshold that waits for a fleet-wide average misses one dispatcher's hot area. Use a ticket or dashboard for small, isolated staleness; page when the affected-view ratio and duration threaten the explicit objective. Keep separate alerts for durable-event loss, authorization failure, and recovery exhaustion because their actions differ.

This approach is not suitable when every intermediate sample is legally or scientifically significant. In that case, don't coalesce the canonical stream: retain every accepted event in durable storage and build the map as a derived latest-value view. It is also a poor fit for tiny deployments where a single process and periodic snapshot already meet the objective; added partition ownership and replay machinery would create more failure modes than it removes.

Capacity planning should model three quantities independently: accepted updates, concurrently interested viewers, and bytes actually written after coalescing. Multiplying producer rate by all connected clients assumes every viewer watches every courier and every point deserves delivery. That estimate is easy, but it hides the controls that matter: subscription scope, replacement ratio, payload size, and slow-consumer limits. Load tests should preserve their correlations, including shift changes and geographically hot areas, instead of generating uniform random traffic.

Close the loop in the postmortem. Record which user-visible signal fired, which earlier signal would have shortened detection, which invariant failed, and which limit contained the blast radius. Don't celebrate a lower queue depth unless display age improved. The map is the service.

References

Further reading