Go Realtime Classroom Fan-Out Under Reconnects (Channels Are Not Storage)

A realtime channel, explained operationally, carries classroom events to connected participants; it does not actually store chat history. That distinction becomes concrete when a page arrives during a live class: chat_duplicate_delivery_ratio has crossed 2% for ten minutes. The on-call view shows 186 reconnecting students, a healthy connection count, and no obvious server errors. Messages are moving. Some students are seeing the same answer twice, while others return from a network interruption and see a gap.

TL;DR: the least complex correct design is to use a realtime channel for low-latency fan-out to currently reachable participants, and a durable conversation log for identity, ordering, deduplication, and reconnect recovery. A channel carries an event now. It does not become a historical record merely because the connection stayed open. Give every accepted message a stable ID and room sequence, persist it before publication, let clients deduplicate by ID, and backfill from the log after reconnect.

This boundary matters more than the transport brand or protocol. I have been paged by missed jobs and duplicate deliveries; the useful reflex is the same here: do not ask an ephemeral delivery path to prove durable state.

Make replay explicit.

What did the realtime channel actually promise?

A realtime channel is the live delivery relationship among a publisher, a fan-out service, and subscribers that are reachable at that moment. It may be implemented over a persistent connection, but connection lifetime and data lifetime are separate concerns. The channel can authenticate a participant, route an event to a room, apply backpressure, expose liveness, and report whether a write was accepted by the next component. None of those behaviors, by themselves, establish a durable transcript.

In an edtech room, a useful event might contain a room ID, message ID, author ID, room sequence, and payload. The stable identifiers matter because transport delivery can be retried and clients can reconnect. A timestamp is useful for display and diagnosis, but it is a poor substitute for an ordering token: clocks can disagree, and two messages can share the same display precision.

WebRTC makes this separation visible. Its data channels carry arbitrary data between peers, and the specification defines channel behavior such as ordering and retransmission controls. That describes transfer semantics. It does not define a classroom message archive that a returning student can query. An application that needs history must create that history outside the channel.

The socket is not the ledger.

The clean ownership split is:

Presence is the common trap. “Student is typing” and “student is online” are short-lived observations, not transcript entries. A submitted answer, moderation decision, or teacher announcement may need durable retention because users expect it after a reconnect. Decide by product semantics, not by which events happen to share a socket.

The page starts with an ambiguous symptom

The 2% alert does not yet tell the on-call whether the server published twice, the network retried, the client merged a backfill incorrectly, or the metric counted a harmless duplicate that was suppressed before rendering. A single aggregate hides the point at which the invariant broke.

Start the trace with one message ID. The intake path should record that the command was accepted once under its idempotency key. The durable log should show one room sequence. The publisher should record attempts separately from logical publications. The client should report receipt and render decisions as different events. With those stages, the investigation can distinguish “delivered twice” from “displayed twice.” Those are different incidents.

The invariant I want in the runbook is compact: for each room sequence, the durable log contains one canonical message; a client may receive that message more than once, but it renders it at most once. Exactly-once end-to-end delivery is not assumed. The system instead combines an idempotent write, an ordered replay cursor, and deduplication at the final consumer.

A Go client-side merge can be small enough to audit. The real application would persist lastSeq locally if a process restart must resume from the same point.

package chat import "sort" type Message struct { ID string RoomID string Seq uint64 Body string } type Inbox struct { seen map[string]struct{} lastSeq uint64 } func NewInbox() *Inbox { return &Inbox{seen: make(map[string]struct{})} } func (i *Inbox) Merge(messages []Message) []Message { sort.Slice(messages, func(a, b int) bool { return messages[a].Seq < messages[b].Seq }) merged := make([]Message, 0, len(messages)) for _, message := range messages { if _, exists := i.seen[message.ID]; exists { continue } i.seen[message.ID] = struct{}{} if message.Seq > i.lastSeq { i.lastSeq = message.Seq } merged = append(merged, message) } return merged } func (i *Inbox) ResumeAfter() uint64 { return i.lastSeq } 

This example deliberately tolerates repeated receipt. It also refuses to infer missing history from connection state. When the connection returns, the client asks for messages after its durable cursor, merges that backfill with live arrivals, and suppresses IDs it has already rendered. The server must define whether the cursor is scoped to a room and what happens when retained history no longer covers it. Silence is not a recovery policy.

Instrument the boundary before tuning the alert

The earlier signal should have been a growing recovery gap, not a complaint-driven duplicate ratio. Measure the system at the boundary between accepted state and attempted delivery. At minimum, retain counts and latency distributions for accepted messages, persisted messages, publish attempts, connected recipients, reconnects, backfill results, duplicate receipts, duplicate renders, and cursor-too-old responses. Keep labels bounded; message IDs belong in traces or structured logs, not metric dimensions.

Three correlated views make the page actionable. First, track the age of the newest durable room sequence that has not yet reached the publisher. Second, track reconnect backfill latency and failures. Third, compare duplicate receipts with duplicate renders. A rise in receipts with zero duplicate renders describes extra work and capacity risk. A rise in renders is a user-visible correctness failure.

Alert on broken user invariants, then attach transport signals for diagnosis. A page for duplicate renders should identify affected rooms, client version, reconnect state, and a trace sample. A separate capacity alert can cover sustained retry amplification before correctness degrades. The runbook then has an order: confirm durable-log health, inspect the persist-to-publish lag, sample one message trace, check backfill behavior, and only then inspect connection churn.

Do not skip deployment behavior. Roll out cursor or envelope changes with readers that accept both old and new forms, then switch writers, then remove the old reader after the compatibility window. Test forced disconnects between persistence and publication, during backfill, and immediately after the live subscription resumes. Also test a duplicated event, an out-of-order arrival, a stale cursor, and two browser tabs using the same account.

Short tests catch the expensive assumptions.

Delivery guarantees are an application decision

“At most once” can lose a classroom message during a disconnect. “At least once” can repeat it. An ordered channel can preserve order for what traverses that channel while still lacking events created during an absence. These labels do not answer the product question until the boundaries are named: accepted where, acknowledged by whom, retained for how long, and ordered within which scope?

For classroom chat, I choose persist-before-publish because it gives the on-call one durable invariant to check. The sender receives success only after the message is accepted into the durable log; publication follows from that accepted record. If publication is retried, the ID stays constant. If a student reconnects between those steps, backfill and live delivery may race, which is fine because the merge is idempotent. The trade-off is visible latency from the durable write and extra storage and replay capacity. Publish-before-persist can look quicker, but a process loss between those actions can expose a message live and omit it from later history.

Retention still needs a product decision. The channel cannot decide how long a course transcript, deleted message marker, or moderation action should remain available. Define retention, deletion, and authorization on the durable store, and apply the same authorization to backfill that applies to live subscription. A reconnect must not become a way to read a room the student can no longer enter.

Capacity follows from the recovery contract. Model normal live fan-out separately from reconnect bursts: a campus network recovery can make many clients request overlapping history while live classes continue. Bound each backfill, paginate it, and keep live events mergeable with the replay. Load tests should include that overlap rather than benchmarking a steady stream on permanently healthy connections.

A useful threshold can still wake the wrong person

After adding stage-level instrumentation, tune the page around user-visible failure. Duplicate receipt rate alone is a noisy threshold because retries can increase while client deduplication continues to work. Paging on every such increase trains the on-call to ignore the signal and penalizes a recovery mechanism for doing its job.

The opposite mistake is waiting for a large duplicate-render percentage. A low global rate can hide one badly affected classroom. Use a sustained condition, a minimum event count, and a room-level view alongside the fleet aggregate. Route retry amplification to a lower-urgency capacity signal when renders remain correct; page when durable gaps grow, backfill fails, or users actually receive an inconsistent transcript. Exact thresholds must come from the service's traffic distribution and error budget, then be reviewed against real alert outcomes. There is no universal 2%.

That is the final operational boundary: a channel gives the application a path for events in motion. Storage gives it evidence of accepted state. Observability tells the on-call which side violated its contract. Confusing those jobs produces both missed messages and noisy pages.

References

• W3C, WebRTC 1.0: Real-Time Communication Between Browsers: https://www.w3.org/TR/webrtc/