Short answer: model room discovery as a replayable evidence stream, then measure discovered, watermark, and replayed separately. A control panel can reconnect and look healthy while showing an old device state, so one “connected” metric is not enough.

I build developer tooling, and I distrust dashboards that turn a state machine into a green dot. The useful question is narrower: after a browser loses its socket, can the panel prove which rooms it found, which events it missed, and where the backfill stopped? That evidence is the product.

This build uses a small TypeScript client, a monotonically increasing event cursor, and structured signals that work with any WebSocket or WebRTC data-channel transport. WebRTC defines the browser-facing connection primitives; it does not decide how an application discovers rooms or repairs an event gap. That boundary is where most IoT consoles get vague.

Start with the reconnect contract

Before choosing a transport, write the contract in plain language. A room is a logical stream such as plant-7/line-2, not a socket. Discovery returns a snapshot and a cursor. Live events advance that cursor. A reconnect asks for events after the last cursor, then verifies that the resulting state agrees with the snapshot.

I use four IDs in every record: roomId, connectionId, eventId, and cursor. eventId is for deduplication; cursor is for ordering. They are not interchangeable. On one early prototype I used a timestamp as the cursor. Two devices emitted the same millisecond, and the panel silently skipped one update. The visible symptom was a stale temperature card. The useful signal was a cursor gap.

Keep the protocol boring:

type Room = { roomId: string; label: string; deviceCount: number }; type RoomEvent = { roomId: string; eventId: string; cursor: number; kind: "device-state" | "device-alert"; payload: Record<string, unknown>; }; type Discovery = { rooms: Room[]; nextCursor: number; }; type DiscoverySignals = { discoveredAt: number; roomCount: number; snapshotCursor: number; connectionId: string; };

The server-side discovery response should be idempotent. Calling it twice with the same authorization context may produce a newer cursor, but it must not create a second room or mutate device state. Record the response time and the cursor, not the whole payload, in the high-cardinality metrics store. Put the room ID in logs or traces instead.

What should realtime room discovery observability measure after a reconnect?

Three signals make the failure visible. discovered says a room was returned by discovery. watermark says the client has applied every event through a cursor. replayed says backfill filled a gap after reconnect. A fourth signal, rejected, is useful when an event is outside the requested room or has an invalid cursor.

Here is the smallest client-side collector I would ship first. It deliberately emits facts, not prose. The same shape can go to an OpenTelemetry adapter, a local ring buffer, or a test assertion.

type Signal = | { name: "discovered"; roomId: string; cursor: number; connectionId: string } | { name: "watermark"; roomId: string; cursor: number; connectionId: string } | { name: "replayed"; roomId: string; from: number; to: number; connectionId: string } | { name: "rejected"; roomId: string; cursor: number; reason: string; connectionId: string }; type SignalSink = (signal: Signal) => void; export class RoomEvidence { private applied = new Map<string, number>(); constructor(private readonly sink: SignalSink) {} discovery(room: Room, cursor: number, connectionId: string): void { this.sink({ name: "discovered", roomId: room.roomId, cursor, connectionId }); } event(event: RoomEvent, connectionId: string): void { const previous = this.applied.get(event.roomId) ?? 0; if (event.cursor <= previous) return; // duplicate or an old delivery if (event.cursor > previous + 1) { this.sink({ name: "rejected", roomId: event.roomId, cursor: event.cursor, reason: `gap-after-${previous}`, connectionId, }); return; } this.applied.set(event.roomId, event.cursor); this.sink({ name: "watermark", roomId: event.roomId, cursor: event.cursor, connectionId }); } replay(roomId: string, from: number, to: number, connectionId: string): void { if (to < from) throw new Error("replay range is reversed"); this.applied.set(roomId, to); this.sink({ name: "replayed", roomId, from, to, connectionId }); } }

The gap-after-* reason is intentionally low-cardinality enough for logs, while the cursor remains a field for drill-down. In production I would avoid making an entire metric series for every room ID. Count gaps by firmware family, site, and transport; attach the room ID to a trace or sampled log.

A healthy reconnect has a recognizable sequence: a new connectionId, one discovery record per visible room, one replay record when the cursor moved, and a watermark at or beyond the discovery cursor. If the sequence ends at discovered, the panel is showing a snapshot without proof that live delivery caught up.

Ship the cursor.

For incident review, I want operators to read the signals in this order:

  • discovered without watermark: the room exists, but current state is unproven.
  • replayed followed by watermark: a reconnect gap was repaired.
  • rejected with a gap reason: pause live application and request backfill.

This tiny vocabulary also keeps alert rules understandable. A sustained rise in rejected events points at ordering or retention. A rise in discovery latency points at the index. A rise in replay span points at network churn or a long-lived tab. Those are different incidents, even if the user reports the same thing: “the panel looks wrong.”

Build the backfill loop around a cursor, not a timer

Timers are a poor repair mechanism. “Wait 500 ms and refresh” hides slow storage and makes a fast connection do needless work. The client should request the exact missing interval and stop only when its local watermark reaches the server's advertised cursor.

type Backfill = (roomId: string, after: number) => Promise<RoomEvent[]>; export async function catchUp( roomId: string, targetCursor: number, currentCursor: number, fetchEvents: Backfill, evidence: RoomEvidence, connectionId: string, apply: (event: RoomEvent) => void, ): Promise<number> { let cursor = currentCursor; while (cursor < targetCursor) { const batch = await fetchEvents(roomId, cursor); if (batch.length === 0) throw new Error(`empty backfill at cursor ${cursor}`); for (const event of batch) { if (event.cursor !== cursor + 1) { throw new Error(`non-contiguous event at cursor ${event.cursor}`); } apply(event); cursor = event.cursor; } } evidence.replay(roomId, currentCursor + 1, cursor, connectionId); return cursor; }

That loop has two useful properties for tests. It fails loudly when storage returns an empty page, and it refuses a non-contiguous batch. Your UI can then show “syncing” while the retry policy runs, rather than claiming the room is current. Retry with a bounded backoff and preserve the original connectionId in the trace; the attempt number belongs in a separate field.

The hard trade-off is retention. A short event window lowers storage work but makes a long-offline panel require a full snapshot. A long window makes backfill cheap for the client but expensive to index and secure. Choose the window from the maximum offline period you actually support, then test just beyond it. The right fallback is an explicit snapshot-required result, not an infinite replay loop.

Here is the failure that convinced me to keep the evidence separate. A panel opened six rooms, lost Wi-Fi, and reconnected with a fresh socket. Discovery completed in 180 ms, so the connection chart stayed green. Five rooms replayed normally. The sixth had a stored cursor of 812, while the snapshot advertised 819; the event store returned 813, 815, and 819 because a compaction job had moved records. A UI that only counted messages displayed a plausible device list and missed two state changes. The cursor checks rejected that batch, the trace named the room, and the operator could request a snapshot. That is a longer path than “retry until it looks right,” but it leaves an explanation behind.

Test the evidence, then decide what to change at scale

The test I want is not “the socket reconnected.” It is a timeline assertion:

const signals: Signal[] = []; const evidence = new RoomEvidence((signal) => signals.push(signal)); evidence.discovery({ roomId: "plant-7/line-2", label: "Line 2", deviceCount: 3 }, 40, "c-2"); evidence.event({ roomId: "plant-7/line-2", eventId: "e-41", cursor: 41, kind: "device-state", payload: { deviceId: "pump-4", running: true }, }, "c-2"); if (signals.at(-1)?.name !== "watermark") throw new Error("room never reached a watermark");

Add cases for duplicate delivery, a cursor gap, two rooms reconnecting in different orders, and a snapshot whose cursor is already behind the first live event. Assert signal order and cursor values. Those tests catch more real defects than checking a reconnect counter.

At scale, I would add trace links from discovery to backfill and sample payloads only when a gap occurs. I would also cap room fan-out per connection, because an IoT operator opening one site should not accidentally subscribe to every site. The catch is that this signal-first design is not suitable when your product only needs ephemeral presence and can discard history; a simpler heartbeat is enough there. Stick with a full event log when operators must explain a device command after a flaky network.

Your mileage may vary on the exact retention window. I’m not sure any universal number exists; measure offline duration, replay volume, and the percentage of sessions that need a snapshot in your own fleet. The decision rule is stable, though: if the panel cannot show its last applied cursor, it cannot prove that room discovery succeeded.

References