Our WebSocket server used to treat every event the same way: something happened, tell everyone. It is the obvious design and it survives contact with reality right up until a hundred people do something at once, which in a quiz is not an edge case, it is the format.

Here is what that cost, and the four changes that fixed it. None of them is clever. All of them are the same observation from a different angle: a broadcast is not one thing.

The ten thousand messages

When a player answers, the host view shows a counter: eleven of twelve tables have answered. That is genuinely useful, because it is how a host knows whether to let the clock run or move on.

The first implementation sent it like any other event. Each submitted answer triggered three queries and a broadcast to every client in the session. Do the arithmetic for a full venue on one question:

100 answers × 3 queries = 300 queries 100 answers × 100 connected phones = 10,000 messages

Ten thousand messages per question, carrying a number that is rendered on exactly one of the recipients. The other ninety-nine phones parsed it and threw it away, at the exact moment they were least able to spare the work, because they were all answering.

Two changes, in the same function. First, coalesce:

const PROGRESS_THROTTLE_MS = 250;

Answers arriving inside a 250 ms window share one query and one broadcast. The host's counter is at most a quarter second behind, which is invisible to a human reading a number that is changing anyway. The ten thousand messages become a few dozen.

Second, address it:

broadcastToSession(sessionId, { type: 'answer_received', payload: { ... } }, { audience: 'host' });

An envelope, not an event

That audience field is the general version of the fix, and it turned out to be the right abstraction for three separate problems. What travels between instances is no longer an event, it is an envelope:

export type Envelope = { event: WsEvent; /** 'host' keeps progress chatter (answer counts, joins) off the players' phones. */ audience: 'all' | 'host'; /** * Per-participant events, delivered to that player only and just before the * shared event. Used for each player's own question result. */ personal?: Record<string, WsEvent>; /** Sent to identified players who have no entry in `personal`. */ personalDefault?: WsEvent; /** Close every local socket for the session afterwards (session over). */ closeAfter?: boolean; };

audience solves the progress counter and player joins: chatter the host displays and players do not need.

personal solves something I had been dreading. When a question closes, every player needs the shared outcome and also their own individual result. The naive shape is one broadcast plus a loop of individual sends, which means two passes over the client set, two code paths that can disagree about who is connected, and an ordering bug waiting to happen when the shared event arrives before the personal one for some clients and after it for others.

As a field on the envelope it is one pass with a defined order:

if (envelope.personal && meta?.participantId) { const own = envelope.personal[meta.participantId]; const data = own ? JSON.stringify(own) : defaultPersonal; if (data) sendRaw(ws, data); } sendRaw(ws, shared);

Your result, then the shared one. Every time, for every client, including the ones that connected two instances away.

personalDefault is the case I would have missed: a player who did not answer at all still needs a result event, or their screen sits on the question while the rest of the room moves to the leaderboard. One field, instead of a diff between the answer list and the participant list.

Serialise once, not once per client

This one is almost too small to mention except that it is in the hottest loop in the application:

const shared = JSON.stringify(envelope.event); const defaultPersonal = envelope.personalDefault ? JSON.stringify(envelope.personalDefault) : null; for (const ws of clients) { ... sendRaw(ws, shared); }

JSON.stringify outside the loop. A hundred clients used to mean a hundred identical serialisations of the same leaderboard object, which for a leaderboard is not a trivial amount of string building. Now it is one, and the loop is a hundred writes of a string that already exists.

The only reason this is worth a paragraph: the obvious API is send(ws, event) and it pushes the serialisation into the per-client path where you cannot see it. The shape of the helper decided the performance.

The client that has stopped reading

The last one is about a failure I had not thought about until I saw a memory graph.

A WebSocket write does not fail when the client has gone quiet. It buffers. A phone whose WiFi has dropped but whose socket has not yet been detected as dead will happily accept writes into a buffer that nobody is draining, and your process grows for as long as the quiz lasts.

/** * A client this far behind is not keeping up (dead WiFi, frozen tab). Dropping * it is kinder than buffering without bound: it reconnects and gets a snapshot. */ const MAX_BUFFERED_BYTES = 2 * 1024 * 1024; export function sendRaw(ws: WebSocket, data: string): void { if (ws.readyState !== WebSocket.OPEN) return; if (ws.bufferedAmount > MAX_BUFFERED_BYTES) { ws.terminate(); return; } ... }

Two megabytes behind is not a slow client, it is a gone client. Terminating it is better for the client too: a dropped socket triggers reconnection, and a reconnecting client is handed a full state snapshot, so it rejoins the question the room is actually on. Draining two megabytes of backlog would walk it through the last ten minutes of a quiz that has already finished.

That is the general principle in all four of these. The system already has a cheap way to get a client back in sync, so anything that tries to deliver history is strictly worse than something that gives up and lets the sync happen.

Where the behaviour is visible

  • pub-trivia.app/features/big-screen-display is the host view the audience: 'host' events exist for. The answered-count sits on that screen and nowhere else, which is why the other ninety-nine phones no longer receive it.
  • pub-trivia.app/features/team-scoring explains why the per-participant result is a real requirement rather than a nicety: in team mode one table shares one score and one phone, in individual mode every player has their own, and the envelope is what makes the same close event serve both.
  • pub-trivia.app/features/live-leaderboard is the object being serialised once instead of a hundred times.
  • Run a free session with a laptop and a few phones. The counter on the host screen is the throttled one, and the result each phone shows before the leaderboard is its personal entry.