In our 2D virtual office, voice reaches only nearby people because the media server refuses to forward it to anyone else. The app server works out who is near whom and gives each speaker a list of allowed listeners. Each speaker's client passes that list to LiveKit, our SFU, as track subscription permissions, so nothing is mixed on the server and a modified listening browser cannot pull a voice it was not given.

An SFU (selective forwarding unit) forwards each person's tracks instead of combining them. The stack here is a Python (FastAPI) WebSocket server, a React and Phaser client, and self-hosted LiveKit, which is Apache-2.0 open source.

Why not mix audio on the server?

A mixer gives you privacy for free. The server decodes every microphone and encodes one mix per listener, so each browser only ever receives its own mix. There is no other voice to steal.

The price is one mix per listener. On a map, people rarely share the same set of voices at the same volumes, so nearly every listener needs separate decode, mix, and encode work.

An SFU does none of that. It forwards encoded packets, each voice stays a separate track, and the browser sets the volume per track. Each office is one LiveKit room, so walking around only changes subscriptions.

The trade-off is that the free privacy boundary is gone. Every browser in the room is one request away from every track. You have to build the boundary yourself.

How does the server decide who is near?

The client sends only movement. The server owns every position and rejects a move of more than 2 tiles or into a wall. Positions go out on a 100 ms tick, and after each tick with movement the server recomputes near: the set of people whose voice should reach a given listener. Distance is Chebyshev: max(|dx|, |dy|).

Where you are Your near Volume
Open floor The 8 closest people within 3 tiles, excluding people in meeting rooms 1.0 at 0 to 1 tiles, 0.7 at 2, 0.45 at 3
Meeting room Everyone in the same room, no cap Always 1.0
Focus desk Nobody, and you are in nobody's near No voice

The server sends near only when it changes. Note the cap: it makes near directional. In a crowd, you can be in my 8 closest while I am not in yours.

Version 1: the listener chose

The client joined with autoSubscribe: false and subscribed to the people in near (condensed):

// Version 1: the listener decides alone for (const pub of participant.trackPublications.values()) { pub.setSubscribed(nearIds.has(memberIdOf(participant))) }

It looked right on screen, but it trusted the listener's browser. By default, any published track can be subscribed to by all participants. Calling setSubscribed(true) from the developer tools would let you hear someone at the far end of the map. The access token cannot help: its grants cover the whole room, and proximity changes as people walk.

Version 2: the speaker's permission, enforced by the SFU

LiveKit lets a publisher restrict who may subscribe to its tracks:

room.localParticipant.setTrackSubscriptionPermissions( false, // do not allow everyone [{ participantIdentity: 'u42', allowAll: true }], // allow only these )

Anyone left off the list gets nothing from the SFU, whatever their browser does. The call comes from the speaker's client, but the decision should not. Positions live on the server, so the server computes each list and pushes it down.

Server: invert near to get allow

allow is the set of people who may receive my voice and video. It is not my near. Because near is directional, copying it would admit people who should not hear me and block people who should. The correct set is everyone whose near contains me, so the server inverts all the near sets:

def compute_allow(members, gmap): allow = {m.id: set() for m in members} for listener in members: for speaker in listener.near: if speaker in allow: allow[speaker].add(listener.id) # In a meeting room, add everyone in the same room  # so that allow cannot disagree with near  meetings = {} for m in members: zone = gmap.zone_by_id(m.zone) if m.zone else None if zone and zone.kind == "meeting": meetings.setdefault(zone.id, []).append(m.id) for ids in meetings.values(): for mid in ids: allow[mid].update(i for i in ids if i != mid) return {mid: frozenset(ids) for mid, ids in allow.items()}

The server recomputes allow with near. It sends {"t": "allow", "ids": [...]} only to members whose list changed, plus once right after they join.

Client: pass the list to the SFU 

private applyAllow(ids: string[] | null): void { // null (not received yet, or WebSocket down) means: allow nobody this.room.localParticipant.setTrackSubscriptionPermissions( false, (ids ?? []).map((id) => ({ participantIdentity: `u${id}`, allowAll: true })), ) }

allowAll: true covers every track a speaker publishes: microphone, camera, and screen share. This matters because LiveKit's reference notes that track-level grants do not extend to tracks published later.

The listening side did not change: near still drives setSubscribed. Subscription says what a listener wants. Permission says what a listener may have.

Six pitfalls

1. The default is open

Right after joining, the first allow has not arrived, and LiveKit's default lets all participants subscribe. A microphone published in that gap is open to the whole room. So the client sets "allow nobody" before it joins. The SDK sends that on connect, and we send the current list again before publishing the microphone.

2. A dropped WebSocket must revoke everything

While the WebSocket is down, the client cannot hear about revocations, so a stale permission could outlive its reason. On disconnect the client discards allow and allows nobody. The next welcome does not restore it; only a fresh allow does. A network blip can cut your voice for a moment, and we accept that.

3. Send allow before near

The server sends allow first and near second. A revocation reaches the SFU sooner, and someone walking up is more likely to be permitted before their browser asks to subscribe.

4. An old allow must not land last

Sending is async, so another recompute can run during an await. If each pass sent its own snapshot, a stale list could arrive last and restore a revoked permission. So the server computes and assigns with no await in between, then reads the current value at send time:

for m in allow_changed: # m.allow is read here, at send time  await self.send(m, {"t": "allow", "ids": _id_order(m.allow)})

Whichever pass sends last sends the newest value.

5. Permission can arrive late

A listener can ask for a track before the speaker's permission lands. LiveKit connects the subscription on its own once permission arrives. As a safety net, the client waits 600 ms after a grant or a failed subscription, then asks again if it still wants the track, up to 5 times per track (simplified):

const stillWanted = this.desired.get(sid) === true && // still in near p.trackPublications.get(sid) === pub && !pub.isSubscribed && pub.permissionStatus === TrackPublication.PermissionStatus.Allowed if (stillWanted) pub.setSubscribed(true)

6. Meeting rooms need one rule for both lists

Meeting rooms first inherited the cap of 8, so from the ninth person on, voices could not be heard. We removed the cap there. near and allow now both mean everyone in the room, and nobody outside it.

How do we test it?

Most of this needs no media server. The server tests use a fake WebSocket that records messages in order:

  • On the open floor, allow is the inverse of near.
  • With 8 people packed around a listener, a speaker 3 tiles away falls outside the cap, so that speaker's allow leaves the listener out, but not the reverse.
  • In a meeting room of 12, everyone reaches everyone and someone just outside the door reaches nobody inside. Whoever walks out drops from the other 11 lists.
  • A focus desk gives empty lists in both directions.
  • allow arrives before near, and a volume-only change resends near but not allow.

To test pitfall 4, we stall one socket so that a second recompute overtakes the first (condensed):

gate = asyncio.Event() class SlowWS(FakeWS): async def send_text(self, text): await super().send_text(text) if len(self.sent) == 1: await gate.wait() # stall after the first message  a.ws = SlowWS() first = asyncio.create_task(room.refresh_near()) for _ in range(5): await asyncio.sleep(0) # let it reach the stall b.x = 30 # b walks away await room.refresh_near() # a newer pass overtakes gate.set() await first assert a.ws.of("allow")[-1]["ids"] == []

The client tests stay off the network too. They fake LiveKit's connection and log, in order, what would go to the SFU. They check that "allow nobody" comes before the microphone, that a WebSocket drop revokes everything until a new allow arrives, that the last list is re-sent on every rejoin, and that retries stop at 5 and skip anyone outside near.

Finally, with real audio on a local setup, a far-away browser forced a subscription and received 0 bytes.

What does this not protect?

  • A speaker's own voice. The speaker's client applies the permissions, so a modified speaker browser can open its own voice to anyone. It still cannot reach anyone else's.
  • Walking up to listen. Anyone nearby can hear. The server rejects jumps, so getting close means walking there, visible on the map.
  • Metadata. Everyone in the room still sees who is present and whose microphone is muted.
  • The SFU itself. This design controls who receives a track. It does not hide audio from whoever runs the server.

What to take from this

If you build proximity voice on an SFU, treat subscription as a preference and permission as the boundary. Compute allow on the server by inverting near, not by copying it. And fail closed whenever the client cannot be sure: before the first allow, and while the control channel is down.

Sources

We use this in Hotori (https://hotoriplace.com/en/), a voice-first virtual office that runs in the browser, made in Tokyo by a small team.