How we stream a Linux desktop into a browser tab

Gryphana is a browser-based development platform. You open a project, edit files together with your team in real time, and run the code in an isolated container with a terminal and console right in the tab.

Command line programs fit this model well. Desktop apps don't. A Swing window or a Qt tool has no place to draw, so we needed to show the container's screen in the browser, with a working mouse and keyboard. This post explains how we built that and where we hit a wall.

The pipeline

The container runs a virtual X server and we encode the screen as H.264 and send the video to the browser over WebRTC. Browsers already decode and play video well, so the client side is mostly a video tag.

Inside the container, Xvfb gives us a screen with no GPU. FFmpeg grabs the screen and encodes H.264 Baseline, a profile every browser handles. FFmpeg writes into a unix socket, not the network. On the host, outside the sandbox, a small Go program reads the socket and publishes the frames to a LiveKit room. LiveKit takes care of signaling, ICE and bandwidth estimation.

We like this split because the container never touches network credentials, and the WebRTC code never runs next to user code.

Users bring their own images, so we don't get to install anything in them. All the tooling (static FFmpeg, Xvfb, a window manager, a tiny Go init program) ships as a read-only bundle mounted into the container. The init program starts the screen and the encoder, then reports when the display is ready. A run counts as started only when there is something to look at. The bundle's libraries stay private to its own processes, and the user's command sees a DISPLAY variable and nothing else.

Mouse and keyboard

Input travels back through a LiveKit data channel. The browser sends small JSON events with coordinates normalized to 0..1, so the desktop resolution doesn't matter to the client. The Go program on the host validates the events and limits their rate, then passes them over a second socket to the init program, which injects them into the X server with XTEST.

Mouse moves use the unreliable mode of the data channel. A late position is worthless, so we drop old ones. Clicks and key events use the reliable mode, because a lost key release leaves a key stuck down inside the remote session.

A real desktop

A black screen with floating windows works for a single app. People who test on a desktop want the real thing, with a panel, dialogs and a tray.

On Ubuntu 24.04 we run GNOME this way. GNOME is still an X11 client there, so capture and input stay exactly the same. The shell needs logind and a system D-Bus, which a plain container lacks. Those jobs run under the Sysbox runtime with systemd as init. We start the user's systemd instance and launch gnome-shell on the same virtual X server.

The Wayland wall

Ubuntu 26.04 ships GNOME 50, and GNOME 50 has no X11 session. The shell is a Wayland compositor now, and on Wayland the compositor is the display server. No other process is allowed to read its pixels or inject input. Our x11grab and XTEST approach sees nothing.

On a normal machine you never notice. GDM, logind and the GPU stack do the setup, and screen sharing goes through the Mutter ScreenCast and RemoteDesktop APIs, with PipeWire carrying the frames. In a container you rebuild all of this by hand. That remains a topic for later.