For the two weeks after I published Sluice 0.8.0, I ran it every day on my two machines, both with 30 GiB of RAM, with up to 10 AI agents working at once on one of them. Sluice puts the tests, builds and heavy commands of all the AI agents on a Linux machine into one queue, with a shared CPU and memory budget, and runs each check once per git tree; I told how it started in the first article. Even with that queue, my agents' test suites could still run a machine out of memory. How they share it now, in short: the queue decides who runs, the kernel's ceilings keep each job within its share, and the desktop guard keeps the desktop's share free. This is what I learned, and each lesson became a version, from 0.9.0 to 0.12.2:
- A retry is still the same suite (0.9.0). Memory admission carries a gate's estimate to its variants and waits after every memory kill.
- When the machine thrashes, only the kernel arrives in time (0.10.0). Every gate job (tests, linters, type checks) runs under a memory ceiling the kernel enforces, even while everything else is stalled.
- A browser doesn't stay where you start it (0.11.0). Chromium, and so every Playwright browser, moves itself out of its job; Sluice brings it back, so its memory counts, is limited and dies with the job.
- The desktop needs a say (0.11.0). Each job gets a swap ceiling too, and no job starts while your desktop session is stalling on memory.
- The environment is part of what you test (0.11.0). Variables such as
NODE_OPTIONSgo into the cache key. - A daemon shouldn't inherit whoever starts it (0.12.0). One agent session's environment, token included, no longer reaches everyone's jobs.
A retry is still the same suite
A week after publishing 0.8.0 I went through what had happened on my second machine: 12 threads, 30 GiB of RAM, 8 GiB of swap, and between 4 and 10 agents working at once. In 37 hours there were 11 global out-of-memory kills, 7 kills by systemd-oomd, 16 test jobs killed for memory and one agent session killed along with them. Swap was full every time.
There were four causes:
- After a kill, the agents retried with
--filteror--test-concurrency=1. That was a new command for Sluice, with no history, so it guessed 1.3–1.7 GiB for a suite that needs 11–14 GiB. - The safety margin after a kill compounded up to 1.5×.
- Memory outside the queue (agents, builds, browsers) grew while a suite ran.
- The agents'
systemd-oomdwindow was 5 s, and it ended an agent 6 s after a test job was killed.
Version 0.9.0 changed the memory admission:
- A variant of a command with other arguments inherits the estimate of its gate instead of a guess.
- Only a guess can start as a probe.
- The margin after a kill is 1.25×.
- After any memory kill, nothing starts for 10 s, and not until the pressure goes down.
- The memory reserve grows to what the rest of the machine grew while heavy jobs ran (the 90th percentile, at most a quarter of the RAM).
- The agents'
systemd-oomdwindow is 15 s and the jobs' is 8 s, so the job always goes first.
sluice status now says where the reserve comes from, and flags a job that started alone beyond its estimate. No config change needed.
When the machine thrashes, only the kernel arrives in time
With 0.9.0 the same machine hit the next wall. Between 30 October and 1 November one test suite ran 61 times, every time alone on the machine and beyond its estimate: 46 passed, 9 failed their tests and 6 were killed for memory. Each of those six kills was a global OOM, and before each one the system journal fell silent for 22, 30, 37, 67, 72 and about 150 seconds. The whole machine was thrashing, and systemd-oomd and Sluice's own guard were stalled with everything else: anything that runs in userspace arrives too late.
Version 0.10.0 hands that job to the kernel. Every second the daemon writes the memory.max of each job's cgroup, so that all jobs together can never take the machine below a floor. The kernel enforces it even while the daemon is frozen. It is on by default with full containment:
# ~/.config/sluice/config.toml [admission] memory_reserve = "auto" # a tenth of the RAM: 3 GiB on 30 GiB memory_ceiling = true memory_floor = "auto" # half the reserve: 1.5 GiB on 30 GiB sluice status shows each running job's ceiling. From 2 November, the next 387 jobs on that machine ran with no OOM and no memory kill, and swap never went above 36 %.
The trade-off is deliberate: a run that used to pass by pushing the rest of the machine into swap is now killed inside its own job. The machine stays usable, and the message tells the agent to retry once memory is free, or to tell you that the gate doesn't fit.
A browser doesn't stay where you start it
On 5 November it was my development machine's turn: 20 threads, 30 GiB of RAM, 8 GiB of swap, GNOME on Wayland, Sluice 0.10.0. Between 18:40 and 19:20 a frontend test suite of about 14.5–15 GiB started at three times its estimate, Playwright jobs took 7.7–9.4 GiB each, and a production build ran outside Sluice at the same time. The machine reached 21 GiB used (66 %), swap went from 3.6 to 5.2 GiB, swap-out peaked at 1,759 pages per second, load hit 32, and GNOME Shell's input lag reached 61–80 ms. There was no OOM and nothing was killed: the 0.10.0 ceiling never engaged. The shell ended up not responding, and at 20:37 I rebooted with SysRq.
I still don't know what hung the shell: it stayed hung after the memory recovered. But looking into it I found something else. That day systemd had created 385 Chromium scopes on the machine, and that is where the browsers started by the test jobs had gone: outside their jobs.
This is what happens. At startup, every Chromium browser process, headless included, prepares the connection to the XDG desktop portal. The portal needs an app id, and Chromium gets one by asking the systemd user manager, over the session bus, for a new scope with its own pid: StartTransientUnit. The process ends up in app.slice/app-org.chromium.Chromium-<pid>.scope. There is no flag to turn it off.
For a job runner that means the browser's memory leaves the job's accounting, its learned peak, its memory.max and its kill. And Playwright starts browsers in a process group of their own, so a process-group kill misses them too.
I checked it on 6 November with a probe: a headless shell running for 6–8 seconds, about 200 MB. The browser process moved to its own scope; the renderers it had already forked stayed in the job.
The cheap fixes didn't hold up:
- Unsetting
DBUS_SESSION_BUS_ADDRESSonly works for Chromium, and breaks the tests that need the session bus. - The
FLATPAK_SANDBOX_DIRtrick, which skips the move, stopped working in Chromium on 14 October. - A cgroup namespace doesn't help: systemd makes the move from the root namespace.
What works is a property of cgroup v2. To move a process, the kernel only asks for write access to the cgroup.procs of the common ancestor of both cgroups, and that ancestor is user@UID.service, which belongs to you. So since 0.11.0, every second the daemon walks each job's process tree and writes any descendant it finds outside back into the job's cgroup. This is a job on 0.12.1 that starts Playwright's headless Chromium (156) directly. The browser leaves its cgroup 265 ms after the job starts, and it's back in the same tick:
INFO sluice_gate_daemon::server: moved a process of the job back into its cgroup job=1794597479890-504544-15a79e pid=643155 comm=chrome-headless from=/user.slice/user-$UID.slice/user@$UID.service/app.slice/app-org.chromium.Chromium-643155.scope When the job ends, whatever is left of it is killed, inside its cgroup or outside it: escapees through a pidfd, after checking the process start time, so a reused pid is never signalled. An integration test reproduces the escape with the same StartTransientUnit call.
It is on by default ([containment] reclaim_escaped = true), and it applies to every job: the setting lives in your user config, not in each gate. If your tests deliberately start something with systemd-run --user --scope, or daemonise a process that must outlive the job, the daemon will pull those back too; for that case, set it to false and Sluice only reports escapes. What Sluice itself places, in its agents scope or in any slice below its own (as when a test runs Sluice inside a job), is not an escape. Anything else, Chromium's scopes included, is moved back.
The desktop needs a say
The same version, 0.11.0, adds two more limits:
[admission] swap_ceiling = true swap_floor = "auto" # a quarter of the swap desktop_guard = true - Swap ceiling. Every second the daemon also writes each job's
memory.swap.max, so that the jobs together can't take the free swap below the floor. A job at its memory ceiling dies in its own cgroup instead of swapping the machine out. - Desktop guard. Sluice reads the memory pressure of your desktop session (
session.sliceof the systemd user manager). While it is at or above 10 % (some avg10, until it drops under 5 %), no job starts, not even alone on the machine, and the ceilings keep the whole reserve free.
Your jobs will wait while the desktop is under pressure; that's the point. sluice status shows each running job's ceiling and the session's pressure on its memory: line. Here with a single light job on an idle machine, so swap, pressure and escapes, which it only prints when they are not zero, don't show up:
The environment is part of what you test
Also in 0.11.0: a variable such as NODE_OPTIONS or RUSTFLAGS changes what a test checks without changing the tree. With key_env its value goes into the cache key, and requests with other values never share a pass. sluice init writes them per toolchain, together with the file the package manager rewrites on every install. On a throwaway npm project:
[gates.test] cmd = "npm test" weight = "auto" timeout = "30m" hooks = ["pre-push"] key_inputs = ["node --version", "npm --version"] key_env = ["NODE_OPTIONS", "NODE_ENV"] key_files = ["node_modules/.package-lock.json"] env = { CI = "1" } sluice doctor warns when a gate leaves them out:
[warn] key-env gate 'test': key_env leaves out RUSTFLAGS, RUSTDOCFLAGS, CARGO_BUILD_TARGET (a change to them reuses a pass made with other values); needs sluice >= 0.11.0 to read key_env Adding key_env changes each gate's key once, so each gate runs once more.
A daemon shouldn't inherit whoever starts it
Every job descends from the daemon, and an agent session can start the daemon on its own. Since 0.12.0 the daemon doesn't keep the environment of whoever started it, so one agent session's id, API address and bearer token never reach anyone else's jobs. A check that asks "am I under an agent?" by walking up the process tree no longer mistakes your own runs for an agent's, and merge trains, which run their gates, their prepare and their key_inputs in the daemon's environment, never see the token.
The daemon keeps only an allowlist (PATH, HOME, the locale, XDG_*, the session bus, SLUICE_* and a few more), and a sluice daemon started by hand re-executes itself clean. The log names the dropped variables that look like credentials, without showing their values. Gates started by run, verify or a hook still get the environment of whoever asked for them. The API keys your shell profile exports no longer reach the daemon: merge trains get them from your login shell (the default tool_env), and with [queue] tool_env = "daemon" you have to name the ones they need, such as a proxy, in pass_env.
# ~/.config/sluice/config.toml [daemon] pass_env = ["HTTPS_PROXY"] And a gate can drop variables from its own environment with env_remove, for markers that env can't unset.
Update to 0.12.2
sluice update It verifies the checksum, and it only stops the daemon if it is idle: one running gates keeps going until its next restart. If you use the Claude Code plugin, update it too, and then restart your agent sessions, which keep the old sluice mcp until they reconnect:
claude plugin marketplace update sluice claude plugin update sluice@sluice Then check it:
sluice doctor # daemon-version and plugin both say 0.12.2, plus any key_env warnings sluice status # the daemon version, and the memory: line with the desktop's pressure 0.12.1 fixes the escape detection of 0.11.0 and 0.12.0. If you're on either and can't update yet, turn off the escape handling in your user config:
# ~/.config/sluice/config.toml [containment] reclaim_escaped = false and restart the daemon while it is idle with sluice daemon stop (it drops queued jobs). With false Sluice neither moves processes back nor kills leftovers at the end of a job; the detection still runs, so you may see a burst of WARN lines. Once updated, remove the line to get the containment back, unless your own tests need it.
None of these versions breaks an existing config. The new keys are one-way: a shared .sluice.toml that uses key_env needs every machine on 0.11.0 or later, and one that uses env_remove, on 0.12.0.
Try it
You need Linux. Containment needs cgroup v2 and a systemd user manager, and the memory policy needs systemd-oomd with systemd 257 or later; in a container or on cgroup v1 Sluice runs without containment, and sluice doctor tells you why. The memory work in this article comes from Ubuntu 26.04 machines with kernel 7.0.
If you're new to it: on a 4-CPU GitHub runner, with 0.9.0 (median of 3 rounds), Sluice cut the time until every agent had committed from 103 to 72 s with 4 agents (−30 %) and from 213 to 119 s with 8 (−44 %).
curl -fsSL https://raw.githubusercontent.com/josegarridodigio/sluice/main/scripts/install.sh | sh sluice doctor sluice init --dry-run sluice init The code, the CHANGELOG and the decision behind each change (ADR-0026 to ADR-0030) are at github.com/josegarridodigio/sluice. If it's useful to you, a star there helps other people find it. And if you want the next version when it ships, with what I learned running it, the newsletter is just below.
