What broke while up to 30 AI agents built my coding agent, and the rule each failure produced

Muse Spark Code, my open source coding agent for Meta's Muse Spark, is mostly built by other agents. Claude orchestrates. Codex and Muse agents build in small lanes, and a different agent reviews every lane before it merges. At its peak, one orchestrator ran up to 30 coding and review agents across three rigs: a Kubuntu box, a Mac mini and a Windows 11 VM. Since the repo went up on September 22 it has merged 229 pull requests and shipped 41 releases.

It also broke, a lot. Every failure that generalises goes into a register in the repo, with what happened, the rule that prevents it, and the milestone in the app that has to enforce that rule. It has 129 numbered entries now. These are the ones that taught me the most.

Silence is never success

A status poll failed, and the orchestrator read the failure as "still running". Finished Windows lanes went unreported for 40 minutes. Rule: a job is running, finished, or unknown since {time}. A failed poll makes it unknown, never running. (G5)
A launch script died on set -e -o pipefail because ls a b | head failed when one path was missing. Two lanes never started, and nothing said so. Rule: every launch is confirmed by a liveness probe (process alive, log growing within 30 s) or reported as failed to start. (G35)
The fleet watcher took a lane tool's own output line, <lane> Gradle head exit=1, for the runner's exit marker and reported a running lane as done. Rule: completion comes only from the runner's own wait on the process. Nothing a lane can write can mark it complete. (G53)

Don't believe the agent's summary

Agents' final messages overstated their results: "checked", "all fixed". Work is now complete only on receipts, such as the hook verdict, test counts and hashes (G19). Two lanes exited 0 at their step budget with edits after their last commit and a final message saying incomplete. Done now means a clean tree, every brief step done, and a moved head (G36).

The orchestrator had the same problem. It reported what it had and hadn't done from recollection, and was wrong (G107). It kept follow-ups like "resume at 11:20" as notes in files, and forgot them (G123). Status now comes from records, and every deferred action is a timer the harness holds.

Unreachable is not dead

The Mac mini dropped off Wi-Fi for 30 minutes. All 11 of its jobs kept running, but two were relaunched elsewhere, which wasted work and produced conflicting results (G6). Agents that were quiet for a few minutes were taken for dead and relaunched while still alive (G103).

Rules: never duplicate a worker's jobs because it looks unreachable, and re-dispatch only after its lease expires. An agent is dead only when its process is confirmed gone; a quiet one is reported as stalled.

Cleanup is where the damage happens

A sweep that picked temp folders by name shape matched three macOS daemon folders, and ran a recursive permission change on them before the delete was refused (G55). Rule: delete only exact paths recorded at creation, re-checked by identity. Never select by name.
On Windows, node_modules is often a junction, and a recursive delete through it deletes the target (G16).
On macOS, du reported 49 GB in a browser's code-sign clone folders. Deleting all of them freed nothing, because APFS clones share blocks (G56).
pkill -f <pattern> matched its own calling shell and killed the caller (G90).

Windows has its own physics

Over SSH, the non-interactive desktop heap (768 KB) capped concurrent processes. Resets started at about seven concurrent lanes, so Windows rigs cap at five (G51).
Three commits that failed lint-staged landed anyway. Each was git commit … 2>&1 | Select-Object -First N in PowerShell: the reader left early, husky's runtime died of SIGPIPE, and Git for Windows read that signal death as exit 0 (G83).
lint-staged handed eslint.cmd 123 files at once and hit cmd.exe's 8,191-character command-line limit (G76).
TZ=... date in Git Bash silently printed UTC and misdated lane records (G91).

Git shares more than you think

A lane ran git stash and git stash pop. Stashes are shared by every worktree of a repository, so the pop applied another lane's lint-staged backup into it (G59). Lanes never stash now.

A test's push started a detached git maintenance run --auto in its local origin, which repacked while the test was deleting its temp folder (G65). Every Git command a job runs now has automatic maintenance off.

Lane-green is not tree-green

Fourteen lanes each passed their own gates. The joined tree failed the duplication gate (10 clones), dead-code checks, two bundle budgets and the startup baseline, and no single lane owned any of it (G73).

A worker with leftover build outputs and fast hardware reported the full suite green, and hosted CI on clean checkouts failed 23 jobs (G43). Integration now re-runs the full gate after the last merge, from a clean tree. A worker's green result is labelled worker-certified unless it ran that way.

Reviews need a stopping rule

One review took about 12 hours of one-finding rounds (G18). Now there is one full review pass per change, and every finding is fixed at once. A second fix round must be structural, and a third failed round means a redesign.

Where the rules go

The coding agents were not the hard part. The scheduling, liveness and review discipline around them was.

Muse Spark Code has its own orchestration milestones: teams and a scheduler, paired devices, a resource governor, an orchestrator playbook and a capacity estimator. Each register row names the milestone that has to enforce its rule and says whether it already does. Anyone running their own fleet will hit the same failures, so the register is public: