From 21 Minutes to 5½: Parallel Tests With Nothing but xargs

An AI agent once told me my test suite had hung.

It hadn't. The suite was just slow. The agent watched all 960 cases print ok, waited, decided the
wrapper was stuck "during post-suite cleanup", and stopped it. When I ran it again unattended, it finished with exit code 0 and 960 of 960 passing. It had been working the whole time, at 34% CPU, for 14
and a half minutes.

That was September, with 960 cases. By November the suite had grown to about 4,600 tests (BATS and Python), and the full offline run, make test-all, took 1,238 seconds on my M4 MacBook Air: about
21 minutes. Every fix I verify and every agent change I land waits on that number.

On 2026-11-09 I measured what running files side by side would buy. The best result was 338 seconds, 3.7 times faster. It needed no new tool: just xargs, which already ships with macOS and every
Linux.

The problem: slow tests change how you work

Twenty-one minutes is long enough to change behaviour, mine and the agents':

• Agents read silence as failure. A slow, healthy suite looks idle, because it is I/O and subprocess bound (lots of bash, git, stub binaries), not CPU bound.
• I run subsets instead of the whole thing, and subsets miss what they don't cover.
• A red run costs a second 21 minutes to confirm the fix.

The suite is mostly BATS: thousands of small bash processes per file. Each file is independent of the others by design. Every test gets a scratch HOME and TMPDIR and its own stubs. That isolation was
written for safety, but it's also exactly what parallel execution needs.

The core idea: one job per file, N at a time

find scripts/tests -name '*.bats' -print0 | xargs -0 -n 1 -P 4 bats 

• -0 reads NUL-separated names, so no filename can split.
• -n 1 gives each bats invocation one file.
• -P 4 keeps four of them running at once.

I considered GNU parallel and dropped it. It's an extra install on four machines (two Macs, a hub and Linux control nodes), and every machine that differs is a machine that can break differently. Both BSD xargs (macOS) and GNU xargs take -0 -n -P -r. I checked both on 2026-11-09, macOS 26's /usr/bin/xargs and GNU findutils 4.9.0.

Per-file is coarse. A single slow file can't be split by xargs. That turned out to matter more than anything else (see the floor below).

How many jobs? Fewer than you'd think

The M4 Air has 10 cores: 4 performance and 6 efficiency. All runs were on the same commit (b9c16615, 330 files), one job per file:

Going from 4 jobs to 10 saved 22 seconds. Going from 4 jobs in arbitrary order to 4 jobs in the right order saved 82.

Two things explain the table.

The extra cores are slow cores. Past four jobs, the new ones land on efficiency cores and contend for disk and process creation. So the runner defaults to the performance-core count when the OS reports
it (sysctl -n hw.perflevel0.physicalcpu on Apple silicon), then to getconf _NPROCESSORS_ONLN (portable), then to 1.

The slowest file is the floor. scripts/tests/bin/codex_dispatch.bats takes 292 seconds alone and
346 under load. No job count can finish the run before its slowest file does. If that file starts last, it adds its whole length to the end of the run. Starting it first lets the other 329 files run in its
shadow.

That's the old longest processing time first scheduling rule, and it costs one text file: each run saves seconds file lines, and the next run starts the slowest files first. Files it has never seen run
before everything else, because unknown is more dangerous than slow.

The next step isn't more cores either: it's splitting that one 292-second file into files of about 60 seconds each, moving cases unchanged.

What parallelism brought out

Running in parallel didn't break anything that worked. It showed what was already fragile.

1. A load race

At 6 jobs, one Python test (test_webhook_analyze_routing.py, a case checking a 413 response) failed with ConnectionResetError. That test is timing-sensitive, and serial runs had hidden it. The same 13
known-red files failed at every job count, and nothing else changed. That's the check that matters:
parallel must give the same per-file result as serial, or the speed is worthless.

So the runner's retry rule is strict. A failed file reruns once, alone, after the pool drains. If it passes, it's reported FLAKY by name, and the run stays red unless you explicitly pass --flaky-ok. A blanket retry would turn every race into a silent pass.

2. A dropped last item

The runner reads its input with read -r -d ''. That loop silently drops the last item when the input doesn't end with a NUL, because read returns non-zero on the final partial record. An agent wrote the first version. Review caught the drop, and the fix is one clause:

while IFS= read -r -d '' item || [[ -n "$item" ]]; do items+=("$item"); done 

A test runner that loses the last test file is the worst kind of bug: everything it does run is green.

3. Two xargs dialects for failure

When a child fails, BSD xargs exits 1 and GNU xargs exits 123. GNU also runs the command once on empty input unless you pass -r. Rather than decode exit codes per platform, the worker always exits 0 and writes each file's real exit code, log and seconds to its own files. The summary is built from those files, never from xargs's exit status. Its format (the names and times here are illustrative):

PASS scripts/tests/lib/system.bats 12s FAIL scripts/tests/bin/example.bats 41s (rc 1) SLOWEST scripts/tests/bin/example.bats 41s TOTAL 45s 

4. Parallel multiplies leaks

My previous article, My Test Suite Has 4,656 Tests. Some of Them Were Lying. (link once published), was about tests that reached the real machine: one rewrote a real launchd agent, another a real
credentials file. Run four at a time, and a leak happens four times as often, and they can race each other. So the plan made two fixes prerequisites before the parallel runner joins the main suite: the
tripwire's new check that no real file under HOME changed, and moving the tests that touched the real macOS Keychain behind it. Each job still runs inside the tripwire, in its own TMPDIR.

The AI workflow, honestly

The runner is a small bash function, _run_parallel, in my shared library (lib-foundation). An agent implemented it from a spec with an Acceptance block. Every job runs exactly once with at most N at a time. Slowest-first order is honoured. A flaky file is named. A failure can never produce a green result, whatever the job count or ordering. Then I reviewed it, and the trailing-item drop came out of that review, not out of its tests.

The agent's "the suite hung" report back in September came from the same root. A long, quiet process looks broken to a reader that can't tell slow from stuck. The fix there wasn't patience. It was checking
the log's modification time against the clock: a log written seconds ago at case 642 of 960 is a slow suite, not a hang.

Where it stands

Honest status, as of 2026-11-11:

• The land fast gate, which runs only the suites a change touches, already uses xargs -0 -n 3 -P. Today that gate takes about 90 seconds, and one file (machine_setup_acceptance.bats) accounts for 77 to 83 of them. Same lesson as above: find the floor first.
• _run_parallel is built and tested on a lib-foundation branch (5670513), not yet released.
• make test-all PARALLEL=1, the codex-dispatch file split, and keeping each stage's completion marker for the test dashboard are planned for k3d-manager v1.44.0. Serial stays the default and the reference.
• CI still runs its three suites one after another. That's the next easy gain, and it waits until the local runner has had a clean month.

Final thoughts

Four things, in the order they paid off:

1. Isolation first. Tests that share nothing can run side by side for free.
2. Order beats cores. Slowest-first on 4 cores beat arbitrary order on 10.
3. Find the floor. One file set the minimum time. Split it before buying parallelism.
4. Same result or no deal. Parallel must match serial file for file, and a retry must name what it retried.

None of it needed a new dependency. Both BSD and GNU xargs have had -P for decades.

The runner spec and measurements are in the repo:
github.com/wilddog64/k3d-manager
(docs/plans/v1.44.0-parallel-offline-tests.md, scripts/ci/fast_gate.sh).