Desktop automation agents mostly work by looking at pixels, coordinate math bolted onto screenshots, because every framework can demand an image from every app. macOS already publishes what to click for every app, an accessibility tree that numbers the interactive elements, and once you read that tree, the pixel path stops being a plan and becomes a guess where nothing tells you it went wrong until after the misclick.
On Nov 1 2026 I ran a tool that reads that tree end-to-end on my M4 mini, vercel-labs/agent-desktop v1.0.2, Rust core with an npm CLI layer, Apache-2.0, homepage agent-browser.dev/desktop. Every number below came back from measured calls that night, nothing from their README.
What installed, what answered
npm install -g agent-desktop landed v1.0.2 behind an install-scripts warning, postinstall runs scripts/postinstall.js, npm's allowScripts policy did not auto-allow it, yet the global package resolved and answered anyway: agent-desktop version returns {"os":"macos","target":"aarch64","version":"1.0.2"} under protocol envelope "2.5". Native binary for this box, no shim.
The permission census from agent-desktop permissions --guide read accessibility GRANTED (already on for my terminal automation chain), automation UNKNOWN, screen_recording DENIED, the guide pointing at System Settings Privacy & Security Screen Recording where the launching app gets dragged into an allowed-apps list. Every one of those states stayed exactly as it was while the tree read ran anyway, no new permission needed for it.
Two calls then disagreed. agent-desktop list-apps returned zero apps headless, consistent with their documented prohibition on foreground activation for headless calls. Direct snapshot worked either way, agent-desktop snapshot --app Finder -i returned ok=true, error=null, ref_count=162 interactive elements, complete=false, window title Finder. A partial tree, 162 numbered handles to real UI objects, obtained without a screenshot, without granting anything new, without touching coordinates.
Why stale refs get rejected and stale pixels don't
Refs are namespaced per snapshot, @<snapshot_id>:e<n>, and a bare @e8 comes back INVALID_ARGS. After the UI changes, old refs must be re-snapshotted, that's their retry-safety guarantee, and the macOS runtime keeps a private command host per snapshot namespace and build, exiting after 15 idle minutes, retaining native accessibility objects so a removed row can never silently resolve onto a surviving row sharing an identifier. TMPDIR has to be user-owned mode 0700, the standard per-user temp dir qualifies.
The pixel path has no equivalent channel. Click remembered coordinates after a layout shift and you hit whatever moved into those coordinates, the tool reports success, nothing anywhere says the target changed. The rejection channel is the safety guarantee they ship, and coordinate math has no version of it.
Where the tree loses, measured the same night
System Settings was the counterexample. My existing driver brought the window frontmost and verified it, then both element capture modes, som and ax, returned zero interactable elements for System Settings, background key delivery was refused with off_space_or_ax_unresolved, and a --request variant produced no native prompt, only the same state census again. An unindexable settings surface means autonomous coordinate climbing isn't safe there, so the remaining grant stays manual, one drag of the launching app into the Screen Recording list.
One drag later the census read accessibility GRANTED, screen_recording GRANTED, automation still UNKNOWN, their docs say that last state only matters for Notification Center and System Events flows. The screenshot verb also has argument shape worth noting: agent-desktop screenshot --app Finder <path> takes a positional path, passing --out returns INVALID_ARGS marked safe-to-retry. The Finder capture wrote 253,877 bytes at 1318 x 918 RGBA scale_factor 1.0, envelope ok=true format=png, kept local on disk. Their privacy note matches what any operator should do regardless: snapshots, clipboard reads, screenshots and debug artifacts can carry private desktop content, keep them out of public logs and uploads.
What didn't get tested
Act-by-ref clicking, click @<snapshot_id>:e<n>, never ran against a live app that night since no target was requested, each call needing a fresh namespaced ref plus re-snapshot verification after. So the verified chain is read-tree-and-number-elements, not click-element-and-confirm, half the value proposition stays unproven by me as of Nov 1, and the gap gets stated the way it stands.
Rules I'm carrying out
Prefer OS-published element trees over pixel math for desktop GUI automation, because the tree hands back numbered handles with rejection channels and the screenshot path hands you coordinates to guess at. Treat headless zeros as contract questions before breakage, list-apps returned nothing while snapshot --app answered 162, check the documented calling convention before debugging either endpoint. Re-verify refs after UI change and let the tool's INVALID_ARGS channel do the safety work, nobody assumes identifiers stay stable across rebuilds. Some surfaces are unindexable by design, System Settings returning zero interactable elements in both capture modes is where manual grants belong, coordinate climbing against an unindexable surface isn't safe autonomy.
Does your platform expose interactive element counts to agents without screenshots? Mine published 162 from Finder on macOS with only the accessibility permission that was already on, and if you've found which surfaces refuse that read besides System Settings, put them in the comments, the counterexample list matters more than the success story.
