What a device tool costs your coding agent: a React Native benchmark

I gave Claude Code five bugs that only show up in a running React Native app. We asked it to find the cause, fix it and check the fix on
the simulator. The app, the bugs, the prompt and the model stayed the same. Only the device tool changed: a bare shell, agent-device,
Argent and Parallax.

These runs used Parallax 0.1.0-beta.3, so the Parallax numbers describe that build. The current version is beta.11.

Every setup fixed every bug. A script checked each fix with real taps on the simulator after the agent had finished, and a separate
model graded the named causes blind. On these bugs, the device tool did not decide whether the agent succeeded. It decided what success
cost.

Parallax costs about what a bare shell costs. With Parallax, a run had a median cost of $0.07, took 36 s and used 205k tokens. With
only a shell (which on this Mac had simctl and AXe), the figures were $0.07, 40 s and 195k tokens. Parallax gives the agent the screen,
the component tree, logs and the app's stored values, and that adds next to nothing to the bill.

Results (runtime bugs, 15 runs per setup)

Cost per fix is the total cost of every run divided by the number of fixes, so it is a mean. Every other figure is a median per run. Cost
is the API list price that Claude Code reports for the run, not a bill.

What each setup did well

• agent-device had the fastest median (35.6 s) and the fewest model calls (11). It was the fastest setup on three of the five bugs (the race between two requests, the invisible overlay and the missing Intl API) and level with Parallax on a fourth (the decimal-comma price).
• Argent reaches into the app's JavaScript runtime. On the storage bug, it read the stored value through the debugger instead of navigating to it.
• The bare shell is a strong baseline. It read the app's storage files straight from the simulator's container.
• Parallax had the lowest cost per fix ($0.08). Its context stayed the smallest by the last call (23k tokens).

Where the tokens go

The model reads the whole conversation again on every call, so a run costs roughly its calls times its context. The table above has the
model calls. Context at the first and the last call (median):

• shell 17k → 26k;
• Parallax 17k → 23k;
• agent-device 18k → 32k;
• Argent 25k → 55k.

Claude Code loads each MCP tool's full description only when the agent asks for it, so a long tool list barely shows up.

One thing this benchmark changed in Parallax

The first round ran Parallax 0.1.0-beta.2. On the stale-storage bug it was slower and costlier than the bare shell: 92 s and $0.20,
against 76 s and $0.17. Without our SDK in the app, Parallax could not read the app's storage, so the agent tapped through screens to find
the value. 0.1.0-beta.3 reads stored values from the simulator. Short, non-sensitive values are shown as they are, and everything else is
masked. On the same bug, beta.3 took 63 s and cost $0.16. Both rounds are in the raw results.

Since then, 0.1.0-beta.4 added a built-in navigate command. It takes the agent to a screen through the app's own navigation, instead of
a chain of taps. We ran beta.4 on the storage bug only, three runs. The agent used navigate in every run, and the median run took 57 s
and cost $0.12 (beta.3: 63 s, $0.16). The other bugs were not rerun, and the tables above are beta.3's.

How it was run

• Claude Code 2.1.318 with Claude Sonnet 5.5, headless. Every setup got the same prompt, the same allowed tools and a fresh configuration.
• The tools, each installed by its own setup:
  • agent-device 0.21.36 (https://agent-device.dev) : npm i -g agent-device, claude mcp add agent-device -- agent-device mcp, and its skill with npx skills add callstack/agent-device;
  • Argent 0.31.1 (https://www.npmjs.com/package/@swmansion/argent) : npm i -g @swmansion/argent, then argent init --yes --no-telemetry;
  • Parallax 0.1.0-beta.3: npx -y @parallax-rn/mcp@0.1.0-beta.3 setup claude.
• The app: a small shop app on React Native 0.86.3 (new architecture, Hermes) with a local mock API. It ran on an iPhone 17 Pro simulator (iOS 26.3, Xcode 27.0) on one Mac.
• Every run, for every setup, started cold. The simulator was rebooted, and the app folder and the app's storage were reset. No tool's background process was left from an earlier run: agent-device's daemon was stopped with daemon stop --clean, and Argent's tool server with argent server stop, after each of their runs.
• A fix counts when the bug's own check passes after the agent has finished. A script relaunches the app and drives it with real taps, using none of the tools under test.
• The causes were graded blind by a separate model, with every tool name masked.
• The bugs:
  • a price string the server sends with a decimal comma;
  • a race between two requests;
  • an invisible overlay over a button;
  • a stale feature flag in AsyncStorage;
  • an Intl API that Hermes does not have. Two controls have their cause in plain sight in the code. Every setup fixed both.
• Runs: three per setup on each runtime bug and one on each control, 68 counted runs in this comparison. runs.csv also has the 17 beta.2 runs and the 3 beta.4 runs. With three runs per bug, a median is a fair middle, not a precise figure.
• Limits: one model, one app, one machine, seven bugs. This compares what the setups cost on the same work. It does not rank how capable they make an agent, and it does not test what a human and an agent do together.

Raw results, the code and corrections

The page has the method and every counted run: https://parallax-rn.dev/benchmark

Code, app, bugs and raw runs: https://github.com/parallax-rn/benchmark [pending repo]. Each bug's answer is in bugs/*/key.json
there, so don't open it if you want to try the bugs yourself first.

runs.csv has a row per run: bug, setup, run, fixed, cause, seconds, tokens, cost, model calls and device calls.

Thanks to @thymikee for reviewing the method before we published. Two of his notes are in it: every run starts
cold, and cost per fix is a mean.

If you maintain one of these tools and we set it up or used it wrongly, or if you find an error, write to us. We will fix the page and
rerun if needed.