Every game dev I know has the same quiet dread: the bug a player finds in the first five minutes that nobody on the team ever hit.

So when Aura open-sourced Autopilot, an Apache-2.0 agent that plays your game to find bugs, I cleared an evening. The pitch is bold: about 43% more accurate and roughly 2x faster than other AI agents at real playtesting tasks.

I wanted to see it with my own eyes. Here's what I ran, what it found, and what it cost.

What Aura Autopilot actually is

Autopilot is not one big model clicking around. It's four parts working together:

  • An engine plugin for Unreal Engine 5 (5.4–5.8), Unity 6 or Godot 4.7+. It reads the game's state and presses real keys and mouse buttons, the way a player would.
  • An agent service, a Python program on your machine. It talks to the plugin over a local WebSocket, runs model-written code in a sandbox and keeps memory for each game.
  • A coding agent (OpenCode by default, or Claude Code or Codex). It reads your source and writes a small "harness": the list of moves that make sense for your game, like "dodge left" or "open inventory".
  • A decision model called Jev, from TypeSafe. During play it picks one of those moves roughly every 0.25 seconds. Fast "reflexes" like dodging run as plain code so they don't wait for the model.

The clever bit is the harness. Aura's team found that giving a model raw keys ("press W", "click") plays badly, while game-specific moves play well. So instead of hand-writing those moves for each game, the coding agent writes them for you.

Then it remembers. Each project gets a memory folder with a copy of your code, notes, harness code that worked, and past reports. Later runs skip the research and start from a harness that already works.

You give it one of four kinds of prompts:

  • Explore: "Explore the crafting system in Lvl_Main."
  • Test: "Go through this level and tell me everything that looks wrong."
  • Verify: "Verify that the tree shakes when you hit it with an axe."
  • Goal: "Kill the boss while keeping your health above 20%."

Where the 43% and 2x numbers come from

I always read the method before I quote a number. Here it is, from Aura's own blog:

  • They tested on Easy Survival RPG V5 in Unreal Engine 5.8.
  • That was 7 tasks: 5 verify tasks and 2 goal tasks, like "craft a stone hatchet and use it to kill a zombie".
  • Each tool got one cold attempt with empty memory, on the same Windows machine, all using Claude Opus 5.5 on medium effort.
  • Accuracy: Autopilot 7/7, Claude Code 5/7, and an unnamed "leading Unreal/Unity AI agent" 4/7.
  • Mean time: Autopilot 262s, Claude Code 437s, competitor 508s.
  • Cheating (teleporting, spawning items, god mode) counted as a fail. Both of Claude Code's goal runs cheated.

So the "43% more accurate, ~2x faster" headline is against that unnamed competitor: 100% vs 57%, and 262s vs 508s. Against Claude Code it's 100% vs 71%, and about 1.7x faster.

That's a real result, but a small one: seven tasks, one game, one attempt each, and the vendor ran it. I'd treat it as a promising signal, not a law.

My test setup

My setup:

  • Machine: a Linux box with a real X display. Officially Autopilot ships for Windows and macOS, but the source runs anywhere Python 3.12 does, and the Godot addon is pure GDScript.
  • Engine: Godot 4.7.2, the Linux build.
  • Game: "Dodge the Creeps", the classic open-source Godot demo.
  • The trap: I planted 4 bugs a player would notice:

- The score keeps ticking after you die.

- The Start button never comes back after Game Over, so you're stuck.

- The player can walk 240px off the right edge of the screen and hide there.

- Moving diagonally is about 1.4x faster than moving straight.

  • I removed the git history so no tool could just read my diff.

Install from source was smooth:

  • Cloned the repo, then uv venv and uv pip install -e ".[dev]". Under a minute.
  • Ran the test suite, which uses a simulated game with no keys and no network: 492 passed, 5 skipped in 4m50s.
  • Copied addons/autopilot into the game and enabled the plugin.
  • Signed in to Aura, started the agent service with autopilot serve --aura, then launched the game with --playtest=explore --playtest-minutes=5.

What happened when I ran it

I gave it the laziest prompt possible: explore the game for bugs, five minutes of play. No hints about what I'd planted.

Here's how the run went, start to finish:

  • The game window opened and the plugin connected to the agent service.
  • The coding agent read my 10 project files and wrote a harness: moves like "start game", "hold right against the edge", "move diagonally", "steer away from the nearest creeps" and "wait on the game-over screen".
  • Before playing, it ran a quick probe to check that each move actually worked.
  • Then Jev took over and played one 150-second episode with real key presses. I watched it start a game, dodge, push into the right wall and die around 29 seconds in.
  • After the death it kept trying to start a new game. It tried 10 times, and nothing happened.
  • It wrote a report, saved notes to the game's memory and exited cleanly.

Total wall time: about 4 minutes from launch to exit (222 seconds of agent time). It also saved a video recording and a frame-by-frame flipbook of the episode.

What it found

All 4 of 4 planted bugs, each one backed by numbers from the run:

  • Walking off screen: the player reached x=720 on a 480px-wide screen. It saw this five separate times while holding right.
  • Score after death: the score went from 16 at death to 146 two minutes later, with no game running.
  • Stuck after Game Over: the Start button stayed hidden for 120 seconds, and Enter did nothing in 10 tries. It called it a softlock, which is exactly right.
  • Fast diagonal: it measured about 580 px/s diagonally against 400 px/s straight, so roughly 41% faster.

It also found a fifth issue I hadn't planted: the hidden player moves when you hold a key on the title screen, before any game starts. That's a real bug in the demo. It flagged one more thing from code reading only (D-pad mappings that look swapped) and was honest that it hadn't checked that one in play.

No false alarms. Every claim pointed to a moment in the trace.

What it cost

The run went through my Aura plan, and the report logged about $1.11 of model usage.

One surprise: I'd set the coding agent to OpenCode with Gemini, but in Aura mode the report shows Aura ran it on Claude Opus 5.5 for me. So on Aura you don't need a Claude account to get Claude doing the code-reading part. If you bring your own keys instead, you choose the model and pay that provider plus TypeSafe for Jev.

A few things to know before you try it:

  • I couldn't test the autopilot connect claude or connect codex path. That hooks Autopilot into your own Claude Code or Codex, and I don't use either plan.
  • Run sharing is on by default. I turned it off with AUTOPILOT_SHARE=0 and --playtest-share=0. Do the same if your game is private.
  • Without an Aura account or a TypeSafe key, it stops about 20 seconds in. Jev is the player, and there's no swapping it out.
  • A small Linux gotcha: if something already uses port 8765, start the service with --port and pass --playtest-url to the game. That cost me two 2-minute timeouts.

The baseline: just ask an LLM to read the code

For comparison, I gave Gemini the full source of the bugged game in one prompt: "play it in your head and list every bug a player would hit."

gemini-3-flash-preview (31.6s, about 6.7K tokens in):

  • Found all 4 planted bugs, with the right file and line for each.
  • Listed 5 extra issues. A couple were fair, like the sprite staying upside-down after moving down and then sideways. The rest were guesses or design choices, like the trail particles always emitting.

gemini-3.1-flash-lite (25.6s):

  • Found 1 of 4: the score that keeps counting.
  • Missed the stuck Start button, the off-screen hiding spot and the fast diagonal.
  • Padded the list with timing "races" that don't really happen in play.

Both runs were free on the Gemini free tier.

Autopilot vs a plain LLM

On my little test, here's how they compare:

  • Bugs found: Autopilot 4/4 plus one real extra. Gemini Flash 4/4 plus five extras, mostly noise. Flash-Lite 1/4.
  • Time: Autopilot about 4 minutes. Gemini about 30 seconds.
  • Cost: Autopilot about $1.11. Gemini $0 on the free tier.
  • Evidence: Autopilot gave me measured numbers, a trace and a video. Gemini gave me line numbers and a confident guess.

So on speed and cost, the plain LLM wins easily, and that's fair: my bugs were all visible in the code. If that's your kind of bug, a good model reading your source is hard to beat.

But the difference I felt was trust. Gemini said I could hide off-screen. Autopilot walked there and showed me x=720. Gemini mixed real bugs with design choices, and I had to sort them myself. Autopilot's list had nothing to throw away, and it found a real bug I didn't know the demo had.

And code-reading has a hard ceiling. It can't catch bugs that only show up in play, like physics, timing, level layout or a door you can clip through. That's the space Autopilot is built for, and my test didn't even reach it.

My result doesn't confirm Aura's "43% more accurate, 2x faster" claim, and it doesn't refute it. That was against other playing agents on 7 Unreal tasks. What I can say is that on a cold run, with a lazy prompt, on Linux, which isn't even officially supported, it caught everything I hid.

Should you try it?

My take:

  • Yes, if you ship on Unreal, Unity or Godot and have repeatable QA checks ("can I still open the inventory after a new game?"). Verify and goal prompts with pass/fail exit codes fit straight into CI.
  • Use a plain LLM first for cheap code review. It's faster and free. Bring Autopilot in for what code can't prove.
  • Expect it to get better per game. The memory design is the real idea here. My next run starts from a harness that already works.
  • Read the benchmark as a start. Seven tasks is a demo, not a verdict. Run it on your own game.

The repo is at github.com/ramenvr/autopilot. If you try it on your own project, tell me what it finds. I'd love to hear about a bug that only shows up in play.