Sometimes your agent is confident that it has tapped a certain button at (1054, 2051) but the message is still sitting in the text box, as it is.
This exact bug report landed in our GitHub issues (#456). The user was running a vision model on a Pixel 7, and the Send button's centre was around x = 990. The tap landed 64 pixels to the right, right on the edge of the screen.
Missed taps are one of the most common problems people bring to the Mobilerun repo. They're also one of the least understood, because "the agent tapped the wrong thing" can come from at least seven different causes. So let's break down how an agent decides where to tap, and why each approach misses.
Three ways to find a tap target
Every phone agent ends up sending the same thing to the device: "touch the screen at x, y". The difference is where that x, y comes from.
1. Coordinates
This is the oldest method where you record where the button was and replay it:
adb shell input tap 990 2170Copy It's fast and costs nothing. It also breaks as soon as anything changes: another screen size, a font-size setting, a banner that pushes the layout down by 80 pixels, or an app update. Fine for a one-off script, painful for anything you want to run next month.
2. Accessibility tree
Android exposes every screen as a tree of nodes, the same data screen readers like TalkBack use. Each node has a class, an ID, text, and its bounds on screen.
The Mobilerun Portal app runs as an accessibility service and sends that tree to the agent. Here's a simplified version of what the model reads:
index. className: resourceId; checkedState, text - bounds(x1,y1,x2,y2)':12. EditText: "com.example.chat:id/input", "Message" - (24,2112,912,2232)13. ImageButton: "com.example.chat:id/send", "Send" - (936,2112,1056,2232)Copy The model doesn't guess pixels. It says "click 13", and the framework taps inside node 13's bounds. This is precise, cheap in tokens, and doesn't care about screen size.
3. Vision
The model gets a screenshot and outputs a coordinate. This works on anything a human can see, including screens with no tree at all. The cost is precision, because the model has to estimate pixels, and larger images mean more tokens.
Seven reasons taps miss
1. Scaling and DPI
This was the cause in #456. Vision models rarely see the screenshot at its native size. Many providers shrink images before the model looks at them. Anthropic, for example, resizes to a 1,568 pixel long edge. If the agent thinks the model saw a 2,048 pixel image, every coordinate comes back about 23% short, toward the top-left (1568 / 2048 ≈ 0.77). We tracked that exact case in #365.
Hard-coded coordinates have their own version of this. Android lays out in density-independent pixels, and the conversion is px = dp * (dpi / 160). A 48 dp button is 126 px on a 420 dpi phone and 96 px on a 320 dpi phone. Same layout, different pixel coordinates.
def dp_to_px(dp: float, dpi: int) -> int: return round(dp * dpi / 160)dp_to_px(48, 420) # 126dp_to_px(48, 320) # 96Copy 2. WebViews
Lots of apps render login screens, help pages and checkout flows as web content inside a WebView. Sometimes that content shows up in the accessibility tree, sometimes only partly, sometimes not at all. In #277 a user couldn't drive the WebView login of a personal finance app, and it turned out the web page itself wasn't exposing its fields. In #309, a pop-up showed up as a single empty FrameLayout with no text, so the agent had nothing to pick.
3. Flutter, games and custom canvases
Flutter draws its own pixels and builds a separate semantics tree for accessibility. If developers don't label their custom widgets, the agent sees nodes with no text. Games built on Unity or similar engines usually render to one surface, so the tree is basically one big rectangle. For these, the tree can't help, and vision is the only way in.
4. Overlapping elements
The tree can say a button is at (0,341,1080,446) while a transparent overlay sits on top of part of it. Tap the centre and you hit the overlay instead. A contributor documented this well in #199, with a case where a filter bar was partly covered by another clickable layout.
5. The keyboard covers the target
The agent types into a field, the keyboard slides up, and the button it wants next is now behind the keyboard. The tree from before typing still says the button is at the bottom of the screen. Several issues (#164, #171) came from keyboard pop-ups blocking the flow.
6. Animations and stale state
The agent reads the screen, decides, and taps, but the screen moved in between. A list was still scrolling, a dialog was fading in, or a page was loading. The tap was right for the screen that existed 300 ms ago.
7. The centre isn't the target
Tapping the centre of a node is usually right. Not always. A wide row might have its link text on the left edge, and the centre lands on empty space. #141 was exactly that, from a user who wrote their own "tap the left side of this element" tool.
How Mobilerun combines them
We don't think any one method wins, so the framework uses all three in a set order.
- The tree is the default. Taps go through element indexes. The tap point is chosen inside the element's bounds and moves away from known overlapping siblings, so the overlay case from #199 gets a clear point instead of the centre.
- The phone state is part of every step. The agent sees whether the keyboard is visible and which element has focus, so it can close the keyboard before reaching for a hidden button.
- Vision adds a screenshot to the tree. With vision, the model gets both. Pixel tools like click_at stay disabled until vision is on, so a text-only model can't guess coordinates.
- There's a coordinate contract. The screenshot is resized to the exact size the model will see, with a labelled grid drawn on it, and coordinates are converted back to device pixels. Taps outside that space are rejected and the model retries. That's the fix for #365 and #456.
- Vision-only covers the rest. For games, canvases and apps whose trees are empty, --vision-only drops the tree and works from screenshots alone.
In practice:
mobilerun run "Open Settings and turn on dark mode" # tree onlymobilerun run "Log in on the WebView screen" --vision # tree + screenshotmobilerun run "Finish the first level of the game" --vision-only # screenshots onlyCopy Debugging a missed tap
When a tap misses, work through this in order.
- Check the basics. mobilerun doctor checks your setup, and mobilerun ping confirms Portal is answering. Update first if you're on an old version, because many tap bugs in old issues are already fixed.
- Look at what the agent saw. Turn on the Portal overlay to see the rectangles it detected. Then run with --debug --save-trajectory action to save the state and screenshot for every action.
- Compare with Android's own view. If an element is missing from the agent's tree, check whether Android sees it:
adb shell uiautomator dump /sdcard/ui.xml && adb pull /sdcard/ui.xmladb shell wm size # physical resolutionadb shell wm density # dpiCopy If the node isn't in the dump either, the app isn't exposing it. Switch to --vision or --vision-only.
- Slow down for animated apps. After every action the agent waits for the screen to settle. Raise these values for apps with heavy transitions:
agent: after_sleep_action: 1.5 # seconds after each action (default 1.0) wait_for_stable_ui: 0.8 # seconds to wait for a stable UI (default 0.3)Copy - Cap the screenshot for local vision models. If you run a local model that resizes images internally, set agent.model_screenshot_max_side so the declared size matches what the model really sees.
- Try a stronger vision model. If the error "Coordinates ... are outside the ... coordinate space" keeps showing up, the guardrail is doing its job, and the model is just weak at visual grounding.
The honest part
Vision costs more. A screenshot on every step adds a lot of input tokens, so use it where the tree fails, not everywhere. Some apps will stay hard. Apps that block screenshots with FLAG_SECURE give vision nothing to look at. Apps with no accessibility labels give the tree nothing to read. Apps that do both are hard for every agent, ours included.
And some misses aren't about taps at all. If the agent taps the right button on the wrong screen, that's a planning problem. reasoning helps there more than any coordinate fix.
Frequently Asked Questions
Is the accessibility tree better than vision?
Why does my agent tap a few pixels off with vision on? Can Mobilerun play games?
The takeaway
Read the tree when the app gives you one, add eyes when it doesn't, and always check what the agent actually saw before blaming the model. If you want to see the tree and the screenshot for every step on real devices, try it at cloud.mobilerun.ai.
Also Read:

