I've been wiring AI agents up to real Android phones. Two things kept going wrong. The agent misclicked, and some apps ignored the taps entirely. Here is what I found and what I ended up building.
Problem 1: guessing coordinates from screenshots
The usual loop is simple. Take a screenshot, show it to a model, ask where to tap, send adb shell input tap x y.
It works until it doesn't. The screenshot you send is often scaled down from the real framebuffer. The model's coordinates are then off by some factor, and the tap lands next to the button. Animations and overlays make it worse. A floating tool or a caption bubble can sit right on top of your target.
The fix is to stop guessing. Android already knows where every button is. UIAutomator can dump the view hierarchy as XML, with text, descriptions, IDs and bounds. Find the node, take the center of its bounds, tap there.
Problem 2: synthetic input gets rejected
adb shell input is handy. But it injects a synthesized event from a shell. It is not a kernel input device. Some apps simply ignore it.
The README for my project puts it plainly: adb shell input is "a synthesized shell event, not a kernel HID device, and is rejected by some apps". Accessibility-based automation has a similar profile, since it acts through a service rather than through a device.
So I wanted input that looks like a real touchscreen and a real keyboard to the OS.
The approach
scrcpy ships a server with UHID support. UHID lets you create virtual HID devices on the phone. I use that to create a touchscreen and a keyboard at the physical input level. Then I combine it with UIAutomator node lookup for targeting.
The split is the useful part. UIAutomator only observes: it reads the screen. HID does the actual touching and typing. Each tool does the job it is good at.
A few details that mattered in practice:
- Starting a new HID session per keystroke makes the soft keyboard jump and drop input. A persistent daemon fixes that.
- Some apps ignore very short presses. Node taps hold for 120 ms.
- With a phonetic IME, hardware key sequences can turn into composition text. Paste via clipboard for names and mixed-language strings, and type digits and ASCII over HID.
-
fillfocuses a field, clears it, enters the text, and reportsOKorMISMATCH, so the agent can check its own work.
When to use which
-
adb shell input: fine for quick scripts on your own test devices where nothing objects to synthetic events. - uiautomator2: great for reading the hierarchy and clipboard work. I still use it for observation.
- HID plus node lookup: my pick when taps get ignored, or when coordinates from screenshots keep missing.
The tradeoff is setup. You need a scrcpy-server binary that matches your scrcpy version, and you provide it yourself. The repo does not bundle it.
Quick start
Commands straight from the README:
# Put the matching scrcpy-server file somewhere outside this repository. python .\hidctl.py --serial YOUR_SERIAL --server C:\tools\scrcpy-server tap 500 900 python .\ph.py --serial YOUR_SERIAL --server C:\tools\scrcpy-server ui python .\ph.py --serial YOUR_SERIAL --port 18733 daemon # In another terminal, using the same serial and port: python .\ph.py --serial YOUR_SERIAL --port 18733 tapn "Save" python .\ph.py --serial YOUR_SERIAL --port 18733 fill 500 900 "123456" python .\ph.py --serial YOUR_SERIAL --port 18733 stop You need Python 3.11+, adb on your PATH, a matching scrcpy, and pip install uiautomator2 for the UI commands. Mixing client and server versions of scrcpy is unsupported.
ui prints what is on screen with bounds and centers. tapn "Save" finds a visible node by text, description or ID and taps it. There is also an optional serial allow-list, so a script can't drive the wrong device by accident.
Safety
Only use this on devices you own or are authorized to operate. Follow the rules of the apps and networks involved.
Where it stands
I just open-sourced this under the MIT license. It is small and it has rough edges, and the README has a long list of pitfalls I hit along the way. Feedback is welcome, especially on setups I haven't tried.