Open-source Android AI agent: say a goal, it taps, types, swipes and opens apps for you. Gemini, OpenAI, Claude. Banking apps off-limits.
tags: android, ai, opensource, showdev

cover_image: https://raw.githubusercontent.com/igwe-starking/eccles-agent/main/docs/media/social-preview.png

I typed one sentence:

"Open Facebook and post I love programming on red background."

My phone opened the app, picked the red background, typed the text, and then stopped to ask my permission before posting.

No script. No per-app integration. Just a sentence.

Eccles Agent posting on Facebook

Meet Eccles Agent: an open-source Android app that operates your phone for you.

⚡ What it can do

If a human can do it on screen, Eccles can try. In my own testing it has:

  • 🔦 Flipped system toggles. "Turn on the flashlight" opens Quick Settings, finds the switch and taps it.
  • 📘 Posted to Facebook, including picking a coloured background and typing the text.
  • 📞 Placed a call by finding a contact, opening it and pressing call.
  • 📸 Taken a photo. "Take a picture of my laptop, it's facing the rear camera" opened the camera and pressed the shutter.
  • 🚀 Opened apps reliably, even when Android refuses to launch them from the background.

Flashlight demo

🧰 The toolbox

The model answers with small JSON actions, and Eccles carries them out:

open_app · click · tap · long_press · swipe · type · key · scroll · wait

{ "thought": "Facebook is open; the composer is item 7", "status": "continue", "need_vision": false, "actions": [ {"type": "click", "target": {"index": 7, "text": "What's on your mind?"}}, {"type": "type", "text": "hello everyone"} ] }

🧠 Bring your own brain

Pick the AI you already use and paste your own API key:

  • Google Gemini (the default)
  • OpenAI
  • Anthropic Claude
  • Anything OpenAI-compatible: OpenRouter, Groq, DeepSeek, or a local Ollama or LM Studio server

Switch in Settings, then tap Test connection.

🎯 It checks its own work

Android will happily report a tap as "performed" when nothing happened. Eccles counts a node action only if the UI actually reacted. If it didn't, it re-plans from a fresh screenshot instead of blindly retrying and double-tapping a "Like". Screenshots go to the AI only when the screen tree isn't enough, and you choose whether that means never, only when needed, or every step.

🛡️ An agent that can tap anything needs a hard "no"

Safety is the architecture, not a toggle:

  • Banking and payment apps are off-limits. They're never read, screenshotted or sent to the AI, and the guard fails closed.
  • Password screens stop the task.
  • Send, post, delete and buy need your Allow. No answer in 60 seconds counts as Deny.
  • A floating Stop button sits on top of every app.
  • Your API key is encrypted with an AndroidKeystore key, never stored in plain preferences.

The confirmation gate

📲 Install it (no computer needed)

Eccles is an Accessibility Service, so on Android 13+ an APK installed from a browser gets its toggle greyed out ("Restricted setting"). The fix takes a minute:

  1. Install the APK and open it.
  2. Try to enable the service, then tap OK on the warning.
  3. Settings → Apps → Eccles Agent → ⋮ → Allow restricted settings.
  4. Switch the service on.

Or use adb install -r EcclesAgent-1.0.apk from a computer. The README has both routes and a SHA-256 to verify your download.

⚠️ Honest limits

It's an AI driving a UI, so it can get things wrong. Watch the timeline and keep Stop in reach. Whatever is on screen during a task goes to the AI provider you choose. Some apps hide from accessibility. It isn't on Google Play yet.

🚀 Try it, break it, improve it

⭐ https://github.com/igwe-starking/eccles-agent

I built this solo on a low-end Chromebook with no budget. If Eccles Agent makes you grin, you can support the project here: https://paystack.shop/pay/support-eccles-agent-tie1

The question I'd love your answer to: what should an agent like this never be allowed to do?