Most agent tutorials treat the cloud API as plumbing — something that is always there. Run the same agent fully on-device and the architecture conversation changes in five places: latency shape, privacy boundary, failure modes, memory math, and what "good enough" means for the model.

Latency stops being a single number. With a cloud API, latency is a round trip plus queueing; it is variable but mostly out of your control. On-device, latency splits into prefill and decode, and each step of a GUI loop pays both. A 4B quantized model (w4a16) running on an M4 Pro handles prefill around 476 tokens/s and decode around 76 tokens/s. A screenshot-plus-instruction step that prefills 2,000 tokens costs about four seconds before the first action token appears. Nothing about the cloud was fast; the cloud just hid where the time went.

The privacy boundary moves from policy to physics. With cloud calls, privacy is a contract: screenshots go out, the vendor promises not to store them, and there is no way to verify locally. On-device, the guarantee is structural — screen content never leaves the machine. That difference decides whole categories of work: anything touching credentials, patient records, or internal tools becomes feasible without a security review that takes months.

Failure modes get honest. A cloud agent fails when the provider has a bad day, and retries are cheap. A local agent fails when memory runs out mid-task or the quantized model starts losing the thread of a long instruction chain. These failures are more debuggable — they are processes and memory maps, not status pages — but they are yours to own.

Memory math is the new capacity planning. The same 4B model peaks around 4.3GB. On a 16GB laptop that leaves a thin margin; on 32GB there is room for a second model loaded on demand. Screen perception can partly move to OS-level OCR, which costs no model memory at all. This accounting replaces rate-limit dashboards.

Model size is a floor, not a ceiling. Small models stopped being the interesting constraint once GUI-specific training arrived. A 4B model trained for GUI work ranks first on OSWorld at 58.2%, ahead of general-purpose models many times its size — evidence that task-specialized training beats raw parameter count for this workload.

The reference implementation for all of this is Mano-P, an open-source GUI agent that runs entirely on Apple Silicon with screenshots that never leave the device: https://github.com/Mininglamp-AI/Mano-P — a star helps more people find the memory numbers the hard way.