What top Actually Shows You (And What It Doesnt)

dev.to Linux2026-10-30 01:090 views

You run top or htop. You see CPU at 80%. You think "my app is using 80% CPU." That is not what that number means.

The Breakdown

top shows you four main categories:

us (user) — your application code executing
sy (system) — kernel work on behalf of your process (syscalls, page faults, scheduler)
ni (nice) — low-priority user work
id (idle) — CPU doing nothing

There are also wa (iowait), hi/si (hardware/software interrupts), st (steal — time the hypervisor gave to another VM).

The problem: us + sy is not "your application's CPU usage." It is "CPU time accounted to your process context." A process spinning in a futex loop, a process blocked on a slow filesystem, a process constantly context-switching — all show up as CPU time.

What Actually Matters for Performance

Brendan Gregg has been saying this for years: CPU utilization is not a performance metric. It is a capacity metric.

What you actually want to know:

IPC (Instructions Per Cycle) — are you getting work done per cycle? Low IPC = memory-bound, cache misses, branch mispredicts.
Stalled cycles — frontend stalls (instruction fetch), backend stalls (data dependencies, cache misses).
Run-queue latency — how long tasks wait for CPU. This is what pidstat -w or perf sched shows you.
Context switch rate — high voluntary switches = locking/contention. High involuntary = time slice exhaustion.

The Practical Commands

# Per-process scheduler stats (voluntary/involuntary switches) pidstat -w 1 # System-wide run-queue latency perf sched latency # CPU-wide IPC and stall cycles (requires PMU access) perf stat -e cycles,instructions,cache-references,cache-misses -a sleep 10 # What top *doesnt* show you: steal time in VMs # If st is high, your hypervisor is overcommitted — your process isnt slow, it is not running top -1 # press 1 to see per-CPU breakdown

A Real Example

Go service, CPU at 90% in top. pidstat -w shows 50k voluntary context switches/sec. perf stat shows IPC of 0.3 (terrible). The culprit: a mutex hot path in a logging library. Every log call locked a global mutex. Threads spent cycles spinning, not working.

top said "90% CPU." The reality: "90% CPU, 10% useful work."