The backup window opens at 01:00. By 01:05 the database's p99 query latency has gone from 4 ms to 90 ms, and the on-call alert fires. The backup is not corrupt, not slow, and not even using much CPU. It is simply writing 400 MB/s to the same NVMe device the database lives on, and the database is now waiting in the block layer.
The usual first response is ionice -c3. It doesn't work, and it hasn't worked for years on most systems. The second response is a cgroup v2 io.max limit, which does work, but only if you understand what it does and does not throttle.
This is a practical tour of the io controller: what the three knobs actually do, how to wire them to systemd, how to prove the limit is being enforced, and the traps that make people conclude "cgroups don't work" when they do.
Why ionice stopped helping
ionice sets an I/O priority on a process with ioprio_set(2). Whether that priority means anything depends entirely on the block scheduler:
- BFQ acts on I/O priorities and also has cgroup weight support.
- mq-deadline and none do not schedule on ioprio at all. On NVMe,
noneis the common default.
CFQ, the scheduler that made ionice famous, was removed in Linux 5.0. If your disk is running mq-deadline or none, passing -c3 records a value and nothing reads it.
Check what you actually have:
lsblk -o NAME,MAJ:MIN,SIZE,ROTA,SCHED # or per device cat /sys/block/nvme0n1/queue/scheduler Even where BFQ is in use, ioprio is proportional, not a ceiling. A backup at idle priority will still consume every spare IOPS the database isn't using, and on a fast device "spare IOPS" is enough to ruin tail latency.
What the io controller actually is
The cgroup v2 io controller is not one mechanism. It is several, with different guarantees:
| File | Mechanism | Guarantee | Costs |
|---|---|---|---|
io.max | blk-throttle | Hard ceiling on bytes/s or IOPS/s per device | Not work-conserving; unused capacity is wasted |
io.weight | iocost or BFQ | Proportional share of I/O time among siblings | Only active if iocost is enabled or BFQ is the scheduler |
io.latency | blk-iolatency | Protects a group's latency target by throttling peers | Only compares siblings; observes, then reacts |
Two design details explain most surprises:
-
io.maxis enforced on block-layer requests, not onwrite(2)calls. Buffered writes go into page cache first and become device I/O later, during writeback. A process can dirty hundreds of megabytes before the throttle ever sees them. - The kernel documentation explicitly permits bursts: "BPS and IOPS are measured in each IO direction and IOs are delayed if limit is reached. Temporary bursts are allowed."
So io.max is a ceiling on sustained device traffic, not a per-syscall speed limiter. That is fine for a backup job, as long as you measure the right thing.
Step 0: confirm the pieces exist
# unified hierarchy? stat -fc %T /sys/fs/cgroup # expected: cgroup2fs # is the io controller compiled in and delegated? cat /sys/fs/cgroup/cgroup.controllers # expected to include: cpuset cpu io memory pids # device numbers (the io files are keyed by MAJ:MIN) lsblk -o NAME,MAJ:MIN,SIZE,ROTA If io is missing from cgroup.controllers, the kernel was built without CONFIG_BLK_CGROUP and none of this applies. If it is present but not in cgroup.subtree_control, the controller is not delegated to children yet.
Also check the device, not the partition. systemd.resource-control(5) is explicit that these settings should reference "the originating, physical device", not a partition or loop device.
Step 1: give the backup its own slice
Resource control needs a boundary. Putting the limit on the backup service works, but a slice is better: everything the job spawns (restic, rclone, a shell pipeline) inherits the same budget, and you can raise the limit later without editing the service.
/etc/systemd/system/backup.slice:
[Unit] Description=Background maintenance workloads [Slice] IOAccounting=yes CPUWeight=20 /etc/systemd/system/restic-backup.service:
[Unit] Description=Nightly restic backup After=network-online.target Wants=network-online.target [Service] Type=oneshot User=backup ExecStart=/usr/local/bin/backup.sh Slice=backup.slice IOSchedulingClass=best-effort IOSchedulingPriority=7 The IOScheduling* lines are advisory on most modern schedulers. They cost nothing and help on BFQ systems, but do not rely on them.
Step 2: put a hard ceiling on it
/etc/systemd/system/backup.slice.d/io.conf:
[Slice] IOAccounting=yes IOReadBandwidthMax=/dev/nvme0n1 200M IOWriteBandwidthMax=/dev/nvme0n1 100M IOWriteIOPSMax=/dev/nvme0n1 2K Three things worth knowing before you copy that:
- Suffixes are base 1000.
100Mis 100,000,000 bytes per second, not 104,857,600. The man page says so directly. If you were thinking in MiB, your limit is about 4.6% lower than you intended. - Bandwidth and IOPS both apply. A 100 MB/s write cap with a 2,000 IOPS cap means the IOPS cap bites first for small random writes and the bandwidth cap bites first for large sequential ones. That is usually what you want for a backup: bound the streaming rate, and stop it from shredding the device with metadata storms.
- The file path is resolved to a device. Pointing at
/srvworks for simple layouts. It does not work through RAID or most volume managers, so name the physical device explicitly on anything more complex than LUKS on a partition.
Apply and inspect:
sudo systemctl daemon-reload sudo systemctl restart restic-backup.service cat /sys/fs/cgroup/backup.slice/io.max # 259:0 rbps=200000000 wbps=100000000 riops=max wiops=2000 That line is the ground truth. If it is missing, the controller is not active for the slice, and no amount of restarting the service will help.
Step 3: prove the limit is real
io.stat is a live per-device counter for the cgroup:
cat /sys/fs/cgroup/backup.slice/io.stat # 259:0 rbytes=1459200 wbytes=314773504 rios=192 wios=353 dbytes=0 dios=0 Read it before and after a run. wbytes divided by wall-clock seconds should land near your cap, not far above it. On a fast NVMe without a limit, a restic run will show write throughput in the hundreds of MB/s; with the cap it should flatten out.
For a controlled test, use fio with direct I/O inside the same slice:
sudo systemd-run --unit=fiotest --property=Slice=backup.slice --wait --pipe \ fio --name=seqwrite --filename=/srv/fio-testfile --size=2G \ --rw=write --bs=1M --direct=1 --iodepth=16 \ --runtime=30 --time_based --group_reporting Expect roughly 100 MB/s. If you see 2 GB/s, the limit is not attached: check io.max again, and confirm the unit really is in backup.slice with systemctl status fiotest.
A one-liner also works for a quick sanity check outside systemd:
sudo systemd-run --scope --property=IOReadBandwidthMax="/dev/nvme0n1 20M" \ dd if=/dev/nvme0n1 of=/dev/null bs=1M count=200 iflag=direct The traps
Buffered writes lie to you. dd without oflag=direct or conv=fsync reports the speed of copying into page cache, which is memory speed, not disk speed. It will happily report 3 GB/s while the cgroup is capped at 100 MB/s, because the bytes have not reached the device yet. Always test with direct=1 or force writeback.
Cgroup writeback needs filesystem support. The kernel attributes writeback to a cgroup only on ext2, ext4, btrfs, f2fs, and xfs. On other filesystems, writeback I/O is charged to the root cgroup, so a limit on your backup slice will not catch data the page cache flushes later. Check findmnt -no FSTYPE /srv before trusting the numbers.
io.weight is inert more often than people think. It defaults to 100 and ranges from 1 to 10,000, but proportional control is provided by iocost or BFQ. On a stock system running none or mq-deadline, weights can be set and read back while changing nothing. Confirm the mechanism is live:
cat /sys/fs/cgroup/io.cost.qos # empty until a device line has been written; enable=1 turns iocost on io.latency only compares siblings. It is work-conserving: when everyone meets their target, nothing is throttled. When one group misses, its peers with higher targets get queue-depth throttling and artificial delays. Set it on the groups that compete at the same level, not on a single child three levels down.
io.latency is blind without debug stats. The missed and total fields only appear when blkcg_debug_stats is enabled, and they are off by default. Without them, you are tuning a controller you cannot observe.
Network filesystems are out of scope. There is no block device to throttle. io.max on an NFS-backed backup directory does nothing.
%util on NVMe is not a saturation signal. It pins at 100% while the device still has headroom. Watch await and r_await from iostat -x 1, or io.pressure, which reports actual stall time.
cat /sys/fs/cgroup/backup.slice/io.pressure # some avg10=0.00 avg60=1.42 avg300=0.31 total=... The other tool: protect the database instead
A hard cap is the right answer when you know the budget. When you don't, invert the problem: tell the kernel what latency the database needs, and let it throttle whoever gets in the way.
/etc/systemd/system/db.slice.d/io.conf:
[Slice] IOAccounting=yes IODeviceLatencyTargetSec=/dev/nvme0n1 25ms And on the competing slice, a looser target:
[Slice] IOAccounting=yes IODeviceLatencyTargetSec=/dev/nvme0n1 100ms Now when db.slice misses 25 ms, backup.slice is throttled until it recovers. When the database is idle, the backup runs at full speed. No static budget, no wasted capacity.
Two caveats: this needs CONFIG_BLK_CGROUP_IOLATENCY, and the target must be achievable. Setting 1 ms on a device that cannot deliver it means permanent throttling of everything else and no benefit to anyone. Measure first, then set the target slightly above the latency you see under normal load.
Which one should you use?
- Predictable ceiling, guaranteed isolation:
io.max. Works regardless of scheduler, does not need iocost, and is easy to explain in a postmortem. - Fair sharing with no hard cap:
io.weight, after confirming iocost is enabled or BFQ is in use. - Protecting a latency-sensitive workload from everything else:
io.latency, as a complement to a cap, not a replacement.
Most hosts want io.max on the known offenders (backups, container image pulls, log shipping) and io.latency on the workload that must stay responsive. Weights are the nicest model and the easiest to misconfigure.
Rollback
Remove the drop-in and reload:
sudo rm /etc/systemd/system/backup.slice.d/io.conf sudo systemctl daemon-reload cat /sys/fs/cgroup/backup.slice/io.max # 259:0 rbps=max wbps=max riops=max wiops=max max in every field means the limit is gone. That is the check to run after every change, in both directions: the file is the configuration, and it never lies about what the kernel is enforcing.
References
- Linux kernel documentation, Control Group v2, sections "IO" and "Writeback"
- systemd.resource-control(5), "IO Accounting and Control"
- systemd.slice(5)
- ioprio_set(2) and ionice(1)
- Linux kernel documentation, Pressure Stall Information
- Linux kernel documentation, BFQ I/O scheduler
If a backup job is degrading your database, the fix is a budget, not a priority. Give the job a slice, give the slice a ceiling, and verify the ceiling in io.max — the rest is tuning.
