On one disk, at one moment, the same call to shutil.disk_usage told one free-space guard to start a 2 GiB batch of camera clips and told another to refuse it. The only difference between the two guards was which fields of that one result they read.

Here is the check, fed one reading taken from a real mount, asking for 2 GiB:

# Two versions of a "is there room for the next clip?" check, fed the # reading printed by read_mount.py so the result does not drift. from collections import namedtuple usage = namedtuple("usage", "total used free") u = usage(total=10464022528, used=8166391808, free=1743982592) need = 2 * 2**30 # room asked for: 2 GiB  for name, room in [("u.total - u.used", u.total - u.used), ("u.free", u.free)]: verdict = "start" if room >= need else "refuse" print("%-17s %11d bytes %.2f GiB -> %s" % (name, room, room / 2**30, verdict)) 
u.total - u.used 2297630720 bytes 2.14 GiB -> start u.free 1743982592 bytes 1.62 GiB -> refuse

The two numbers differ by 553,648,128 bytes, a little over half a GiB. Both are correct arithmetic. They answer two different questions, and only one of them is the question a clip writer running as an ordinary user should be asking.

Disclosure: I make an Android camera app. Nothing below uses its code or describes how it records or stores anything. Every output is pasted unedited from the scripts shown, run as an ordinary user (not root) in a Linux sandbox with Python 3.10 and e2fsprogs 1.46.5.

Free blocks, and free blocks you are allowed to use

shutil.disk_usage on Linux is a thin wrapper over statvfs. The Linux man page for statvfs(3) describes two separate counts of free space in the struct it fills in. f_bfree is the "Number of free blocks". f_bavail is the "Number of free blocks for unprivileged users".

The gap between them is space the file system holds back for the super-user. On ext4 it is set when the file system is created. The mke2fs(8) man page describes the -m option this way: "Specify the percentage of the file system blocks reserved for the super-user." It gives the reason, "This avoids fragmentation, and allows root-owned daemons, such as syslogd(8), to continue to function correctly after non-privileged processes are prevented from writing to the file system." And it gives the default: "The default percentage is 5%."

So an ordinary process can run out of room while f_bfree still says there is plenty.

Here is the raw reading from the mount used above (read_mount.py), next to what shutil.disk_usage returned for the same path:

import os, shutil, sys path = sys.argv[1] st = os.statvfs(path) f = st.f_frsize print("f_frsize ", f) print("f_blocks ", st.f_blocks) print("f_bfree ", st.f_bfree) print("f_bavail ", st.f_bavail) print("reserved ", st.f_bfree - st.f_bavail, "blocks =", (st.f_bfree - st.f_bavail) * f, "bytes") u = shutil.disk_usage(path) print(u) print("total - (used + free) =", u.total - (u.used + u.free)) 
f_frsize 4096 f_blocks 2554693 f_bfree 560945 f_bavail 425777 reserved 135168 blocks = 553648128 bytes usage(total=10464022528, used=8166391808, free=1743982592) total - (used + free) = 553648128

f_bfree minus f_bavail is 135,168 blocks of 4,096 bytes, which is the 553,648,128 bytes from the first section. I did not format this mount and do not know which options were used. The reserve comes to about 5.3% of its blocks, not exactly 5%, so I would not assume the default on a disk you did not set up yourself.

The last line is the part that surprises people. total is not used + free. It is used + free + reserved.

Why total - used is the wrong subtraction

Python 3.10's shutil.disk_usage builds its three numbers like this (from Lib/shutil.py, the Unix branch, with the indentation removed):

st = os.statvfs(path) free = st.f_bavail * st.f_frsize total = st.f_blocks * st.f_frsize used = (st.f_blocks - st.f_bfree) * st.f_frsize

free comes from f_bavail, the count for unprivileged users. used comes from f_bfree, the count that includes the reserve. So total - used works out to f_bfree * f_frsize, the free space a root process could use. free is the free space everyone else can use.

On a laptop disk where the job runs as you, the version that subtracts is the one that is wrong, and wrong in the dangerous direction: it reports more room than the writer has. The guard in the first section starts a 2 GiB batch with 1.62 GiB of usable space.

The same reasoning in a shell script lands on the same mistake if it takes df's total and subtracts df's used. df prints an Avail column for a reason.

A reserve you can see without root

I wanted to check the 5% default directly instead of trusting the arithmetic on a disk I did not create. mkfs.ext4, dumpe2fs and tune2fs all work on a plain file, so this needs no root and no mount:

#!/usr/bin/env bash # Format a sparse 1 GiB image file as ext4 with default settings, read the # reserve from the superblock, then lower it to 1% and read it again. # Needs no root and no mount: mkfs, dumpe2fs and tune2fs work on a plain file. set -eu dir=$(mktemp -d) img="$dir/card.img" truncate -s 1G "$img" mkfs.ext4 -q -F "$img" dumpe2fs -h "$img" 2>/dev/null | grep -E '^(Block count|Reserved block count|Block size|Reserved blocks uid):' tune2fs -m 1 "$img" | tail -n 1 dumpe2fs -h "$img" 2>/dev/null | grep -E '^Reserved block count:' rm -rf "$dir" 
Block count: 262144 Reserved block count: 13107 Block size: 4096 Reserved blocks uid: 0 (user root) Setting reserved blocks percentage to 1% (2621 blocks) Reserved block count: 2621

262,144 blocks of 4,096 bytes is the 1 GiB image. 13,107 reserved blocks is 5% of that, rounded down, and the reserve belongs to uid 0. After tune2fs -m 1 it drops to 2,621 blocks, 1%.

That is the knob if you own the disk and the writer is not root. A clip volume that only holds recordings does not need a reserve for system daemons to keep running. On the disk your system boots from, I would leave it alone; that is the case the reserve was made for.

df says 83% and the subtraction says 78%

The second place this shows up is a percentage threshold. A cleanup job that deletes the oldest clips once the disk passes some percent full has to get its percentage from somewhere, and there are two obvious places:

#!/usr/bin/env bash # Print df's own size/used/avail/pcent for a path, then redo the percentage # two ways from those same numbers. set -eu df -B1 --output=size,used,avail,pcent "$1" read -r size used avail pcent < <(df -B1 --output=size,used,avail,pcent "$1" | tail -n 1) python3 - "$size" "$used" "$avail" <<'PY' import math, sys size, used, avail = map(int, sys.argv[1:]) print("used / size = %.2f%%" % (100 * used / size)) print("used / (used + avail) = %.2f%% -> rounded up: %d%%" % (100 * used / (used + avail), math.ceil(100 * used / (used + avail)))) PY 
 1B-blocks Used Avail Use% 10464022528 8166391808 1743982592 83% used / size = 78.04% used / (used + avail) = 82.40% -> rounded up: 83%

The GNU coreutils manual describes the pcent field as "Percentage of used divided by size." Taken literally, that would be 78.04% on this mount. df printed 83%. The number it printed matches used divided by used plus available, rounded up, which leaves the reserve out of the bottom of the fraction. The rounding is documented elsewhere on the same page: "Non-integer quantities are rounded up to the next higher unit." I did not read the df source to confirm which formula it uses. What I can say is that the two calculations above, made from df's own numbers, produce 78.04% and 83%, and df agreed with the second.

The practical effect: a job that computes used / total itself and cleans up at 80% would not have fired here, while df, and the person looking at it, would have said the disk was already past that line. At 100% in df's terms an ordinary writer is out of room, while used / total still reads somewhere near 95% on a disk with the default reserve.

Who is asking matters

The man page defines f_bavail for unprivileged users in general, not for the process asking, so I read it as the same number whoever calls; I only ran as an ordinary user and did not compare a root reading. What does change with the caller is what they are allowed to write. The reserve belongs to the super-user, so a check that runs as root and a writer that runs as an ordinary user can disagree in a way neither one notices alone:

  • A cron job running as root that measures with total - used and then writes as root is consistent: it can use the reserve, and its measurement includes it.
  • The same measurement used to decide whether to start a recorder that runs as an ordinary account is not. The recorder hits the wall while the check still sees room.
  • A check that uses free (or df's Avail) is the conservative one in both cases. For a root writer it leaves the reserve unused; for anyone else it is exact.

A short checklist for a free-space guard

  • Use shutil.disk_usage(path).free, os.statvfs(path).f_bavail * f_frsize, or df's Avail column. Do not compute total - used.
  • If the guard and the writer run as different users, measure for the writer, not for the guard.
  • For a percent-full threshold, decide which denominator you mean, and if a human will compare it with df, use used / (used + free) so the two numbers agree.
  • On a disk you set up only for recordings, look at the reserve with tune2fs -l or dumpe2fs -h and decide whether you want it. On the boot disk, leave it.
  • Leave headroom. A check that runs before the batch can only see the disk as it was then.

Things I did not test

  • Writing past the boundary. I could not mount the image as an ordinary user, so I did not watch a write fail at f_bavail reaching zero while f_bfree was still above it. That part rests on the man page's description of the reserve, not on a run of mine.
  • Other file systems. Everything here is ext4 or a mount whose type I did not check. XFS, Btrfs, FAT on an SD card and network mounts report these fields their own way, and the statvfs(3) page warns that "It is unspecified whether all members of the returned struct have meaningful values on all filesystems."
  • Other Python versions and Windows. I read the 3.10 source. On Windows shutil.disk_usage takes a different code path that I did not run.
  • Quotas. A user or project quota can stop a writer well before either count reaches zero. Nothing above measures that.

Sources. Field descriptions are from the Linux statvfs(3) man page. The reserve's purpose and default are from the mke2fs(8) man page for creating ext4 file systems, the -m option. The pcent description and the rounding sentence are from the GNU coreutils manual page df invocation. The disk_usage lines are from CPython 3.10's Lib/shutil.py as installed on the machine above.

If you want an old Android phone recording in the background so there is a folder of clips worth guarding disk space for, the app I make is Background Camera RemoteStream on Google Play. Nothing above depends on it.

When a check asks how much room is left, the answer depends on who will do the writing, and for anyone but root the honest number is the one shutil.disk_usage already labels free.