Every time I look at the server that carries this blog, I read the same two
numbers, and for a long time I assumed I knew what both of them meant. Last
night I saw this: since boot, 186,852,284 flushes completed, and a total of
115,846,756 milliseconds spent on them. The machine has been up for 15 days.
So it has shouted "put my data on stable storage" at the disk 186 million times.

Then I realised something: a single echo is enough to drive that number to
zero. An echo the kernel does not try to stop, prints no warning about, and
after which fsync() calls keep returning successfully.

I did not try this on my production server — I will explain why at the end.
I tried it in an isolated virtual machine on my Mac, and I counted every step.
The resulting table is a reminder of how thin the agreement is that we call
durability on Linux.

Two numbers: where the flush counter lives

/proc/diskstats and /sys/block/<device>/stat carry the same 17 fields. The
last three are relatively new and most monitoring dashboards never read them:

Field Meaning
15 Milliseconds spent discarding
16 Flush requests completed
17 Milliseconds spent flushing

A small warning: if you try to confirm the field count from the ABI
documentation you will get confused, because the sysfs-block document still
says the stat file contains 11 fields — it was written before the discard and
flush fields were added and never updated. The 17 comes from the iostats
document; count from that one.

While defining field 16, the kernel documentation sets up two traps in two
sentences: the block layer combines flush requests and executes at most one at a
time, and this counter is not tracked for partitions. I paid for the second
sentence while preparing this article; I will get to it shortly.

Here is my own server's identity card: Linux 6.8.0-147-generic, a KVM guest,
disk /dev/sda (QEMU HARDDISK), with /dev/sda1 mounted as ext4 using
rw,relatime,discard,errors=remount-ro,commit=30. The write_cache file says
write back, the fua file says 0, and rotational says 0. In other
words: the device declares that it has a volatile write cache, it does not
support the FUA flag, and it is not a spinning disk.

I can divide out the cost of an average flush: 115,846,756 / 186,852,284 ≈
0.620 ms. Turning that into a fraction of uptime, however, required me to
back out of a mistake first; that part is at the end.

The lab: counting every flush

I ran the measurement inside Docker Desktop's virtual machine; a --privileged
container can write to sysfs and there is no customer data there to damage.
Identity card: Linux 6.10.14-linuxkit, aarch64, disk /dev/vda (virtio), ext4
on /dev/vda1, and the measurement directory is a named Docker volume — so
overlayfs is not in the path and writes land directly on ext4. This disk makes
the same declaration as my server's: write_cache is write back and fua is
0. With one difference: under /sys/block/vda/mq there is a single
hardware queue. That detail becomes decisive at the end.

The measurement tool is a twenty-line C program: it opens a file, does a 4 KiB
pwrite followed by an fsync on every iteration, and exits immediately if
fsync returns anything but zero. I leave the counting to the kernel rather
than the program; I only read fields 16 and 17 before and after the run.

The noise floor came first. At idle, 6, 5 and 3 flushes pass in five seconds —
roughly one per second. My actual runs take half a second, so that noise stays
in single digits, but I did not want to leave it unknown.

The first series, with write_cache in its default state (write back), single
threaded, 1,000 fsyncs, nine runs — each entry is elapsed(ms) · flushes ·
flush_ms:

478.7 2000 125 492.4 2000 134 484.0 2000 144 453.0 2000 129 462.4 2000 133 467.3 2000 138 485.6 2000 135 457.7 2002 134 468.8 2000 123 median: 468.8 ms | 2000 flushes | 134 ms

A thousand fsyncs, two thousand flushes. Exactly 2000 in eight of the nine runs
and 2002 in one — the excess being that once-per-second background noise. The
ratio is not a coincidence; it is written in the source.

Why exactly two

The comment at the top of block/blk-flush.c explains that a flush is
decomposed into a three-step sequence: PREFLUSH, DATA, POSTFLUSH. In today's
mainline, the place that decides which steps actually run is
blk_insert_flush(), and the decision hangs on a single condition:

if (blk_queue_write_cache(q)) { if (rq->cmd_flags & REQ_PREFLUSH) policy |= REQ_FSEQ_PREFLUSH; if ((rq->cmd_flags & REQ_FUA) && !supports_fua) policy |= REQ_FSEQ_POSTFLUSH; }

On the 6.10 the lab runs on, the same decision lives in a separate function,
blk_flush_policy(), and reads queue flags; the meaning is identical, but
someone grepping 6.10's blk_insert_flush for this will not find it.

The same file's header comment states the rule plainly: if the device has a
writeback cache and does not support FUA, REQ_PREFLUSH is translated to a
PREFLUSH and REQ_FUA to a POSTFLUSH.

On my disk the fua file says 0. ext4's jbd2 journal writes the commit block
with REQ_PREFLUSH | REQ_FUA when a transaction commits — not once per fsync
call, but whenever there is something to commit. In my loop every iteration
writes new data, so every fsync forces a commit and the two coincide. Because
FUA is unsupported, the second flag turns into a POSTFLUSH. One commit, two
flushes. A thousand calls, two thousand flushes.

There is also a reason the number is exactly two and not three: jbd2 has two
other places where it issues a flush, but one depends on having a separate
journal device and the other on the async_commit feature — both are off in my
setup. Measurement and source produce the same number; that happens rarely, and
when it does you briefly feel like you actually understand the mechanism.

Diagram

The critical spot in the diagram is the false branch: a request that lands
there never reaches the disk, yet fsync still returns zero. I went down that
branch deliberately.

One line

The write_cache file is readable and writable. The kernel's own ABI
documentation says what writing does without leaving any room for escape:
writing to this file can change the kernel's view of the device but does not
alter the device state, which is why switching from write back to
write through may not be safe, since it also eliminates the cache flushes the
kernel issues.

The documentation says "may not be". I wanted to count how much. Same disk,
same program, the only difference being
echo "write through" > /sys/block/vda/queue/write_cache. Nine runs again,
this time elapsed(ms) · flushes:

270.7 0 259.9 0 265.3 0 263.2 0 275.1 0 342.8 0 260.0 0 262.4 0 287.8 0 median: 265.3 ms | 0 flushes | 0 ms

Zero. Zero in all nine runs. None of the thousand fsync calls returned an
error — the program would have exited if one had, and it did not. For anyone
looking from the application side, nothing changed: fsync was called, it
returned successfully, the data is "durable". The disk knows nothing about
this conversation.

The mechanism is one sentence in the blk-flush.c header comment: if the
device has no writeback cache, PREFLUSH and FUA make no difference, and requests
carrying no data are completed immediately. The flush request is not going
anywhere; it is resolved in software. The kernel is not lying — I lied to it,
and it believed me.

How much of the saved time can I actually account for

The medians: 468.8 ms → 265.3 ms. The same work finished 1.77 times faster.
So far so good. The real question is this: how much of that speed-up does the
flush time the kernel reported actually explain?

The saving is 203.5 ms. The flush time that disappeared is 134 ms. So 66
percent of the speed-up can be read directly out of field 17; the remaining
70 milliseconds I cannot account for. I have a plausible guess — queueing,
dispatching and completing a flush as a request has a cost of its own, and
field 17 does not capture all of it — but that is a hypothesis, not a
measurement. I do not want to round an unmeasured thing up into "I removed the
flush, that is where the speed came from".

On this disk, which has a single hardware queue, I can compare field 17 against
wall-clock time (the header comment's C1 condition: only one flush proceeds at a
time), so I can compute the share of each write back run spent executing
flushes: a median of 28.5 percent, ranging from 26.1 to 29.8 across the nine
runs. A small honesty note: field 17 also includes time the request spends
queued, so that share is slightly wider than "time the disk was busy flushing".

Had I seen that number in a database benchmark, I would have been excited. Here,
though, the gain is pure accounting trickery: what got faster is not the work
but the work's guarantee. On a power loss, whatever sits in the device's
volatile cache is gone, and because fsync told you "written", your application
has no grounds for complaint. ext4's journal does not save you either: the
journal's consistency guarantee rests on those flushes actually reaching the
disk. In the run where I enabled fast_commit and watched it never fire across
4,000 fsyncs
I learned to distrust the journal's counters; here the counter is right and the
guarantee is absent.

The strings sysfs accepts

The queue_wc_store() function uses strncmp to check three strings:
write back (10 characters), write through (13 characters) and none (4
characters). The third never appears in the ABI documentation — the docs only
describe the first two. Curious, I tried all of them:

[write back] ACCEPTED -> write back [write through] ACCEPTED -> write through [none] ACCEPTED -> write through [write backwards] ACCEPTED -> write back [nonetheless] ACCEPTED -> write through [writeback] REJECTED (EINVAL) [hayir] REJECTED (EINVAL)

write backwards and nonetheless were accepted, because strncmp only looks
at the first n characters — the comparison is a prefix match. Drop the space and
write writeback and you get EINVAL. This is not a vulnerability, but for
anyone writing configuration management it carries a lesson: the string you
write to this file being accepted does not mean the thing you thought you were
writing was accepted. Read it back before moving on.

Which kernel, which behaviour

Memory from training is useless here, because this code moved house within two
years. In the 6.10 the lab runs on, the store function keeps the cache view in a
queue flag (QUEUE_FLAG_WC) and requires the device to have declared a hardware
cache before you can write write back. I tested that: writing write back to
the ram0 device, which has no cache at all, returned EINVAL.

In today's mainline, the setting has moved into the queue_limits structure and
its logic has been inverted: there is now a disable flag called
BLK_FLAG_WRITE_CACHE_DISABLED, and the comment in the header file states its
purpose outright — do not send FLUSH/FUA commands despite a write cache having
been advertised. blk_queue_write_cache() is now computed as "the feature is
present and the disable flag is not set", and the hardware requirement in the
store function is gone.

The practical consequence: a write that was rejected with an error on 6.10
succeeds silently on today's kernel and changes nothing — because clearing a
disable flag does not conjure a cache feature that was never there. I did
not measure this second half; I derived it by reading the mainline source.
I cannot exercise mainline behaviour in my 6.10 lab. I am leaving the
distinction visible here, because "the source says so" and "I saw this on screen"
are not the same thing.

Moving the flag into queue_limits has one more side effect. Drivers only touch
their own fields when they update limits — the SCSI disk driver, for instance,
writes only the cache and FUA feature bits — and because an update starts from
the current limits, a disable flag set by hand stays where it is. So do not rely
on the behaviour known from older kernels, where "revalidating the device undoes
the setting". A reboot is a different matter: the flag lives in memory and
nothing survives the next boot.

Where I went wrong reading the counters

The partition counter is always zero. My filesystem sits on /dev/vda1, so
that is where I would want to watch. But when I read field 16 from the
partition, the disk sits at 1,369,613 flushes while the partition shows 0.
That is exactly the cost of the documentation's "not tracked for partitions"
sentence: if you built your dashboard on the partition, your flush count stays
zero forever, and you read that as "no flushes".

You cannot derive the flush count from the fsync count. I ran the same 1,000
fsyncs across 8 threads at 125 each instead of on one thread, with write back
in place, nine times again:

158.4 504 160.0 502 155.6 500 151.6 502 145.8 500 154.2 500 145.8 500 150.6 500 157.0 504 median: 154.2 ms | 500 flushes

Five hundred, not two thousand. Four times fewer, because the block layer
double-buffers pending flushes and executes them together: only one flush
proceeds at a time, and the requests that pile up while it runs share the next
one. As concurrency rises, the flush cost per fsync falls — so "the flush count
is high" says nothing on its own and cannot be interpreted without knowing the
workload's concurrency.

Be careful dividing total flush time by wall-clock time. That is precisely
the first thing I did while writing this article: I was about to divide my
server's 115,846,756 ms by its 1,296,505 seconds of uptime and write "the
machine spent 8.9 percent of its time waiting on flushes". Then I remembered
that the flush queue is allocated per hardware queue (hctx->fq) and looked
at my server's queue count: there are 18 queues under /sys/block/sda/mq.
The "only one flush at a time" guarantee holds per queue, not per disk — so 18
flushes can be in progress at once, and field 17 sums their durations on top of
one another.

That makes 8.9 percent not a measurement but an upper bound: the true
wall-clock share is below it, and with the queues fully busy it could fall to
roughly a twentieth of that. There is a more honest way to read the same number:
0.089 is the average number of flushes in flight. Because the lab's vda has a
single queue, I can quote the 28.5 percent there as a share; for my server I
cannot.

That leaves two numbers I can defend on my server: an average of 0.620 ms per
flush, and 186.8 million flushes in 15 days. Counting across a ten-second
window, I saw 1,038 flushes and 1,384 ms — a momentary average of 1.33 ms, more
than double the lifetime average. With a load average of 19.57 that is not
surprising, but it shows how fragile the sentence "the average flush is 0.6 ms"
really is. Something similar happened in
the article where I cut the request queue from 31 to 1 and measured the IOPS
ceiling not moving:
the same sysfs directory, the same kind of wrong summary.

Asking the device a second time

Up to this point I had built the article on "you cannot tell the kernel's view
apart from the device's reality". That was wrong. For SCSI disks there is a
second opinion: /sys/class/scsi_disk/<h:c:t:l>/cache_type. Unlike
queue/write_cache, this file does not show the kernel's intent but the actual
bits read from the device's mode page; and when you write to it, the driver
sends a real MODE SELECT command to the device, that is, it genuinely changes
the disk. It can take four values, and the fourth is named
write back, no read (daft) in the source — kernel developers' humour leaks
into the documentation too.

On my server I read the two side by side:

/sys/block/sda/queue/write_cache = write back /sys/class/scsi_disk/0:0:0:0/cache_type = write back

The two say the same thing, so on my disk the kernel's declaration is backed by
the device. Had they diverged, I would have known one of them had been changed
by hand. I cannot ask the same question of the lab's vda: a virtio disk is not
in the scsi_disk class, and there is no cache_type file there. So whether a
second opinion exists depends on the device type — present on SCSI/SAS/SATA
disks, absent on virtio, and on NVMe the same information comes from the
controller's own feature query.

So what should you do

I did not touch the write_cache file on my server, and I will not. This is not
a performance setting; it is changing the kernel's declaration about a device by
hand. My list of reasonable uses goes like this:

  • Look at it to read it. write_cache, fua and fields 16/17 together tell you a disk's durability cost without installing any tooling. Put them on the dashboard — but read them from the disk, not the partition.
  • If you are suspicious, ask the device. On a SCSI disk, cache_type, sdparm or hdparm -W give you the device's own declaration; on NVMe the same information sits in the controller features. If the two sources diverge, somebody wrote to queue/write_cache by hand.
  • In stacked setups, do not stop at the bottom disk. LVM, md, dm-crypt and loop devices each have their own write_cache file, and suppressing flushes on the disk underneath lowers the guarantee of every layer above it at once.
  • Remember the privileged container. The only reason I could run this measurement is that a --privileged container can write to the host's sysfs. On a shared disk, such a container can switch off durability for the whole machine without asking anyone. Keep that in mind when granting containers sysfs write access.
  • Consider writing write through only if you know the device truly has no volatile cache — a battery- or capacitor-backed RAID controller, say. In that case what you are doing is not lying but correcting the kernel's wrong assumption. Make the decision with a written rationale.
  • Size the loss. The risk is as much data as the device's volatile cache, which on typical enterprise disks is tens to hundreds of MiB. PostgreSQL, MySQL/InnoDB, etcd and SQLite — anything that pins durability on fsync — will quietly give the wrong answer inside that window.
  • Do not make it persistent, but do not trust its impermanence either. The setting disappears on reboot, which sounds like good news; buried in a udev rule, though, it travels with you when the hardware changes and nobody notices it is no longer true. Alert on the value changing.

Conclusion

What I took from this experiment is not the flush counts but a small shift in
how I look at sysfs. For years I read the files under /sys/block/*/queue/ as
"the place where the device describes itself to me". But some of them are not
measurements, they are declarations — and declarations that run in both
directions. The kernel writes what the device said there, but I can also write
there what I want the kernel to think about the device. The file reads
identically in both cases.

The consolation: you cannot tell them apart by looking at one file, but the way
to tell them apart is not closed. Ask a second source — on a SCSI disk, the
device's own mode page — and it becomes clear whether there is a device behind
the declaration or just an echo. Trusting a single source is the common cause
of every mistake I counted in this article: reading from the partition, looking
at a single run, dividing field 17 without knowing the queue count.

fsync's contract is therefore more indirect than it looks. It does not say
"the data is on stable storage"; it says "based on the information I was given,
what needed to be done to make the data durable was done". Whether that
information is true is not fsync's problem. The only person who will worry
about it is the sysadmin who accepts 186 million flushes in 15 days and agrees
to wait on the disk.

Official Sources