When write() returns zero errors, almost nothing has happened. The bytes are in memory. The disk has not been told. If the machine loses power in the next thirty seconds, they are gone — and the program that wrote them was told everything went fine.
That gap between “the call succeeded” and “the data exists” is where every storage problem lives. Between them sit seven layers, each of which caches, reorders, batches, splits, encrypts or duplicates your write, and each of which has a different opinion about when it is finished. This page follows one write all the way down, then follows a read back up, with the command that inspects each layer as you go.
Filesystems covers inodes, journalling and choosing between ext4, XFS and Btrfs. This is the level below it: not which filesystem, but what the whole stack does with a single write.
The layers, once, so the rest of the page makes sense. One buffered write passes down through all of these:
- The page cache — your bytes are copied into memory and marked dirty.
write()returns here. - Writeback — later, on a timer or under pressure, the kernel decides to write them out.
- The filesystem — allocates blocks, updates metadata, and writes a journal entry or a new copy.
- Device mapper — LVM remaps it, dm-crypt encrypts it, md computes parity. Any or none of these.
- The block layer — merges and reorders requests, then queues them for the driver.
- The device’s own cache — the drive accepts the write into volatile RAM and reports success.
- The medium — and only a cache flush, or a write flagged FUA, gets you here.
The single most useful diagnostic on this page is two lines of /proc/meminfo. Dirty is data waiting to be written; Writeback is data being written right now. If Dirty is large and growing, your application is producing faster than the stack below can absorb, and the problem is somewhere in layers three to seven. If Dirty stays near zero while the application is still slow, it is doing synchronous I/O and the problem is in layer one — in how it is calling, not in your disks.
1. The page cache, where write() actually ends
A buffered write() copies your bytes into the page cache, marks those pages dirty, updates the file size, and returns. No device has been contacted. This is why writing a gigabyte to a file can appear to take a fraction of a second on a machine whose disk could not possibly have absorbed a gigabyte in that time — you have measured a memcpy.
The same cache serves reads. A read that finds its page already present is a memory copy; one that does not becomes a device request and blocks. This is the single largest performance difference in the whole stack, and it is why benchmarking anything twice gives two different answers.
grep -E '^(Dirty|Writeback|Cached|MemFree):' /proc/meminfo
Dirty: 184320 kB
Writeback: 4096 kB
Cached: 12841236 kB
# watch it move while something is writing
watch -n1 "grep -E '^(Dirty|Writeback):' /proc/meminfo"Two things about the cache have changed recently and are worth knowing because the older mental model is now wrong. The page cache no longer works purely in 4 KB pages: large folios let it manage bigger contiguous runs, which cuts per-page overhead substantially — ext4 enabled them for regular files in Linux 6.16, with roughly a third more throughput reported on sequential I/O, and Btrfs made them the default in 7.2. These are not transparent huge pages and there is no sysfs knob for them; the filesystem opts in and you get the benefit or you do not.
And since Linux 6.14 there is RWF_DONTCACHE: buffered I/O whose pages are dropped from the cache as soon as they are written out. It is the honest answer to “I am streaming a hundred gigabytes through this machine once and I do not want it to evict everything else”, which people previously attempted with posix_fadvise and mostly got wrong.
2. Writeback, and the tunables that ruin large machines
Dirty pages are written out by kernel flusher threads, triggered by two thresholds and two timers. All four are sysctls, and their defaults were chosen for machines much smaller than the one you are probably running.
| Knob | Default | What it does |
|---|---|---|
vm.dirty_background_ratio | 10 | At this percentage of available memory, flusher threads start writing in the background. Nothing blocks. |
vm.dirty_ratio | 20 | At this percentage, the writing process is made to do the writeback itself. This is where a stall comes from. |
vm.dirty_expire_centisecs | 3000 | Dirty data older than 30 seconds becomes eligible for writeout regardless of size. |
vm.dirty_writeback_centisecs | 500 | Flusher threads wake up every 5 seconds to see whether there is work. |
Read the second row again with a real machine in mind. On a server with 256 GB of RAM, dirty_ratio = 20 permits roughly 51 GB of dirty data before anything is forced to block — and when the threshold is finally crossed, the process that crossed it is made to push that backlog to a device that may manage a few hundred megabytes a second. The result is an application that runs perfectly for minutes and then freezes for a very long time, with no error anywhere.
The fix is to stop expressing the limit as a fraction of RAM and start expressing it in bytes, sized to what the storage can actually absorb in a second or two:
# see what you have
sysctl vm.dirty_ratio vm.dirty_background_ratio vm.dirty_bytes vm.dirty_background_bytes
# absolute limits instead, for a device doing roughly 500 MB/s
sysctl -w vm.dirty_background_bytes=$((256*1024*1024))
sysctl -w vm.dirty_bytes=$((1024*1024*1024))The two forms are mutually exclusive, and the kernel enforces it rather than merging them: setting either _bytes value makes the corresponding _ratio read back as zero, and vice versa. Do not go looking for the interaction between them; there is not one. (The minimum accepted value for dirty_bytes is two pages — anything lower is silently ignored and the previous setting kept.)
3. What each durability call actually promises
If you want data on the disk before you continue, you have to ask. The calls that do the asking differ in ways that matter.
| Call | What it guarantees |
|---|---|
fsync(fd) | Data and metadata for this file are on stable storage, including flushing the device’s own write cache. |
fdatasync(fd) | The same, minus metadata that is not needed to read the data back. A size change is flushed; an mtime change is not. Faster, and correct for append-style workloads. |
syncfs(fd) / sync() | On Linux, waits for completion — equivalent to fsync on every file in that filesystem, or in the system. |
sync_file_range() | Almost nothing. Its own man page calls it “extremely dangerous”. It writes no metadata, gives no crash guarantee except for pure overwrites of already-allocated blocks, does not work on copy-on-write filesystems, and provides no integrity at all on a device with a volatile cache. |
Fsyncing the file is not enough. The manual page says it outright: fsync() “does not necessarily ensure that the entry in the directory containing the file has also reached disk”. A brand-new file can be fully durable and still not exist after a crash, because the directory entry pointing at it was never written.
The correct sequence for replacing a file atomically — the one every configuration writer, editor and package manager should be using — is four steps, and the last one is the one everybody skips:
1. write the new content to a temporary file in the same directory
2. fsync(tmp_fd) # the data is durable
3. rename(tmp, target) # atomic: readers see old or new, never partial
4. fsync(dir_fd) # the rename itself is durableSkip step two and you can end up with a file that exists and is empty. Skip step four and you can end up with the old file, or on some filesystems no file at all. “Same directory” in step one is not optional either — rename() is only atomic within a filesystem, and a temporary file in /tmp may be on a different one.
4. The filesystem, and what it does with your block
When writeback finally happens, the filesystem decides where the data goes and what else must be updated to make it findable. Broadly there are two strategies, and the difference explains most of their behaviour under crash and under load.
Journalling filesystems — ext4, XFS — write a description of the metadata change to a journal, flush it, then apply the change in place. After a crash the journal is replayed, so metadata is consistent even though a partially written file may contain rubbish. Copy-on-write filesystems — Btrfs, ZFS — never overwrite live data at all: they write new blocks elsewhere and atomically swing a pointer, which gives cheap snapshots and checksummed self-verification at the cost of fragmentation on random-write workloads such as databases and virtual machine images. What a journal does during a crash covers the guarantee in detail.
What you are actually running, on a default install in 2026:
| Distribution | Default root filesystem |
|---|---|
| Fedora Workstation | Btrfs, since Fedora 33 |
| openSUSE Leap and Tumbleweed | Btrfs |
| RHEL 10 and rebuilds | XFS. Btrfs is not shipped at all; the layered answer is Stratis, which is XFS on LVM |
| Debian 13 | ext4 — the installer guide says so explicitly |
| Ubuntu 26.04 LTS | ext4; nothing in the release notes suggests a change |
And one that is no longer a candidate, because this is the storage story of the last two years. bcachefs was removed from the kernel. It was merged in 6.7, marked “externally maintained” in 6.17 after a sustained disagreement with upstream, and deleted outright in 6.18 in November 2025 — Torvalds’ commit message noting that since it had become a DKMS module, leaving stale code in the tree would only cause version confusion. It still exists and is still developed; Arch packages bcachefs-dkms, and the project maintains its own repositories. But it is an out-of-tree module now, openSUSE disabled it in its kernel builds, and no mainstream installer offers it. If you read an enthusiastic article about bcachefs, check its date.
# what is where, with filesystems and UUIDs
lsblk -f
# the whole stack for one device, including physical/logical block sizes
lsblk -t
# what a filesystem thinks of itself
xfs_info /
tune2fs -l /dev/nvme0n1p2 | head -30
btrfs filesystem usage /5. Device mapper: the layers you forgot were there
Between the filesystem and anything resembling a disk there are usually one to three more block devices, each pretending to be the real thing. lsblk draws the tree; the point of reading it is that every layer in it is somewhere a problem can live.
lsblk
nvme0n1 259:0 0 931.5G 0 disk
├─nvme0n1p1 259:1 0 1G 0 part /boot/efi
└─nvme0n1p2 259:2 0 930.5G 0 part
└─luks-9f3c... 253:0 0 930.5G 0 crypt
├─vg0-root 253:1 0 60G 0 lvm /
└─vg0-data 253:2 0 800G 0 lvm /srv
dmsetup ls --tree
cat /proc/mdstatLVM is address translation and little else; its cost is close to zero, and what it buys is the ability to resize and snapshot. dm-crypt is where real CPU cost enters the stack. On a modern LUKS2 volume the default is AES-XTS with a 512-bit key, and the passphrase is stretched with Argon2id — which is memory-hard by design, so unlocking is deliberately slow and cannot be meaningfully accelerated by an attacker’s GPU. LUKS2 has been the default since cryptsetup 2.1.0 and Argon2id since 2.4.0. LUKS1 is not formally deprecated, but upstream’s own guidance is blunt: unless you have a specific reason, use LUKS2.
dm-crypt’s default arrangement queues work through kernel threads, which on very fast NVMe becomes the bottleneck rather than the cipher. There are flags for this, and they are worth knowing about because the symptom — an NVMe drive performing like a SATA one — looks like a hardware fault:
# what the mapping is actually doing
cryptsetup status luks-9f3c...
cryptsetup luksDump /dev/nvme0n1p2
# performance flags, persisted into the LUKS2 header
cryptsetup --help | grep -A2 perfThe relevant target options are no_read_workqueue and no_write_workqueue, which process requests synchronously instead of handing them to a queue, plus a newer high_priority flag that improves dm-crypt’s throughput and latency at the cost of general system responsiveness. Check the spelling against cryptsetup --help on your version rather than trusting any article, this one included — the flags moved between manual pages.
Two newer things are worth naming. cryptsetup 2.8 added --integrity-inline, which uses the extra per-sector metadata space on enterprise NVMe to store authentication tags, removing the dm-integrity journal that made authenticated encryption painfully slow. And Linux 7.2 merged dm-inlinecrypt, which hands encryption to hardware engines in the storage controller instead of doing it on the CPU at all.
6. RAID, and the hole that is not where people think
If md is in the stack, a single write to a RAID 5 or 6 array is not a single write. Changing part of a stripe means reading the old data and old parity, computing the new parity, and writing both — the read-modify-write penalty that makes parity RAID poor at small random writes and fine at large sequential ones.
It also creates the write hole. The manual page for md states it plainly: because those writes are not atomic, an interruption partway through can leave the parity inconsistent with the data. The array does not know, and a later rebuild will faithfully reconstruct corruption from the wrong parity. Two mechanisms close it, and neither is on by default:
- A write-ahead journal on a separate fast device, supported since Linux 4.4, for RAID 4, 5 and 6.
- Partial Parity Log, since Linux 4.12, stored in spare space on the member drives so no extra device is needed — but RAID 5 only. It does nothing for RAID 6.
cat /proc/mdstat
mdadm --detail /dev/md0 | grep -i consistency
mdadm --grow /dev/md0 --consistency-policy=ppl
# verify parity without correcting it, then read the result
echo check > /sys/block/md0/md/sync_action
cat /sys/block/md0/md/mismatch_cntThat periodic verification is not something the kernel does on its own — it is a distribution timer or cron job, so on a hand-built array it may simply never happen. A non-zero mismatch_cnt on a parity array is worth investigating; on RAID 1 it is often benign, because of writes in flight to memory-mapped pages during the check.
And the thing people most often get wrong: Btrfs’s own RAID 5 and 6 are still marked unstable by the Btrfs project’s own status page, not by critics. That has been true for years and remains true against current kernels. RAID 1, RAID 10 and the three- and four-copy RAID1C3/RAID1C4 profiles are marked fine. If you want parity RAID under Btrfs, put Btrfs on top of md.
7. The block layer, and where the queue actually is
Below device mapper, requests enter the block layer as bio structures, get merged with adjacent requests, and are placed on per-CPU submission queues. The old single-queue design and its schedulers — noop, deadline, cfq — were deleted in Linux 5.0; anything written about them is at least seven years stale. What exists now is none, mq-deadline, bfq and kyber.
For NVMe the default is none, and that is correct: the device has more internal parallelism than any reordering the kernel could usefully do, so a scheduler is pure overhead. Red Hat’s documentation goes as far as recommending you do not change it. On rotational media, where seek order genuinely matters, mq-deadline is the usual choice, and bfq is worth trying on a desktop where interactive responsiveness under load matters more than raw throughput.
cat /sys/block/nvme0n1/queue/scheduler # active one is in [brackets]
cat /sys/block/nvme0n1/queue/rotational # 0 = solid state
cat /sys/block/nvme0n1/queue/nr_requests
cat /sys/block/nvme0n1/queue/read_ahead_kb
cat /sys/block/nvme0n1/queue/write_cache # "write back" or "write through"That last one is the most interesting and the least known. It reports whether the device has a volatile write cache the kernel must flush — and it is writable, transport-agnostic, and rather better than remembering whether this device wants hdparm -W, sdparm, or an NVMe feature identifier.
8. The device, and the two flags that reach it
Almost every drive sold has a volatile write cache. It accepts your write into RAM, reports completion, and writes it to the medium whenever it likes. Everything above — the journal, the parity, the careful ordering — is worthless if the drive is free to reorder the final writes.
Two request flags exist for this, and they are the bottom of the entire durability chain:
REQ_PREFLUSH— the device’s cache must be flushed before this request starts. This is what enforces “the journal is on the platter before the data that depends on it”.REQ_FUA— force unit access. Completion is not reported until this data is on non-volatile storage, without flushing everything else.
The old name for this was write barriers, and here the folk knowledge is half wrong in an interesting way. XFS removed barrier and nobarrier as mount options in Linux 4.19 — passing them now does nothing or errors. ext4 still documents them, and nobarrier is still an ext4 mount option you can set. It is also still a way to lose data on a power cut in exchange for a throughput number, and the only defensible use is a device with a genuinely non-volatile cache, meaning battery- or capacitor-backed.
Worth an honest word on speed, since everyone wants a table of latencies and most published ones are invented. One figure is not: a 7200 rpm drive turns once every 8.33 ms, so the average rotational latency alone — before any seek, before any queueing — is 4.17 ms. Everything above it in this article is an elaborate scheme to avoid paying that number, which is why a page-cache hit and a cold random read on spinning rust differ by something like five orders of magnitude. For actual figures on actual hardware, read the datasheet for the drive you have; the round numbers circulating online are mostly folklore.
Do not use the discard mount option. Continuous TRIM issues a discard to the device on every deletion, and the advice against it has not changed: it causes unpredictable latency spikes, and the XFS documentation specifically recommends against it on production workloads.
The right mechanism is the periodic one, batching every discard into a single maintenance window:
systemctl status fstrim.timer
systemctl enable --now fstrim.timer
fstrim -avFedora has enabled that timer by default since Fedora 32, weekly. RHEL 10 and its rebuilds enable it; RHEL 8 and 9 do not, which catches people out on long-lived servers.
Btrfs is the exception, and you do not have to do anything about it. Since Linux 6.2, Btrfs automatically enables discard=async on devices that support it — a rate-limited background discard that has neither the latency problem of synchronous TRIM nor the delay of a weekly timer. Synchronous discard on Btrfs is still discouraged.
9. The way back up
A read runs the same layers in reverse, with one addition. The kernel watches access patterns and, when a file is being read sequentially, issues requests for blocks you have not asked for yet. read_ahead_kb in the queue sysfs is the window; the default suits general use, and raising it helps large sequential streaming and hurts random access by wasting bandwidth on data nobody wants.
The important consequence for anyone measuring anything: the second run of a benchmark is measuring your RAM. If you want to know what the storage does, empty the cache first — on a machine where a stall does not matter.
sync && echo 3 > /proc/sys/vm/drop_cachesThat is a diagnostic, not a tuning step. Dropping caches on a busy production machine forces everything to be re-read from disk and will make the next few minutes considerably worse. It has never been a fix for anything.
10. A worked diagnosis
A reporting database on a 256 GB server runs an overnight import. Most of the time it is fast. Roughly once an hour it stops completely for forty to ninety seconds — no errors, no crash, no slow query in the log, and monitoring shows CPU near idle throughout. It has been blamed on the network, on the application, and on the SAN, in that order.
Start above the storage, not at it. The first question is whether the stall is data waiting to be written, or the device being slow.
watch -n1 "grep -E '^(Dirty|Writeback):' /proc/meminfo"
Dirty: 48213456 kB
Writeback: 524288 kBForty-eight gigabytes of dirty pages. That is the whole answer in one line, and it did not require touching the disks. dirty_ratio is at its default of 20, the machine has 256 GB of RAM, so the kernel is willing to accumulate roughly 51 GB of unwritten data before it forces anything to block — and the import climbs steadily towards that ceiling all hour. When it arrives, the writing process is conscripted into flushing the backlog itself, and the application freezes until a device that manages a few hundred megabytes a second has drained tens of gigabytes.
Confirm it, rather than assuming. Catch the stall in progress:
iostat -xz 1
Device r/s w/s rkB/s wkB/s aqu-sz %util
dm-1 0.0 982.0 0.0 491008.0 142.31 100.00The device is saturated, the queue is enormous, and it is all writes. This is not a slow disk misbehaving — it is a disk being handed an hour of accumulated work in one go. CPU is idle because everything is blocked in D state waiting for it.
The fix is to stop letting the backlog build. Absolute byte limits, sized so that the worst case is a second or two of writeback rather than an hour of it:
cat > /etc/sysctl.d/99-writeback.conf <<'EOF'
vm.dirty_background_bytes = 268435456
vm.dirty_bytes = 1073741824
EOF
sysctl --systemThe total work does not change. What changes is that it is spread evenly instead of arriving in a wall, and background flushing starts at 256 MB rather than at 25 GB. Throughput drops very slightly. The stalls disappear entirely.
Notice what the method bought. Had Dirty been small during the stall, this diagnosis would have been wrong from the first step and the next question would have been the device itself — iostat for saturation, then down through the lsblk tree asking which layer added the latency, with dm-crypt on a fast NVMe the usual culprit. Had Dirty been small and the device idle, it would not have been a storage problem at all: an application issuing an fsync per row does not stress the disk, it just refuses to let it work in parallel, and the fix is in the application.
11. Symptom, layer, command
The numbers below are the seven layers from the box at the top of this page, not the section numbers — the sections split some layers in two and skip between them, which the layers do not.
| What you see | Layer and likely cause | What to run |
|---|---|---|
| Writes are impossibly fast, then the machine freezes | 1–2 — dirty pages accumulating to dirty_ratio | grep Dirty /proc/meminfo; set vm.dirty_bytes |
| Data lost on a power cut despite the write succeeding | 1 — it never left the page cache; nothing asked for durability | Add fsync; check the rename sequence |
| Benchmark result changes on the second run | 1 — you are measuring the page cache | sync && echo 3 > /proc/sys/vm/drop_caches |
| A config file is empty after a crash | 3 — renamed without fsyncing the file first | fsync file, rename, fsync directory |
| A new file vanished after a crash | 3 — the directory entry was never fsynced | fsync the directory file descriptor too |
| A database on Btrfs slows down over weeks | 3 — copy-on-write fragmentation | chattr +C on the directory before creating files |
The disk is full but du disagrees with df | 3 — a deleted file still held open, or snapshots | lsof +L1; btrfs filesystem usage |
| Out of space with plenty of free bytes | 3 — out of inodes, or Btrfs metadata exhausted | df -i; btrfs filesystem df |
| An article recommends bcachefs | 3 — it left the kernel in 6.18 | Check the date; it is a DKMS module now |
| NVMe performing like a SATA drive | 4 — the dm-crypt workqueue, not the cipher | cryptsetup status; the no_*_workqueue flags |
| Unlocking an encrypted disk takes seconds | 4 — Argon2id is memory-hard on purpose | cryptsetup luksDump; this is working correctly |
| Small random writes are catastrophically slow | 4 — RAID 5/6 read-modify-write per stripe | cat /proc/mdstat; consider RAID 10 |
| Silent corruption found after a rebuild | 4 — the write hole; the parity was already wrong | --consistency-policy=ppl; scrub regularly |
| A parity array has never been verified | 4 — scrub is a distro timer, not a kernel job | echo check > /sys/block/md0/md/sync_action |
| One container starves the others for I/O | 5 — no I/O control anywhere | cgroup v2 io.latency or io.cost.qos |
Everything is blocked in D state and the CPU is idle | 4–6 — a saturated device somewhere below | iostat -xz 1; look at aqu-sz and %util |
| Latency spikes whenever files are deleted | 6 — the discard mount option | Remove it; systemctl enable --now fstrim.timer |
| An SSD gets slower over months | 6 — TRIM never running | systemctl status fstrim.timer; fstrim -av |
| A drive is throwing errors nobody noticed | 6 — nothing is watching SMART | smartctl -a /dev/sda; nvme smart-log /dev/nvme0 |
| Journal or parity survives a crash on one machine and not another | 7 — nobarrier, or a lying write cache | cat /sys/block/sda/queue/write_cache; check mount options |
| You need to know which process is doing the I/O | any — attribution | iotop, or biosnoop/biolatency from bcc |
12. What changed recently, and why it matters
Three developments are quietly rewriting assumptions that have held for two decades.
Atomic writes. Since Linux 6.13, pwritev2() with RWF_ATOMIC on an O_DIRECT file gives a torn-write-free write on XFS and ext4, with the supported sizes exposed through statx. Linux 6.16 extended it to large atomic writes on XFS and multi-block ones on ext4. This is a bigger deal than it sounds: databases have carried double-write buffers for decades purely to survive torn pages, and that entire mechanism becomes unnecessary when the kernel and device can promise a write happens or does not.
Block sizes larger than the page size. XFS gained support in 6.12. A 16 KB block filesystem on a machine with 4 KB pages is now a real thing, which matters for large-block flash and for reducing metadata overhead on very large filesystems.
Filesystems that fix themselves. Linux 7.0 wired XFS’s online repair to health events, so damage detected during normal operation can trigger a repair through a systemd-managed daemon rather than waiting for someone to notice and unmount. The same release added a generic way for filesystems to report I/O errors to userspace through fsnotify. Storage is slowly acquiring the thing it has always lacked: a way to tell you something is wrong before you go looking.
13. The shape of the whole thing
Every layer in this stack exists to lie to the one above it about when a write is finished, and every one of those lies is a good trade almost all of the time. The page cache lies so that applications are not bound to disk speed. The filesystem’s journal lies about ordering so that metadata survives. The drive’s cache lies so that it can reorder for its own convenience. The system works because there is exactly one mechanism — a flush, requested by fsync and enforced by REQ_PREFLUSH — that forces every layer to stop lying at once.
Which is why the diagnosis always starts in the same place. Look at Dirty first. A large and growing figure means the problem is below you and you should walk down the lsblk tree with iostat. A figure near zero on a slow application means the problem is above you, in how it is calling, and no amount of tuning will help. That one observation eliminates most of this page, most of the time.
Next, if this was useful: From Power-On to a Login Prompt follows the eight handoffs between the power button and a shell, Permissions and Privilege, Properly the six layers between a process and a file, and The Life of a Packet everything between an application and the wire. For the encryption layer specifically, disk encryption on a Linux laptop covers LUKS in practice.