Here is a directory listing in which every name is present and not one of the files can be reached.
$ ls d
alpha
beta
$ ls -l d
ls: cannot access 'd/alpha': Permission denied
ls: cannot access 'd/beta': Permission denied
total 0
-????????? ? ? ? ? ? alpha
-????????? ? ? ? ? ? betaThe directory is mode 644: readable, not searchable. The r bit handed over the names. The x bit, which is missing, is the permission to turn a name into a thing — and without it every name in that listing is a dead end.
That row of question marks is the whole subject. A directory entry is a name pointing at an object; path resolution is the machinery for following that pointer; and almost everything written about it describes a different process entirely — a function that takes a string and returns a tidier string. Strip the .., collapse the slashes, check the prefix, and you have “the path”.
The kernel has no such function. It has a walk over a live, mutable graph, performed once, that produces a reference — not repeatable, not reversible, and not a property of the string you started with. Fourteen CVEs in runc since 2017 are that difference, and so is most of the everyday confusion about symlinks, mounts, realpath and /proc.
This page follows one pathname from the first character to an open file, in six stages, in the order the kernel actually does them.
The six stages.
- Where the walk starts — the root and the working directory, both references rather than strings
- One component at a time — the search bit, and the trailing slash that defeats
O_NOFOLLOW - Symlinks — one budget of forty, and why
..is not a string operation - Mounts — the walk can be redirected, and a descriptor and a path can disagree
- Getting a name back out —
realpath,/proc/PID/fd, and why neither is “the path” - Holding on safely —
openat,openat2, and turning a race into an error
The hinge. A directory entry is a name that points at a thing, and the machinery that turns the first into the second runs once.
So: if what you hold is a name, you hold nothing. The binding it depends on can be re-made by anyone with write permission on the directory, and it will be re-evaluated the next time you use it. If what you hold is a thing, you hold it — it can lose every one of its names and survive, but you cannot look anything up through it without doing the whole walk again.
Which half of this article you need is decided by one question: is the object of your worry the binding, or the thing? If a rename, a mount, a symlink or another process’s view could make your name mean something else — stages 1 to 4. If the question is what you are actually holding, what it can still be called, and how to keep hold of it — stages 5 and 6.
With one exception, which is the subject of the worked diagnosis at the end: on a stacked filesystem the kernel reserves the right to change its mind about which thing, and it will not tell you.
Stage 1: Where the walk starts
A pathname beginning with / starts at the process’s root directory. Anything else starts at its current working directory — or, for the *at() family, at the directory named by a file descriptor you pass in, with AT_FDCWD meaning “use the working directory after all”.
Both the root and the working directory are per-process, and both are references, not remembered strings. You can see them, and walk into them, in another process’s /proc:
$ readlink /proc/3679/root
/home/claude/lab/jail
$ readlink /proc/3679/cwd
/home/claude/lab/jail
$ cat /proc/3679/root/inner/f
JAILEDProcess 3679 is inside a chroot. Its root is not a hiding place; it is an entry in its own /proc directory, and anyone able to read that directory walks straight into it.
getcwd() is a rendering, and it can fail
The working directory is a reference. getcwd() is an attempt to render that reference back into a name, and the rendering can fail while the reference stays perfectly usable — three ways, all of which happen in real life:
# 1. someone removed the directory you are standing in
getcwd normal -> /home/claude/lab/gone/inner
getcwd after rmdir(cwd) -> No such file or directory
readlink(/proc/self/cwd) -> /home/claude/lab/gone/inner (deleted)
stat(".") still works -> ok
# 2. the tree is deeper than any string that can name it
created and chdir'd through 2000 nested directories
getcwd(buf, PATH_MAX=4096) -> Numerical result out of range
getcwd(NULL, 0) [glibc autosize] -> ok, len=72911
open(".") relative still works -> ok
# 3. the cwd is outside the process's own root
getcwd now -> No such file or directory
open("etc/hostname") relative -> OK: vmThe second case is the one to keep. PATH_MAX is a limit on strings, not on the filesystem. The tree can be deeper than any pathname that can address it, and a process sitting at the bottom of it goes on opening files by relative path quite happily while being unable to say where it is. If a path were a property that a file had, this would be impossible.
chroot changes the root and revokes nothing
Everything people find surprising about chroot(2) follows from its changing one reference and touching nothing else. Descriptors opened before the call keep working after it — including one on a directory, from which fchdir() and a run of .. walks straight back out. And the new root is published, as above, to any process that can read /proc.
None of that is a bug. It is only surprising if you believe resolution is a property of the string, in which case changing what / means ought to change what your existing handles mean. It does not, because they are not strings. Containers do the same job with mount namespaces and pivot_root(2), which belongs to Containers, All the Way Down.
Stage 2: One component at a time
The kernel finds the next /, resolves the component before it, and repeats. That is the entire loop. There is no point at which a cleaned-up list of components exists, no canonical form computed up front, and nothing you could inspect that would tell you where the walk is going to end.
This is not pedantry. It is the reason string sanitising cannot be made to work: the thing you would have to sanitise never exists as a unit. Every component is resolved against whatever the tree says at the instant the walk reaches it.
Three rules govern the loop.
Runs of slashes collapse. stat("/"), stat("//") and stat("////") all return the same inode. POSIX permits a leading double slash to be given a special meaning and requires three or more to collapse; Linux collapses all of them. Worth testing rather than reading about, because path_resolution(7) — the page named after this subject — does not mention multiple slashes at all.
Every directory in the path needs the search bit. Not just the last one. The x bit on a directory is checked on each component, and failing it is EACCES — which is what produced the row of question marks at the top of this page. The full ordering of that check, with capabilities, ACLs, LSMs and the rest, is in Permissions and Privilege, Properly.
A trailing slash asserts that the last component is a directory. stat("regular/") fails with ENOTDIR; rename("x", "y/") fails if y is not a directory. That much is well known, and it is much less interesting than what follows from it.
One character defeats O_NOFOLLOW
“Final component” is decided during the walk, not before it. A trailing slash means the symlink you named is no longer the final component — so the rules that apply only to the final component stop applying to it.
lstat("l_dir") mode=120777 ino=794859 S_ISLNK = 1
lstat("l_dir/") mode=40755 ino=794858 S_ISLNK = 0 <- lstat followed the link
open("l_dir", O_NOFOLLOW) -> Too many levels of symbolic links
open("l_dir/", O_NOFOLLOW) -> ok
/proc/self/fd/3 -> /home/claude/lab/sym/realdirRead the second block twice. O_NOFOLLOW, the flag whose entire job is “do not follow a symlink”, opened the symlink’s target because one character was appended to the path. lstat(), the call whose entire job is “do not follow a symlink”, reported the target’s mode and inode for the same reason. The same applies to any path where the link is not last: open("l_dir/marker", O_NOFOLLOW) follows it too.
If your code defends an untrusted path with O_NOFOLLOW or an lstat() check, this is the hole, and it is a quiet one: the code looks correct and the tests pass, because nobody writes a test that appends a slash.
One performance fact belongs here and then goes away: the kernel caches the result of a component lookup, including the result that the name does not exist. On the idle machine these numbers came from, two-thirds of the dentry cache was those negative entries, and a cold stat() of a missing file measured 174 µs against a warm one at 277 ns — roughly six hundred times faster — which is the only reason the RESOLVE_CACHED flag in stage 6 makes sense.
Stage 3: Symlinks, and the one budget
When a component resolves to a symbolic link and the walk is going to follow it, the link’s contents are pushed onto the front of the remaining path and the walk continues. Same loop; there is just more of it.
There is one limit, it is forty, and it is a total for the whole lookup rather than a depth:
# a chain: s40 -> s39 -> ... -> target (nesting depth 40)
chain of 40 links: ok
chain of 41 links: Too many levels of symbolic links
# sequential: d/l1/l2/l3/.../lN (each of depth 1)
40 sequential symlinks: ok
41 sequential symlinks: Too many levels of symbolic linksIdentical cutoff. One budget of forty, spent however you like.
If you have read that the nesting limit is eight, that was true and stopped being true in Linux 4.2, August 2015, when Al Viro’s non-recursive rewrite replaced kernel-stack recursion with an explicit symlink stack and deleted the separate depth limit. A page that still says eight is telling you when it was written.
.. is a live pointer, not a text edit
The kernel does not read .. out of the directory, and it does not chop the last component off a string. It follows the current parent link of the entry it is standing on. “Current” is load-bearing, and testable: hold a directory descriptor, move that directory to a different parent, and ask for ...
held dirfd; dir was moved: mv1/sub -> mv2/sub
fstatat(dirfd, "..") ino=795128
mv1 ino=795126
mv2 ino=795128 <- '..' resolved to the NEW parentNothing textual could do that: the string never changed and the answer did. The same mechanism is why .. at the root of a bind mount lands in the mountpoint’s parent rather than the source’s, and why /.. is /.
Your shell disagrees with all of this, on purpose. Bash and dash both keep their own textual idea of where you are, and after you cd through a symlink and then cd .., the two ideas part company:
PWD (shell variable) = /home/claude/lab/link/sub
pwd -P (physical) = /home/claude/lab/real/sub
readlink /proc/self/cwd = /home/claude/lab/real/sub
after cd .. :
PWD = /home/claude/lab/link
/bin/pwd = /home/claude/lab/realAnd then the part that bites: every child process the shell spawns agrees with the kernel, not with the shell. A script started from that prompt gets /home/claude/lab/real from getcwd() while $PWD in the parent says /home/claude/lab/link — so a script that does cd .. and then acts on $PWD is acting on a different directory from the one it is standing in. pwd -P is the escape hatch; the shell options that control this belong to The Shell, in Depth.
The archive that contained things it should not
The everyday version of all this is a backup. A nightly archive of a web root comes back forty gigabytes instead of two hundred megabytes, or contains an SSH private key, because one entry inside the tree is a symlink pointing out of it.
l site/public/uploads -> ../../elsewhere
f site/elsewhere/id_rsa
tar -czf (default) : public/ public/index.html public/uploads
tar -czhf (-h/--dereference): public/ public/index.html public/uploads/ public/uploads/id_rsaWhether the link is followed is decided by one flag, and the widely repeated advice about which flag is wrong. “cp -r follows symlinks, so use cp -a” is false on current GNU coreutils: -r, -R, -a, -P and -d all preserve symlinks, and only -L follows. What dereferences is cp with no recursion flag at all, given a symlink as its source. The mnemonic is wrong in both directions, and the flags that genuinely dereference — cp -L, tar -h, rsync -L — go unmentioned.
The sysctl people reach for after this happens does not help. fs.protected_symlinks (Linux 3.6, 2012) restricts following a symlink in a sticky, world-writable directory when the link’s owner, the follower and the directory’s owner do not line up. It is aimed at /tmp attacks and has no opinion about your web root.
Do not trust a table of which distribution enables the link-hardening sysctls. There are four — protected_symlinks and protected_hardlinks (Linux 3.6, 2012), protected_fifos and protected_regular (Linux 4.19, October 2018) — and what a distribution ships in sysctl.d is a request, not a fact about the running kernel. On the stock Ubuntu 24.04 image these examples were run on, the shipped file sets all four and the kernel has three of them at zero, because systemd-sysctl never ran: the ordinary state of every container built from a base image and every CI runner.
Worse, these values are not namespaced. A container inherits the host’s policy and cannot change it, so its own sysctl.d files are decorative. Ask the kernel you are actually on:
sysctl fs.protected_symlinks fs.protected_hardlinks \
fs.protected_fifos fs.protected_regular # what is enforced
systemctl is-active systemd-sysctl # whether it was ever appliedAnd before you test them and conclude they do nothing: the symlink rule exempts the case where the owners line up, so the naive test passes at every setting, and protected_fifos and protected_regular gate O_CREAT opens only.
Stage 4: Mounts, and the walk being redirected
A mount is one rule: when the walk reaches this directory entry, continue somewhere else instead. Not a copy, not a link, not an attribute of the files — a redirection applied at a particular point in a particular process’s view of the tree.
What that means is visible directly: a descriptor and a path, in the same process, at the same instant, disagreeing about the contents of the same directory.
# hold a dirfd on cov/, then mount something over cov/
openat(held dirfd, "x") -> UNDERNEATH
open("/home/claude/lab/cov/x") -> ON_TOPThe descriptor holds the directory that is now covered; the pathname sees the mount. Neither is stale, neither is wrong, and they will never agree again. That is the whole of why “the file at this path” is not a well-formed idea.
Scale that up and you get mount namespaces: two processes on one machine, one absolute string, two different files, simultaneously, neither misconfigured. That is how containers get their filesystems.
One thing, many names
You do not need a mount to break “the path of a file”. A hard link is enough:
ln1/n1 ino=795189 nlink=3
ln1/n2 ino=795189 nlink=3
ln1/n3 ino=795189 nlink=3
opened via n2; /proc/self/fd says: /home/claude/lab/ln1/n2
after unlink(n2), same fd: /home/claude/lab/ln1/n2 (deleted)The file is entirely alive under two other names, and /proc/self/fd calls it deleted — because it is reporting the name you used to open it, recorded at open time, not a current fact about the object. /proc/PID/fd does not tell you the path of a file. It tells you a path, chosen once, and it is under no obligation to still be true.
Bind mounts do the same thing at the directory level, and realpath cheerfully returns whichever branch you fed it. There is no flag that makes it return “the real one”; the question is malformed.
Your systemd units are doing this to you
This is not container-only. Every directive in the table below is implemented by giving the unit its own mount namespace.
| Directive | What resolution does inside that unit |
|---|---|
ProtectHome=yes | /home, /root and /run/user exist, resolve, and lead nowhere |
ProtectSystem= | /usr read-only, or plus /etc, or the whole hierarchy |
PrivateTmp= | /tmp is a different directory from yours, under the same name |
ProtectProc= / ProcSubset= | changes what resolves under /proc, magic links included — so it can break /proc/self/fd reopening |
RootDirectory= / RootImage= | changes the resolution root outright, via pivot_root(2) or chroot(2) |
Those are systemd’s own semantics, so the table is safe to print; what varies is which units use them. There is no default worth quoting either — Fedora proposed turning this sandboxing on across roughly two hundred system services and its engineering committee dropped the change on 25 February 2025 — so for a given unit you have to go and ask it.
systemd-analyze security myservice # what is actually in effect
PID=$(systemctl show -p MainPID --value myservice)
readlink /proc/$PID/ns/mnt /proc/self/ns/mnt # different => different filesystem
readlink /proc/$PID/root # not "/" => RootDirectory is in play
nsenter -t $PID -m -- ls -l /the/path/in/questionThat last line is the point of the section. The only way to find out what a path means for a process is to ask from inside that process’s namespace — nothing on your own terminal is evidence about what the service sees. And systemd.exec(5) makes the same concession the chroot section did: the effect of these settings “may be undone by privileged processes”. They are resolution scopes, not boundaries.
Stage 5: Getting a name back out
Stages 1 to 4 turned a name into a thing. Everything in this stage runs the other way — you have a thing, or a reference to one, and you want a name for it — and every tool that does it is lossy in a different way.
realpath is two different programs
realpath(3), the C library function, requires every component to exist. GNU realpath(1), the command, allows the last component to be missing by default. Same name, different contracts, identical inputs:
realpath(3) lab/link/sub -> /home/claude/lab/real/sub
realpath(3) lab/link/sub/../nothere -> No such file or directory
realpath(1) lab/link/sub -> /home/claude/lab/real/sub
realpath(1) lab/link/sub/../nothere -> /home/claude/lab/real/nothereA shell script that validates a path with realpath "$p" and a C program that calls realpath(p, NULL) will disagree about whether the same path is valid. readlink -f behaves like the command; readlink -e requires every component; realpath -m allows any number missing; realpath -s does not follow symlinks at all.
And whichever you use, it does not undo a bind mount or a hard link, because there is nothing to undo. One inode with several equally real paths has no canonical one, and code that assumes otherwise — deduplication, “is this file already open”, path-based locking — is broken in a way that only shows up in production.
Magic links: a name that does not need a walk
The entries under /proc/PID/fd/ look like symlinks and are not. Opening one does not resolve the text it appears to contain; it re-opens the object the descriptor refers to. Which is why it works for a file that has no names left at all:
fd after unlink: fd=3 ino=795092 nlink=0
readlink("/proc/self/fd/3") -> "/home/claude/lab/m/file (deleted)"
open("/proc/self/fd/3") -> ok, content = "SECRET_CONTENTS"
stat("/home/claude/lab/m/file (deleted)") -> No such file or directoryNote the trap in the last two lines. The string readlink gives you does not resolve. It is a label, not a path — and ” (deleted)” is not an escaping convention, so it is ambiguous against a real file whose name genuinely ends in that text.
Three things about magic links are worth more than the fact that they exist.
Re-opening does a fresh permission check, so this is not a capability. Half of what is written about /proc/self/fd/N treats it as a handle that carries the rights you had when you opened it. It does not; it re-derives them from the inode.
# unprivileged, uid 65534
open(ro444, O_RDONLY) -> ok
reopen /proc/self/fd/N as O_RDWR (mode 444) -> Permission denied
reopen /proc/self/fd/N as O_RDWR (mode 600, mine) -> ok <- upgradedNot an escalation across users — but an escalation across access modes. A program that deliberately opened a file read-only can silently re-acquire write on a file it owns. An O_RDONLY descriptor is not a read-only capability for any process that can see /proc.
O_PATH is the opposite, and it is the whole article in one flag. It gives you a reference to a thing and no authority over it: an unprivileged process can hold an O_PATH descriptor on a root-owned, mode-000 file and fstat() it, while read() returns EBADF and laundering it through /proc/self/fd returns EACCES. A name for a thing, held open, with nothing attached.
Until this August, the magic link was the only way to re-open an O_PATH descriptor — and it needs /proc mounted, which a hardened container may not have. AT_EMPTY_PATH does not do it for openat(), and neither did openat2(). O_EMPTYPATH was proposed in the original 2019 openat2 series, dropped, and finally landed in Linux 7.2 on 16 August 2026, seven years later. On anything older, a container without /proc simply cannot turn an O_PATH reference back into a usable file.
A magic link is a name that carries a descriptor’s authority, and that has been a container escape. CVE-2024-21626, disclosed 31 January 2024 and fixed in runc 1.1.12: runc changed into the container’s configured working directory before closing an inherited descriptor that still referred to a host directory. Set an image’s WORKDIR to a path under /proc/self/fd/, and the container starts life standing in a host directory, from which ../../.. walks the host root.
The entire exploit is text in a Dockerfile. No race to win, no timing window, nothing to lose — a string was handed a reference’s authority, which is the boundary this article is about, crossed in the one place the kernel deliberately allows it.
One note for anyone coming here from The Life of a File Descriptor: the thing a descriptor points at is an open file description, and that is the level at which all of this happens. This article does not repeat those three levels; it only cares which of them a name has been turned into.
Stage 6: Holding on safely
Every path-safety bug has one shape: you check a name, then you use it, and the two operations are separate walks over a tree somebody else can edit in between.
The standard advice — canonicalise the path, confirm it starts with your base directory, then open it — fails not because the window is small but because there need not be a window at all. renameat2() with RENAME_EXCHANGE (Linux 3.15, June 2014) swaps two directories atomically:
step 1 CHECK : realpath -> /srv/safe/data (prefix approved)
step 2 ATTACK: renameat2(..., RENAME_EXCHANGE) on the two dirs -> ok
step 3 USE : open(the approved path) reads: PWNEDNo timing to get right, no retries, completely reliable. The check validated a string; the open() performed a different walk. Any page that reassures you “the race window is very small” has not understood what it is racing against.
The same reasoning disposes of path sanitising generally. Stripping .., collapsing slashes and checking a prefix operates on text; symlinks, bind mounts and namespaces are not in the text. That is the direct cause of the November 2025 runc vulnerabilities, in which the target of a protective bind mount was replaced with a symlink.
openat is not automatically the fix
The usual next step is openat(dirfd, relative_path, ...), on the theory that holding a descriptor on the base directory constrains the result. It does not, once the relative path has more than one component:
openat(base, "mid/leafdir/t") before -> "GOOD"
[attacker replaces base/mid/leafdir with a symlink to ../../evil]
openat(base, "mid/leafdir/t") after -> "EVIL"The descriptor on base was held throughout. Every component after it was still a name, resolved at the moment of the call, and the walk went straight out of the base directory.
The rule that actually holds: safety scales with how few components you resolve per call. One component at a time, each opened into a new descriptor which becomes the base for the next, is safe. An n-component path in one call is n−1 unguarded lookups. That the kernel had to grow a flag to make the multi-component case safe is itself an admission that openat alone is not.
openat2 and the resolution restrictions
openat2() arrived in Linux 5.6, March 2020. It takes a struct open_how instead of a flags word, and its resolve field lets you constrain the walk itself rather than inspecting the result afterwards. Since glibc 2.43, released 24 January 2026, there is a library wrapper; before that you called it through syscall().
The two you will actually reach for are RESOLVE_BENEATH and RESOLVE_NO_SYMLINKS.
| Flag | Refuses |
|---|---|
RESOLVE_BENEATH | anything that leaves the starting directory: .. escapes, absolute paths, symlinks to absolute paths |
RESOLVE_IN_ROOT | nothing — it reinterprets, treating the starting directory as / |
RESOLVE_NO_SYMLINKS | every symlink, including the final component |
RESOLVE_NO_MAGICLINKS | /proc/PID/fd-style links only |
RESOLVE_NO_XDEV | crossing a mount point — and opening the mount point itself |
RESOLVE_CACHED | anything requiring I/O; fails with EAGAIN so you can retry slowly (Linux 5.12) |
Three behaviours matter more than the list, and none of them is on the pages that print the list.
BENEATH refuses; IN_ROOT reinterprets. They are not two strengths of the same thing. BENEATH fails when an escape is attempted. IN_ROOT clamps — .. at the fake root is the fake root, an absolute path is re-read against it — and then usually fails with ENOENT because the clamped path names nothing. That is the same ENOENT a typo produces, so IN_ROOT cannot tell you that an attack happened. If you are logging attempts, you want BENEATH.
The errno for an escape is EXDEV — otherwise “invalid cross-device link”, which reads like a filesystem hiccup rather than a security event. Anyone grepping logs for EPERM or EACCES will miss every one of them.
RESOLVE_NO_XDEV refuses to open the mount point directory itself, not merely to traverse it. “No crossing” does not include standing on the line.
Two reasons openat2 has not ended this class of bug, six years after it shipped.
The first is that it is frequently unavailable exactly where it is needed. One of its arguments is a pointer, which seccomp filters cannot inspect, so sandbox policies routinely block the syscall outright — and the programs behind those policies are container runtimes, the ones with the problem. They fall back to walking paths in userspace.
The second is that the scoping flags do not remove the race, they convert it into an error you must handle: openat2() returns EAGAIN when RESOLVE_BENEATH or RESOLVE_IN_ROOT meets a rename mid-walk, and the caller retries. Under load that loop can starve — one of the November 2025 runc fixes is named, in so many words, for openat2 resilience on busy systems.
The practical conclusion is the boring one: use a library. libpathrs and filepath-securejoin exist because the correct sequence is long, and the current versions are descriptor-based. A page describing securejoin as a string function is describing the version that was replaced.
Linux 7.2, 16 August 2026, added two things here: O_EMPTYPATH, covered in stage 5, and OPENAT2_REGULAR, which refuses to open anything that is not a regular file and brings a new errno, EFTYPE, to say so. Check the header on the kernel you are running before hard-coding either value; they are new enough that the numbers are not worth trusting from a web page, this one included.
That applies to the manual page too. openat2(2), revised on 8 February 2026, still says glibc provides no wrapper — false two weeks earlier — and mentions none of the three additions above. Not a criticism of anybody: it is the ordinary lag of a volunteer-maintained document against a kernel that ships every nine weeks, and it is why “check the man page” is not a substitute for checking the kernel you are on.
Symptoms and which stage owns them
| What you see | Stage | What to run |
|---|---|---|
ls lists names, ls -l shows -????????? | 2 | stat -c '%A' thedir — no x bit |
getcwd() fails but the process works | 1 | readlink /proc/$PID/cwd |
O_NOFOLLOW or lstat did not stop a symlink | 2 | look for a trailing slash on the path |
Too many levels of symbolic links | 3 | namei -l; the budget is 40 for the whole path |
$PWD and pwd -P disagree | 3 | readlink /proc/self/cwd; you came through a symlink |
| A backup is far too large, or contains files from outside the tree | 3 | find tree -type l -lname '*..*' |
| The service says the config lacks a value you can plainly see | 4 | nsenter -t $PID -m -- cat /the/config |
realpath in a script and in a program disagree | 5 | the command allows a missing last component; the function does not |
/proc/PID/fd says (deleted) for a file that exists | 5 | stat -c '%h' thefile — it has other names |
| A prefix check passed and the wrong file opened | 6 | nothing to run; the design is the bug |
| Two code paths in one process disagree about a file’s contents | — | findmnt -T thefile; see below |
That last row is not a stage, and it is the one that takes longest to find.
A worked diagnosis: the file that reloaded for half the process
A service in a container reads a data file at startup — a GeoIP database, a compiled translation catalogue, something updated occasionally and too big to re-read per request. Operations updates the file inside the running container.
The health endpoint, which read()s the file, reports the new version. The request path, which uses a memory mapping of the same file, keeps serving the old data. Restarting fixes it. It has never once reproduced on a developer laptop running the service directly.
Down the spine.
Stages 1 and 2. Both code paths use the same string, in the same process, with the same working directory. Nothing here.
Stage 4. First real signal.
$ findmnt -T /data/geoip.mmdb
TARGET SOURCE FSTYPE OPTIONS
/ overlay overlay rw,relatime,lowerdir=...,upperdir=...,workdir=...Stage 5. The obvious theory is the classic one: somebody replaced the file, the process is holding the old inode, and /proc will say (deleted). It does not.
$ ls -l /proc/$PID/fd | grep geoip
lrwx------ 1 root root 64 Sep 2 21:37 7 -> /data/geoip.mmdb
$ grep geoip /proc/$PID/maps
7f2a1c000000-7f2a1c400000 r--s 00000000 00:27 827564 /data/geoip.mmdb
$ stat -c '%i' /data/geoip.mmdb
827564The descriptor is not deleted. The mapping’s inode number and the file’s inode number are the same. Every observable says nothing has happened, which is why this costs people days.
What is actually happening is visible only by asking the two code paths separately:
BEFORE copy-up ino=827564
read() first 16: LLLLLLLLLLLLLLLL
mmap first 16: LLLLLLLLLLLLLLLL
AFTER copy-up ino=827564 (same fd, never reopened)
read() first 16: UUUUUUUUUUUUUUUU <- the upper file
mmap first 16: LLLLLLLLLLLLLLLL <- still the lower fileOne descriptor. Two answers about which file it is.
The cause is dated. Since overlayfs gained stacked file operations in Linux 4.19, October 2018, an overlayfs open file holds a real file underneath it and revalidates that on each operation, re-opening against the upper copy if the file has been copied up meanwhile. read() goes through the revalidation; an established shared mapping does not, having been bound to the lower inode’s address space when it was created. And the inode number survives the copy-up because overlayfs’s xino machinery exists to keep it stable. The one property you would reach for to detect this is engineered to lie, on purpose, as a feature.
The kernel’s own overlayfs documentation lists exactly three deviations from POSIX under “Non-standard behavior”, and one of them is this — but only the mapping half: a read-only lower file that is memory mapped with MAP_SHARED will not see subsequent changes. That is true, and its framing points away from the answer, because it invites you to conclude that the descriptor is pinned to the lower file and the mapping shares its fate. The reality is the reverse and stranger: the descriptor moves and the mapping does not. The read() half is not documented anywhere.
The moral. You never opened a file. You opened whatever that name meant at the time — and on a stacked filesystem, which is the root filesystem of essentially every container running today, the kernel reserves the right to re-decide, and to re-decide differently for different parts of your program.
The fix is to stop treating a long-lived mapping as a view of a file: re-open on a version signal, or put the data on a volume rather than on the overlay. The reason to know the mechanism is that no amount of staring at stat output would ever suggest either.
How to tell whether a page about this is worth reading
There is a great deal written about path resolution and most of it is wrong in one specific direction, which makes it unusually easy to filter.
Every wrong page models resolution as a function from strings to strings. Not “it omits mounts” or “it predates openat2” — those are symptoms. The error underneath is the belief that there is a canonicalise(s) producing a definitive s', followed by an open() that looks s' up in a table. Given that model everything else follows: sanitising works, realpath has one answer, /proc/PID/fd reports “the path”, and a directory is a container of files rather than a set of bindings.
So the test is a single question, and it partitions the corpus cleanly:
Does this page ever say that the same string, resolved twice, can give two different objects?
If it does not, it is written in the string model, and everything it tells you about safety is wrong.
Failing that, grep it. These phrases are either artefacts of the string model or facts with an expiry date:
- “sanitise the path”, “strip
../“, “normalise the path”,filepath.Cleanoros.path.normpathpresented as a security control realpathused as a verb of validation — “realpath it and check that it starts with”- “use
lstatto check it is not a symlink, then open it” — the textbook check-then-use bug, andlstatis itself defeated by a trailing slash - a nesting limit of 8, or 5 — predates Linux 4.2, August 2015
O_NOFOLLOWrecommended with no mention of the trailing slash- “glibc has no
openat2wrapper, usesyscall()” — expired 24 January 2026 - a
RESOLVE_*list of exactly five, or exactly six — predates Linux 7.2, 16 August 2026 - “chroot jail” used approvingly;
PATH_MAXdescribed as a filesystem limit
What a page does not say is at least as diagnostic, because a string-model author has no reason to mention any of this:
| If it never mentions… | …then it |
|---|---|
| that a descriptor and a path can disagree about what a directory contains | has not understood the subject; that disagreement is the entire difference between a name and a thing |
| that the same string can resolve to two different files at the same moment | predates mount namespaces being ordinary, so roughly 2013 |
openat2 at all | predates March 2020, or is a pure-userspace page that will get you breached |
that openat2 is frequently blocked by seccomp | has read about the syscall rather than shipped it |
RENAME_EXCHANGE | believes check-then-use races need a timing window; anything it says about small windows is worthless |
| overlayfs, while discussing open files in containers | has only ever tested on a plain filesystem |
that /proc/PID/fd can be wrong | is in the string model, terminally |
O_EMPTYPATH, or that magic-link reopening needs /proc | predates August 2026, or has never deployed into a hardened container |
The primary sources worth reading directly are path_resolution(7), openat2(2), symlink(7) and proc(5) — with the gaps named above — the kernel’s overlayfs documentation, and LWN’s coverage of the openat2 restrictions and of safe path traversal.
One thing deliberately left out: there is a third kind of reference, the file handle from name_to_handle_at(), which survives a reboot. NFS is built on it, resolving one needs CAP_DAC_READ_SEARCH, and you almost certainly should not.
Related reading
- The Life of a File Descriptor — what you are holding once the walk has finished, and the three levels behind the number
- Permissions and Privilege, Properly — the six layers behind the search bit, in the order the kernel checks them
- How filesystems work — inodes, links, and why deleting is really unlinking
- Containers, All the Way Down — mount namespaces,
pivot_root, and the overlay this article’s worked diagnosis runs on - The Shell, in Depth — why your shell keeps its own idea of where you are
- lsof and fuser — finding what is holding a file, once you know a path will not tell you