Here is a small program. It opens a file, forks, waits for the child, asks its own PID and exits — five library calls, and I wrote every one of them by name. This is what actually crossed into the kernel:
$ strace -n ./names
[ 257] openat(AT_FDCWD, "/etc/hostname", O_RDONLY) = 3 <- I wrote open()
[ 56] clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|…) = 2408 <- fork()
[ 61] wait4(2408, NULL, 0, NULL) = 2408 <- I wrote waitpid()
[ 39] getpid() = 2407 <- I wrote getpid()
[ 231] exit_group(0) = ? <- I wrote exit()Four of the five names do not appear anywhere in my source. And that is the friendly case, because at least something crossed. On the same machine, in the same afternoon: a program that calls clock_gettime() twenty-one times, which strace -c reports as six calls. A program that calls read() once, gets one byte and one answer, and produces four read() lines in the trace — three of them returning a value that appears in no header on the system.
None of that is a bug in strace. It is the boundary behaving normally, and the reason it looks like a bug is a model almost everybody carries: that a system call is a function call into a library that happens to live in the kernel. It is not. Four different parties can answer for the kernel at this boundary — the kernel, a seccomp filter, the vDSO, and a userspace dispatch handler — and your tools attribute all four answers to the kernel.
That matters here more than it would elsewhere, because this site now has several long pages resting on one move: read a tool’s output, and route the symptom to a stage. Containers, All the Way Down routes on docker inspect, The Life of a File Descriptor on /proc/PID/fd, How a Path Becomes a File on a resolution walk, The Life of a Unix Socket on an errno. Not one of them tells you when the tool’s output is not evidence — and this is the boundary where it routinely is not. So this page owns the failure mode of the method the rest of them teach.
Every number here came off one machine, and it is named: Linux 6.18.44 on x86-64, glibc 2.39, strace 6.8, an Intel Xeon at 2.80 GHz — and, which matters more than the rest, meltdown: Not affected, so there is no page-table isolation on it. Where something was not measured, the page says so and gives you no number.
What this page does not cover, and where it went instead.
- What
forkandexecvedo, the shebang,PATH, the dynamic linker and zombies. What Actually Happens When You Run a Program owns all of it. This page usesforkonly as an example of a name that is not the name that crossed. - Namespaces, cgroups, capabilities, and seccomp as container policy. Containers, All the Way Down. This page owns one clause that page leaves open: it says the default profile has “a default action of
SCMP_ACT_ERRNO“, and which errno turns out to decide whether unmodified programs keep working. - What a descriptor is,
EMFILEagainstENFILE, and the cost of a syscall loop — The Life of a File Descriptor. Path resolution,openat2and magic links — How a Path Becomes a File, which also proves that a seccomp filter cannot follow a pointer. This page only uses that, to explain why seccomp and Landlock are not the same kind of thing.
And the honest limitation, stated here rather than at the end: this page is about x86-64 Linux with a glibc userspace. On 32-bit userspace the syscall names themselves differ, and on musl the wrapper decisions are different ones.
The seven stages of a system call. Every section below is named after one.
- The name you wrote, and the number the library picked.
- The calls that never leave — and what answers them instead of the kernel.
- The instruction — which is not a call, and destroys two registers to prove it.
- The filter that answers first — before the kernel has looked at anything.
- The far side, and what it costs — to run, and to watch.
- The answer coming back — the error band, the calls that happen twice, and the fourth name.
- What the answer becomes — the stage where “it worked yesterday” lives.
The diagnostic hinge: the number is the kernel’s, the name is everybody else’s.
Your program names a function. The C library picks a number. The kernel only ever sees the number. strace reads the number back out and prints a name again. Four names, one number, and nothing on the machine checks that the first name and the last one are the same name. So do not ask what failed. Ask whether a number crossed, and who answered it.
strace -n -f ./yourprogram 2>&1 | head -40-n prints the syscall number in brackets before each line. Find the operation you are asking about, and read which of these five you are looking at:
- No line for it at all. No number crossed. The vDSO answered it, or the C library did, or a userspace dispatch handler did. Stages 1–2. Nothing in the kernel knows it happened, and no kernel-side tool will ever show it to you — not
strace, notperf, not eBPF. A program can spend all day here and look idle. - A line, under a name you did not write. The wrapper called something else:
open→openat,fork→clone,waitpid→wait4,exit→exit_group. Stage 1. You are reading the trace correctly; you were reading your own source wrongly. - A line, the right number, and an error that makes no sense. A number crossed and something other than the kernel answered it.
ENOSYSmeans there is no implementation at that number. Any other errno on a call that has nothing to do with permissions means a filter. Stage 4. - More lines than your program made calls.
ERESTARTSYS,ERESTART_RESTARTBLOCK,restart_syscall. The kernel started the same call more than once. Stage 6. Your program saw one call and one answer, and both of you are right. - A line, the right number, no error, and it is slow. Now — and only now — is it the kernel’s. Stage 5.
The weak point, said out loud: this hinge needs a reproduction. A process that has already failed in production left you a log line and no trace. The one thing readable after the fact is grep Seccomp /proc/PID/status — and it says only how many filters are stacked, never what they say.
Stage 1: The name you wrote, and the number the library picked
The trace at the top of this page is the whole of stage 1 in five lines. open() is a C function; openat is a syscall; glibc chose and did not tell you. strace printed exactly what crossed. The C library is a translation layer, and translation layers rename things.
The second half of stage 1 is the calls the library will not translate at all. This machine’s asm/unistd_64.h names 373 syscalls; comparing that list against the symbols glibc 2.39 actually exports, 85 of them — 23% — have no library function of the same name. It is not a tail of museum pieces:
openat2 clone3 seccomp statmount landlock_create_ruleset
faccessat2 cachestat fchmodat2 futex_waitv mseal io_uring_setup
... all NO WRAPPER, syscall(2) onlyThat is most of what anybody would reach for in 2026. The picture moves — glibc 2.43, January 2026, added mseal and openat2 — but it does not close, and io_uring will never be in libc at all.
syscall(2) is not a raw system call
This is the most common misunderstanding at the boundary, and it takes four lines to settle. Three ways of closing a descriptor that does not exist, in one program:
before: errno = 0
inline-asm close(-1) returned -9 errno = 0 (unchanged)
syscall(2) close(-1) returned -1 errno = 9 (Bad file descriptor)
close(2) close(-1) returned -1 errno = 9 (Bad file descriptor)syscall(2) is a libc wrapper. It returns −1 and sets errno, exactly like close() does. Almost everybody who says “raw syscall” means syscall(3), and they are describing something they are not using. Only inline assembly gives you the kernel’s number back — -9, which is -EBADF, and which is the subject of stage 6.
Stage 2: The calls that never leave
Some calls are answered without the kernel being asked. The usual telling of this is a performance note — the vDSO makes clock reads fast — and the performance note is true and is the least useful thing about it. The useful thing is that a call answered here is invisible to every kernel-side tool that exists. Here is the whole argument in one block. The program calls clock_gettime() twenty-one times:
the program called clock_gettime() 21 times
% time seconds usecs/call calls errors syscall
0.00 0.000000 0 6 clock_gettimeFifteen calls happened and nothing recorded them. Not strace, not ptrace, not perf, not eBPF — there is nothing to hook, because nothing entered the kernel. A program that spins on time reads looks idle to every tool you own, and “the trace is empty so the program is not doing anything” is wrong in the one case where it is easiest to believe.
The vDSO does not promise to stay in userspace
The vDSO is a small shared object the kernel maps into every process. It has no file on disk — to list its symbols you have to copy the [vdso] range out of /proc/self/maps via /proc/self/mem and run objdump -T on that. On this kernel it holds six symbols — clock_gettime, gettimeofday, time, getcpu, clock_getres and getrandom, each with a __vdso_ prefix. Your kernel may print a different set, which is the point: the list is machine state, and every page that prints one as if it were the list is out of date somewhere. vdso(7) still documents four for x86-64. __vdso_getrandom arrived in Linux 6.11, September 2024, and glibc started using it in 2.41, February 2025 — the largest change to what a trace looks like in a decade, because every program that reads randomness stops appearing to.
Now disassemble the thing. There are five syscall instructions inside the vDSO image. They are fallback paths. The vDSO does not promise to answer in userspace; it promises to try, and whether it succeeds depends on the argument you pass:
clock ns/call
CLOCK_REALTIME_COARSE 5.6
CLOCK_MONOTONIC 21.3
CLOCK_PROCESS_CPUTIME_ID 438.6 <- always a real syscall
CLOCK_MONOTONIC 336.6 (the same clock, forced through syscall(2))The last two rows are the same clock. The bottom one is what CLOCK_MONOTONIC costs when the vDSO cannot serve it — and nothing in your source decides that; the clocksource does. A clocksource that has fallen back from tsc, usually announced in dmesg as “Marking TSC unstable”, cannot be served from the vDSO, and every clock_gettime() silently becomes a system call. A function that was not a system call yesterday can be one today, and nothing in your program changed. I measured that gap here; I could not induce the fallback itself, so take the mechanism as demonstrated and the scenario as reported. current_clocksource is where the answer lives.
The party nobody expects
The vDSO is the famous answerer. It is not the strangest one. Syscall User Dispatch — prctl(PR_SET_SYSCALL_USER_DISPATCH), Linux 5.11, February 2021 — lets a process route its own system calls to a handler in userspace. It exists so that Wine and FEX can run foreign binaries, and it is entirely ordinary to install:
syscall user dispatch is ON, selector currently ALLOW
getppid() with selector=ALLOW : 2235
[SIGSYS] si_syscall=110 rax=110 si_code=2 -> answering 4242 in userspace
getppid() with selector=BLOCK : 4242 (intercepted 1 time(s))
# and all strace has to say about that second call:
--- SIGSYS {si_code=SYS_USER_DISPATCH, si_syscall=__NR_getppid} ---There is no getppid line for the second call, because there was no getppid. The selector that decides is one byte of ordinary memory; the process flips it with a store, not a call. So: four parties can answer for the kernel — the kernel, the vDSO, a filter (stage 4) and a dispatch handler — and three of them can produce something your program will file under “the kernel said no”.
Stage 3: The instruction, which is not a call
Briefly, because almost nothing here is actionable and the part that is gets misprinted everywhere. On x86-64 the syscall number goes in eax — thirty-two bits, not sixty-four — and up to six arguments go in rdi, rsi, rdx, r10, r8, r9. Then syscall executes.
The oddity everybody notices is r10, because the C calling convention on the same machine puts the fourth argument in rcx. The reason is that syscall is not a call. Nothing is pushed; there is no return address on the stack. The instruction stores the return address in rcx and the flags in r11, so both registers are gone before the kernel begins, and the fourth argument had to move. syscall(2) hides that behind a general remark about architectures that “may indiscriminately clobber other registers not listed here”, which is true and tells you nothing about the register you are about to get wrong.
| ABI | Instruction | Number in | Arguments | What the instruction destroys |
|---|---|---|---|---|
| x86-64 | syscall | eax | rdi rsi rdx r10 r8 r9 | rcx (return address), r11 (flags) |
| i386 | int $0x80 | eax | ebx ecx edx esi edi ebp | nothing — but every argument is truncated to 32 bits |
| arm64 | svc #0 | w8 | x0 x1 x2 x3 x4 x5 | x0 (holds the result on return) |
This is the only table between here and the symptom index at the end, and it earns that because it is the one thing on this page that is the same on every machine of its architecture that will ever exist. It is not configuration and it is not a default: it is the ABI, and if it changed, every binary on earth would stop working at once. Everything else this page could have tabulated — the syscall numbers, the vDSO’s symbols, the actions a filter can take — is machine state, and a table of machine state is a lie with columns. Note the middle row while you are here: int $0x80 still works on x86-64, indexes a different table, and truncates silently.
Stage 4: The filter that answers first
A seccomp filter runs at the entry point, before the kernel has looked at anything, and that timing is the whole of its character. It sees a syscall number, an architecture tag and six machine words, and it cannot follow a pointer — How a Path Becomes a File proves that, and it is why a filter can never say anything about a filename. What it can do is answer the call itself, with a return value of the policy author’s choosing, and hand that answer to your program with the kernel’s fingerprints on it.
The number is not the name, and here is the bill for forgetting it
Your machine has more than one syscall table. It has at least two — the x86-64 one and the i386 one — and a 32-bit process on the same kernel indexes the second. They do not agree:
nr x86-64 calls it i386 calls it
0 read restart_syscall
1 write exit
2 open fork
59 execve oldolduname
60 exit umask
numbers defined in BOTH tables: 369
of those, naming a DIFFERENT syscall: 330 (89%)Number 1 is write to a 64-bit caller and exit to a 32-bit one, on the same running kernel. A filter that allows number 1 believing it has allowed write has allowed something else entirely to anyone who asks in the other ABI — which is why every filter must check seccomp_data.arch first, and why one that does not is a bypass rather than a bug. seccomp(2) says it is “usually necessary” to verify that field. For a fact where 89% of shared numbers collide, “usually” is the wrong adverb.
A filter does not refuse a call. It answers it.
SECCOMP_RET_ERRNO carries a number, and somebody chose that number. Here is one filter blocking one syscall — clone3 — against an ordinary program that creates a thread, with nothing varying but the errno the filter returns:
mode=none pthread_create OK
mode=enosys pthread_create OK
mode=eperm pthread_create FAILED: Operation not permitted (1)
=== under ENOSYS ===
clone3({flags=CLONE_VM|…}, 88) = -1 ENOSYS (Function not implemented)
clone(child_stack=0x7f48d57fef70, flags=CLONE_VM|…) <- glibc falls back, and it works
=== under EPERM ===
clone3({flags=CLONE_VM|…}, 88) = -1 EPERM (Operation not permitted)
<- glibc gives upThis is the fact the rest of the page has been walking towards. ENOSYS is not an error. It is the only version number the syscall ABI has. Linux does not negotiate capabilities at this boundary: there is no handshake, no feature list, no uname you can trust. A program finds out what the kernel can do by making the call and reading the answer, and ENOSYS is the answer that means “not here, try the old way.” Every C library on Linux depends on it, and so does every container runtime.
So a filter that answers EPERM to a feature probe is not refusing a call. It is lying about the ABI.
The second hinge, for this stage: ENOSYS is an answer. Every other errno is a refusal.
strace -f ./yourprogram 2>&1 | grep -E 'ENOSYS|EPERM|EACCES'ENOSYS — there is no implementation at that number. This is not a failure, it is how every C library on Linux asks what the kernel can do, and a program that crashes on it has a bug of its own.
EPERM or EACCES on a call that has nothing to do with permissions — clone3, io_uring_setup, faccessat2, statx — the kernel did not say that. A filter said it, and it said the wrong word. The bug is in the policy, not in your program, and the fix is not to allow the call.
Two things about filters that are not in the manual page
A filter can never be removed. I installed four in one process and watched the counter go one way — Seccomp_filters in /proc/self/status running 0, 1, 2, 3, 4, and never back. They stack, all of them run on every call, and the highest-precedence action wins. There is no unload, no reset, and no way to read back what any of them says, only how many there are. seccomp(2) does not state this anywhere, and it is the first thing anyone writing a filter needs to know.
The size of a filter costs nothing. This is the opposite of the advice everybody gives. Same program, same syscall through inline assembly, filters of increasing size, minimum of nine runs each:
0 compares : 261.2 ns 512 compares : 278.8 ns
1 compares : 281.2 ns 4000 compares : 283.9 ns
64 compares : 282.1 nsFour thousand BPF comparisons are indistinguishable from one. The cost is having a filter — about 20 ns, 7.7% — because the process leaves the fast entry path, once. “Keep your seccomp profile small for performance” is measurably false, and it is advice that buys nothing in exchange for a less careful policy.
One last note and this stage is done. Seccomp and Landlock are not two versions of one layer: seccomp runs at entry on a number and six words, while LSM hooks run deep inside the call, after the arguments have been resolved into an inode, a mount, a credential. That is exactly why Landlock can express “this path” and seccomp structurally cannot, and why choosing seccomp for a filesystem policy is choosing the layer that lacks the vocabulary.
Stage 5: The far side, and what it costs
This is the shortest stage on the page, deliberately. Once the number has crossed and nothing has intercepted it, the kernel does the work — and what that work is belongs to the other long pages here. How a Path Becomes a File takes an open from the first slash; The Life of a File Descriptor owns the table a read lands in; The Life of a Unix Socket has the rest of a connect. The boundary is this page’s; the far side is theirs.
What belongs here is the price of crossing. A getppid() — about the cheapest real system call there is — costs 271 ns on this machine, against roughly 1.5 ns for a plain function call and 21 ns for a vDSO clock read. The usual explanation for that gap is Meltdown, and on this machine the usual explanation is not available:
$ grep . /sys/devices/system/cpu/vulnerabilities/meltdown
Not affected
$ grep -r . /sys/devices/system/cpu/vulnerabilities/ | grep -v 'Not affected'
spectre_v2: … retbleed: … spec_store_bypass: … (five mitigations, none of them PTI)No page-table isolation here at all, and a syscall still costs 271 ns. KPTI is enabled only on Meltdown-affected CPUs — roughly pre-2019 Intel, and never AMD — so on a large share of machines bought since then it is not running, and it has not been what you were paying for in years. What is charging you is the list above, and one of those is per-process, which lets you price a mitigation directly: switching Speculative Store Bypass Disable on by hand cost +18.3 ns per syscall, +7.1% here. Whether your containers pay it is a boot-parameter question — “via prctl” means opt-in, “via prctl and seccomp” means every filtered process pays.
Then there is the cost of watching a syscall, which is larger than most people’s estimate by two orders of magnitude. 200,000 getppid() calls, same binary, five invocations:
untraced 309.5 ns 1x
strace -f -o /dev/null 68,565.6 ns 222x
strace -c -o /dev/null 63,705.0 ns 206x
strace -f -e trace=write -o /dev/null 62,341.6 ns 201x
strace --seccomp-bpf -f -e trace=write … 353.6 ns 1.14xThree things fall out of that block. strace -c is not the cheap mode — it is 206×. -e trace= buys you nothing at all: 201× for a trace that printed no output whatsoever, because -e filters inside strace, long after the kernel has stopped the process twice for every call. And --seccomp-bpf is 175× cheaper than the same command without it, because the filtering moves into the kernel and the uninteresting calls never stop the process. strace has had it since 5.3, in 2019, and it is still not the default. If you take one flag away from this page, take that one.
Stage 6: The answer coming back
Every syscall returns one machine word. There is no error flag, no second register, no out-parameter. So how does a kernel say “that failed” through a channel in which every 64-bit value is a legal answer?
It reserves a band. A return value between −1 and −4095 is a negated errno; anything else is a result. MAX_ERRNO is 4095, and glibc’s test is literally (unsigned long) ret >= -4095UL. It has to be a band rather than “any negative number”, because plenty of calls legitimately return negative values, and mmap returns addresses that look enormously negative when read as signed.
You can watch the band bend a system call around itself. getpriority returns a nice value, which runs from −20 to 19 — and −20 is inside the error band. So the kernel does not return it:
nice kernel returns glibc getpriority()
0 20 0
10 10 10
19 1 19The kernel biases nice values into 1…40 so that a legal answer can never land in the error band, and glibc computes 20 - ret to give you back the number you asked about. Three rows, and they prove two things at once: the band is real, and the C library is not a shim. Which also means the convention everybody quotes belongs to the wrong layer. “−1 means error and errno holds the reason” is libc’s convention; the kernel’s is the band. syscall(2) — the manual page whose entire purpose is going around libc — states libc’s convention as if it were the system’s, gives the number 4095 nowhere, and leaves a faithful reader unable to work out that syscall(2) is itself a wrapper.
The answer that means “not here”
Syscall numbers are stable forever; Linux does not break userspace. Implementations are not stable, and the header is not the kernel. Twelve names in this machine’s headers return ENOSYS, including _sysctl — still #define __NR__sysctl 156 today, and gone from the kernel since Linux 5.5, January 2020. That direction is at least documented. This one is not, and it is now the common one:
460 Invalid argument <-- IMPLEMENTED (last name this libc knows: lsm_set_self_attr)
461 Bad address <-- IMPLEMENTED (last name this libc knows: lsm_list_modules)
462 returned 0 <-- IMPLEMENTED (mseal, Linux 6.10)
463…469 <-- IMPLEMENTED (the setxattrat family, file_getattr, file_setattr)
470 ENOSYS (no such call)Eight system calls this machine’s C library cannot name, which this machine’s kernel answers. “Invalid argument” or “bad address” from a number nothing can name is a call that exists, invoked with nonsense arguments. The only way to find out what a kernel implements is to ask it at runtime, one number at a time — which is what “the ABI has no version number” means in practice. One number does not answer with an errno at all: 335 kills the caller with SIGILL. That is uretprobe, added in Linux 6.11 so the kernel can call into userspace and come back — and it is, by coincidence, the last row of the most-linked syscall table on the web, which was generated from Linux 4.7 in 2016.
The call that happened more than once
A signal arriving during a blocking call is the last way the answer can be other than it looks. The rule is well known — the call fails with EINTR unless the handler was installed with SA_RESTART — and the rule is not the whole rule. An interval timer at 40 ms firing into a 250 ms operation, with SA_RESTART set throughout:
read() on a pipe (data at 250 ms) ret=1 ok after 250.3 ms, 6 signals
poll(), 250 ms timeout ret=-1 EINTR after 40.1 ms, 1 signal
select(), 250 ms timeout ret=-1 EINTR after 40.1 ms, 1 signal
nanosleep(), 250 ms ret=-1 EINTR after 40.2 ms, 1 signal
read() on socket w/ SO_RCVTIMEO 250ms ret=-1 EINTR after 40.1 ms, 1 signalFour families never restart no matter what you set: anything multiplexing (poll, select, epoll_wait, ppoll), anything sleeping (nanosleep, clock_nanosleep), the signal-waiting family (pause, sigsuspend, sigtimedwait), and any socket call on a socket with SO_RCVTIMEO or SO_SNDTIMEO set. That last one is the expensive one, because it is conditional on a socket option rather than on the call: setting a receive timeout silently converts a restarting call into a non-restarting one, and nothing warns you. A daemon that worked for a year until it received its first SIGWINCH is usually this.
The top row is the interesting one: the read() restarted, repeatedly, and succeeded, and to the program one call returned one byte. Here is a second run of the same idea under a tracer, with the timer firing three times — and it is the trace this page opened with:
$ ./erestart
read returned 1
$ strace -f -e trace=read,rt_sigreturn -e signal=SIGALRM ./erestart
read(3, 0x7ffcb96d1c0f, 1) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---
rt_sigreturn({mask=[]}) = 0
… twice more, identically …
read(3, "x", 1) = 1Four read() lines for one call, and the three that “return” ERESTARTSYS return a value that exists in no header and that your program can never observe. It is one of three internal restart codes — ERESTARTSYS, ERESTARTNOINTR and ERESTART_RESTARTBLOCK — and the third is why restart_syscall turns up in traces as a call nobody wrote. signal(7) documents the rule impeccably and names none of the three, so a reader who has just met one in a trace has nothing to look up. The manual documents the rule and not the artefact the rule produces, and the artefact is what people search for.
And the fourth name
libc turned the number into −1, set errno, and your code printed something. That printed line is the fourth name the operation has had, and it is the only one most people ever see. One detail worth holding, because it explains three separate pieces of folklore: errno is not a variable. It is a macro expanding to (*__errno_location()) — a function call returning a pointer into thread-local storage. That is why it works correctly in threads, why it cannot be carried across a fork, and why a signal handler that calls almost anything can destroy it under you.
Which is where the page’s argument lands. Your log says Operation not permitted. That string came from strerror, which was given a number by libc, which read it out of a register, which was written by something. The string is four translations away from the event, and it carries no information at all about which of the four parties produced it. The kernel refusing you, a seccomp filter answering on the kernel’s behalf, and a dispatch handler inventing an answer all arrive at your program in the same words.
Stage 7: What the answer becomes
The reader’s problem does not end when the return value arrives. It ends when they understand why the second call behaved differently from the first, and why the same binary behaves differently on two hosts. So there is a stage after the answer, and it is the stage almost every account of this boundary is missing. A syscall’s answer does not evaporate. It becomes durable state, and most of that state is one-way:
- A feature probe that runs once and is never revisited. glibc calls
clone3, and what it learns from the answer it keeps for the life of the process. So does every runtime that probesopenat2,io_uring_setuporstatx. The first call decides; every later call inherits the decision. Which is why “it fails only after it has been up a while” is almost never this boundary, and “it fails identically from the first second” almost always is. - A filter that can never be removed; a bit that can never be cleared —
no_new_privs, set once, with no way back for that process or any of its children; and an audit record that will be read months later by somebody who has no idea which of the four parties answered.
Which produces the sentence this page exists to make usable. A pinned image digest does not pin the system calls available to your program. It pins the image. The seccomp profile, the kernel’s syscall table, the vDSO’s symbol list, the mitigation set and the value of kernel.io_uring_disabled — a sysctl since Linux 6.6, October 2023, and off by default in Docker 25.0.0 and later — all belong to the host. A kernel upgrade can add a call your C library will now start probing for; a library upgrade can start probing for a call your filter has never heard of. Containers, All the Way Down makes the container half of this argument. The syscall half is why “worked yesterday, nothing in the changelog” is a sentence people are still saying.
A worked diagnosis: the service that stopped creating threads
A service that has run for a year starts failing after a routine base-image bump. It logs fatal: Operation not permitted and exits, or dies somewhere in a thread pool with pthread_create returning EPERM.
Every obvious observable says nothing is wrong. The digest is pinned and unchanged, ulimit and memory are fine, and CapEff is byte-identical to the old host. The kernel is newer, not older, and the call demonstrably exists on it. And “Operation not permitted” sends every person who reads it to file ownership in a mounted volume, where there is nothing to find.
The hinge routes it in one command:
$ strace -n -f ./svc 2>&1 | grep -E 'clone|ENOSYS|EPERM'
[ 435] clone3({flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|…}, 88)
= -1 EPERM (Operation not permitted)A number crossed. It came back with an errno that has nothing to do with permissions. That is the hinge’s third branch — stage 4, a filter answered — and it is correct, and it is fast, and it points past the cause.
Because the filter is right. Blocking clone3 may be exactly what the policy intends, and the person who blocked it was not making a mistake. The cause is the word the filter chose to refuse with — the block in stage 4, where the same filter blocking the same call kills the program under EPERM and goes unnoticed under ENOSYS. glibc 2.34, August 2021 began using clone3 in pthread_create, with a fallback to clone that fires on ENOSYS and on nothing else.
The fix is not the obvious one, and it is not the second-obvious one either. --security-opt seccomp=unconfined proves the diagnosis and is not a remedy. Allowing clone3 is also not a remedy, because the policy may be right to block it. The remedy is to change the word — "defaultErrnoRet": 38 in the profile, which is ENOSYS. Refuse the call, but refuse it in a word the ABI defines. Docker’s own default profile made exactly that change in 20.10.10, October 2021. Hand-written and vendored profiles still carry the old default today, which is why this is a live diagnosis rather than a war story.
And note what the hinge did and did not do. It saved you the afternoon in the volume’s file ownership, which is worth the whole page. What it could not tell you is whether the filter was wrong to refuse or wrong about how — and that second question has exactly one place to look. Containers, All the Way Down works a diagnosis with the same first move and a different ending: there the sandbox blocks a call outright and the fix is to permit it. Here the sandbox is right to block and wrong about the word, and the fix is a single number in a JSON file. Same trace, different bug.
The moral, and it is the page in one line: an errno is not a diagnosis. It is a sentence that one of four parties chose to say, and only one of them was the kernel.
Symptoms, and which stage owns them
Numbered by the seven stages above, not by any other scheme.
| What you see | Stage | What to run, or what it is |
|---|---|---|
| The program is clearly working and the trace is nearly empty | 2 | count the calls your code makes, then count strace -c‘s lines; the difference went through the vDSO |
strace names a call you never wrote | 1 | the wrapper renamed it; you are reading the trace correctly |
EPERM from a call that has nothing to do with permissions | 4 | grep Seccomp /proc/PID/status, then read the profile’s defaultErrnoRet |
ENOSYS from a call your headers name | 6 | the header is not the kernel; twelve dead names on this machine |
| More trace lines than your program made calls | 6 | ERESTARTSYS — a signal is arriving mid-call |
| A daemon that breaks the first time it receives a signal | 6 | SA_RESTART is set and does not apply; look for SO_RCVTIMEO |
Latency degrades sharply and the trace fills with clock_gettime | 2 | cat /sys/devices/system/clocksource/clocksource0/current_clocksource |
| Tracing makes the program unusable | 5 | strace --seccomp-bpf — 222× becomes 1.14× |
| io_uring quietly unavailable in production and fine locally | 7 | cat /proc/sys/kernel/io_uring_disabled |
| “Worked yesterday”, digest unchanged, nothing in the changelog | 7 | the host owns the syscall table, the profile, the vDSO and the mitigations |
How to tell whether a page about system calls is worth reading
Everything this page has corrected comes from one model, and naming it is more useful than the list. The corpus treats a system call as a function call into a library that happens to live in the kernel — so it believes the name is the thing, the return is either success or failure, the call happens exactly once, and everything between the call site and the kernel is transparent plumbing. Watch it generate the rest:
- If the name is the thing,
straceprintingopenatwhen you wroteopen()is a quirk of the tracer rather than the wrapper choosing a different call — so pages explain it as a strace behaviour, and send readers looking in the wrong place forever after. - If it is a function call, its number is an implementation detail beneath notice — so filter examples get written without checking
arch, and 89% of shared numbers mean something else to a 32-bit caller. - If the return is success-or-failure,
ENOSYSis just another failure — so a policy author picksEPERMas the safe default and breaks every C library’s feature probe. - If a function call always happens, the vDSO is an optimisation rather than a different answerer — so nobody writes down that a vDSO call is invisible to every kernel-side tool, and readers conclude from an empty trace that their program is idle.
The one-sentence test. Any page that tells you what a system call costs without naming the CPU and quoting /sys/devices/system/cpu/vulnerabilities/meltdown is repeating a number somebody measured in 2018, on hardware you are not using. The cost of this boundary is its most-quoted fact, and between 2018 and now the reason for the cost changed completely while the number stayed in circulation.
Then scan for tokens. getpid() offered as the example of libc caching expired in February 2017. “The vDSO provides four functions” has been incomplete since September 2024. ltrace recommended dates a page to before 2013, its last release. “Keep the seccomp profile small for performance” and “strace -c is the lightweight option” are both measurably false. io_uring recommended with no mention of io_uring_disabled expired in October 2023. And a register table that says rcx rather than r10 was copied from something written before x86-64. Three hits and the page predates the machine you are sitting at.
What the silences prove
- A page about this boundary that never mentions
ENOSYShas never debugged a program inside a container. It is describing the ABI as documentation rather than as a negotiation that happens at runtime. - A page about the vDSO that never says a vDSO call cannot be traced is describing a feature rather than a mechanism. It has been read about, not used.
- A page about tracing overhead with no
--seccomp-bpfin it was written before 2019 or never opened the manual page. Either way its numbers are the wrong numbers. - A page about the return convention that never gives the number 4095 has read
errno.hand never read a C library.
A page that has visibly fixed itself, and the half it did not absorb
syscalls(2) has absorbed exactly half of the ENOSYS story, and the half it absorbed is the easy one: it now names the reserved-but-unimplemented slots outright, saying that although slots are reserved for them in the table, calls such as afs_syscall, break and ftime are not implemented in the standard kernel. That is a real correction, and most pages on the subject have not made it.
It did not absorb the other direction, which is now the common one: numbers the kernel implements that the headers cannot name. Eight of them on the machine this page was written on. And it publishes a static list of holes on a page documenting the calls available as at Linux 5.14, August 2021 — for a boundary whose entire character is that a kernel must be asked at runtime rather than looked up. Its own text says syscall(2) is there “to invoke system calls for which no wrapper function is provided”, and it never tells you which those are. That is a page which learned that the syscall table can lie, and has not yet learned that its own list is one of the liars.
What to hold on to
- Ask whether a number crossed, and who answered it — not what failed.
strace -nis the command, and the five readings in the box at the top are the whole method. ENOSYSis the only version number this ABI has. A filter that answers anything else to a feature probe is not refusing a call; it is lying about the kernel, and the program it breaks will look like it broke itself.- An empty trace is not an idle program, and a number you read somewhere is not a number about your machine. Count your own calls and compare; read
/sys/devices/system/cpu/vulnerabilities/,current_clocksourceand/proc/sys/kernel/io_uring_disabledbefore believing anything about cost or availability; and trace with--seccomp-bpf, which turns 222× into 1.14× and has been there since 2019.
And the sentence to keep: an errno is not a diagnosis. It is a sentence one of four parties chose to say, and only one of them was the kernel.
Related reading
- What Actually Happens When You Run a Program —
fork,execve, the dynamic linker, and where the syscalls at the top of every trace come from. - Containers, All the Way Down — namespaces, cgroups, capabilities, and seccomp as a container policy rather than as a mechanism.