The Commands section tells you how. This one tells you why — the mechanisms underneath, explained so that the commands stop being incantations and the error messages start making sense.
Forty pages, all the same shape: explain the mechanism, show the diagnostic it unlocks, and correct the misreading people usually arrive with. Read them in any order.
The long ones. A few pages here go considerably deeper than the rest — they follow one thing through its entire path, layer by layer, and end with a worked diagnosis rather than a summary. They take half an hour rather than five minutes, and they are the ones to read when you want to stop guessing at a class of problem for good.
- From Power-On to a Login Prompt — the eight handoffs between the power button and a shell, and how to tell which one failed
- The Shell, in Depth — the eleven stages between pressing Enter and an exit status, and why quoting bugs are invisible
- Containers, All the Way Down — the nine stages between
docker runand a running process, and why there is no container - What Ships in Your Container Image — the seven stages between a build and an image nobody can account for any more, and why deleting a file does not unpublish it
- The Life of a Package — the eight stages between
apt installand working software, and why the two families do the same job in the opposite order - Memory, All the Way Down — the eight stages between
mallocand the OOM killer, and whyfreeis not the number to look at - Permissions and Privilege, Properly — the six independent layers between a process and a file, in the order the kernel checks them
- The Life of a Login — the seven stages between a password and a shell, and why having a shell proves less than you think
- The Life of a Write — the seven layers between
write()returning and the data actually existing - How Time Works on a Linux Machine — the seven stages between the clock nobody is keeping and somebody else’s tolerance, and why nothing on the machine writes down that the clock moved
- The Life of a System Call — the seven stages between the name you wrote and the answer your program believes, and why four different things can answer for the kernel
- The Life of a File Descriptor — the seven stages between
open()andclose(), and why closing something rarely closes it - How a Path Becomes a File — the six stages between a pathname and an open file, and why the same string can name two different things
- The Life of a Unix Socket — the seven stages between a socket with no name and a name with no socket, and why every tool on the machine repeats a path none of them has checked
- DNS, All the Way Down — the eight stages between
getaddrinfoand an address, and whydigcannot see the first three of them - The Life of a Packet — everything between your application’s
write()and the wire, and back again - What Happens When You Type a URL — the six stages between a keystroke and a response, and why most of them finish before a packet exists
- The Life of a Log Line — the seven stages between a program’s own buffer and the party who has to be convinced, and why every field in the record was written by somebody different
- The Life of a Scheduled Job — the eight stages between the line you wrote and the person who needed last night’s output, and why a job that ran and did nothing is green on every instrument the machine has
- The Life of a Block Device — the eight stages between the kernel noticing a disk and the other copy of it somewhere else, and why no name your disk has is both stable across a move and unique across a copy
There are nine more in Guides: the eight stages between a client sending COMMIT and data you could still restore a year later, the seven between kubectl apply and a running container, the six between the etcd member that answered a write and a restore that hands the data back on a rewound clock, the six between an ACME account and a certificate your server is actually serving, the seven between a security advisory and a machine whose patching you could actually vouch for, the six between a DNS change you made and the moment it is over, the seven between the Dockerfile you wrote and an image somebody else has to run, the seven between the archive you kept and a machine that works again, and the nine between an application calling sendmail and a person deciding they wanted it.
How things are stored and represented
- The Linux Filesystem — what every top-level directory is for, and where to look for the file you actually want
- How a Path Becomes a File — the long one. All six stages between the first character of a pathname and an open file: the root and working directory as references rather than remembered strings, why
PATH_MAXlimits names and not trees, the trailing slash that defeats bothO_NOFOLLOWandlstat, one budget of forty symlinks, why..is a live pointer rather than a text edit, a descriptor and a path disagreeing about the same directory at the same instant, whatrealpathand/proc/PID/fdcannot tell you — and why checking a path and then opening it has never been safe - Filesystems — ext4, xfs and btrfs, inodes, journals, and what “the disk is full” can mean when it is not
- The Life of a Write — the long one. The page cache and the writeback tunables that freeze large machines, what
fsyncandfdatasynceach actually promise, LVM and dm-crypt and md, the block layer, the drive’s own cache, and the two flags that reach the platter - What a Journal Does During a Crash — what survives a power cut, what does not, and why only
fsyncmakes a write durable - The Life of a Block Device — the long one. All eight stages between the kernel registering a disk and the other copy of that disk somewhere else: the two names nothing in userspace can influence, the two entirely different places a name is read from, one slot per name and the loser that gets no name at all, the single arbiter every consumer is downstream of, the
fsckaimed at a symlink rather than a filesystem, nine brokenfstabentries of which four get identical clean reports — and why “always use the UUID” answers half a trilemma and hands you the other half - How Packages and Dependencies Work — repositories, signing, sonames, and what mixing sources does to a system
- The Life of a Package — the long one. All eight stages between
apt installand working software, covering both families at once: how a candidate version is chosen, where the signature actually is and what it is not on, the two apt solvers and which one your machine has, the transaction test one family runs and the other does not, maintainer scripts and triggers, the package database that is the only honest record of what happened, what a half-configured system is — and the advice that expired on the first of February 2026 - Text Encodings and Locales — text is bytes, nothing records the encoding, and your locale quietly changes what commands do
What is running, and how it got there
- How Linux Boots — firmware to bootloader to kernel to systemd, and how to tell which stage failed
- From Power-On to a Login Prompt — the long one. All eight handoffs in detail: EFI boot entries, shim and Secure Boot, GRUB and the Boot Loader Specification, Unified Kernel Images, the initramfs and what built it, the switch to the real root, systemd’s ordering, and getty
- How Time Works on a Linux Machine — the long one, and the handoff the page above never mentions. All seven stages between a machine with no memory of the time and somebody else’s tolerance: the device that is supposed to know the time while the power is off and why this machine may not have one, four clocks that are separate kernel objects and the single capability that moves any of them, step against slew and the ceiling that turns an hour of error into half a day of it, the
synchronized: yesthat is a latch and goes on saying yes for eight hours and fifty-three minutes after the daemon dies, five ways of waiting of which exactly one is told the clock moved, a timestamp that is a receipt rather than a fact — and the tolerances you are judged against, not one of which is printed anywhere on the machine being judged - What the Kernel Actually Does — the userspace boundary, system calls, modules, and why an “old” kernel version usually is not
- Processes and Memory — what a process consists of, and why load average is not CPU percentage
- systemd Beyond Services — units, targets, sockets and mounts, and the difference between needing something and waiting for it
- The Life of a Log Line — the long one, and the record everything above leaves behind. All seven stages between a program’s own buffer and the party who has to be convinced: the buffer that decides whether a dying program’s last words exist at all, a transport that changes what half the fields mean without saying so, the stamping journald does on arrival and a rate limiter keyed on the unit rather than on the message, a store whose retention is set by whichever of four independent limits binds first, a reader whose filters match fields nobody was obliged to fill in, a relay that keeps the text and drops the metadata you filter on — and the one rule that makes any of it trustworthy stopping at the edge of the machine
The life of a running program
One level below the pages above: what the kernel does to your program between starting it and ending it. Between them these explain most startup and shutdown failures.
- What Happens When You Run a Program — PATH, fork, exec, the shebang and the dynamic linker, in the order they can fail
- The Life of a File Descriptor — the long one. All seven stages between
open()andclose(): the three levels and which flag lives at which, what2>&1actually compiles down to, whatforkcopies andexecfilters, passing a descriptor to a process that never forked from you, why “too many open files” usually names the wrong resource, and why a deleted file can still be eating your disk — plus the one kind of lock that dies on the first close rather than the last - The Life of a System Call — the long one, and the layer underneath the page above it. All seven stages between the name you wrote and the answer your program believes: four of five function names that never appear in a trace and the eighty-five calls your C library has no name for, fifteen of twenty-one calls answered where no tool on the machine can see them, why the fourth argument is in
r10, a filter that does not refuse a call but answers it — and the errno it chooses deciding whether working programs keep working, the error band from −1 to −4095 and the wrapper arithmetic it forces, a call the kernel quietly restarts more than once, and the answer that becomes a decision your process is stuck with for life - Memory, Swap and the OOM Killer — why
freelooks alarming, what overcommit promises, and who gets killed when it cannot be kept - Memory, All the Way Down — the long one. All eight stages between
mallocand the OOM killer: glibc’s arenas and the threshold that raises itself, overcommit, page faults and what RSS really counts, huge pages and the database advice that has stopped being true, the LRU lists and whatswappinessnow means, direct reclaim, zram and zswap and cgroup limits — and why killing is the last stage rather than the subject - Signals — how a process is asked to stop, which two requests it cannot refuse, and why containers take ten seconds to die
How you talk to it
- What a Shell Actually Is — which startup file runs when, and why your alias vanishes over SSH
- The Shell, in Depth — the long one. All eleven stages between pressing Enter and an exit status: parsing and why quoting is decided first, the expansion order, word splitting and
IFS, globbing, how the command is found, the process group it lands in, and every exemptionset -equietly makes - What a Terminal Actually Is — TTYs, ptys and job control, and why a dropped connection kills your job
How it talks to the world, and who it trusts
- Networking Explained — addresses, routes, ports and name resolution, in the order you should check them
- The Life of a Packet — the long one. Socket buffers, the route lookup that also picks your source address, the five netfilter hooks, conntrack, ARP, queueing, offloads, MTU, the receive path, and what containers change
- The Life of a Unix Socket — the long one, for the family that never goes near any of the above and for which much of it is false. All seven stages: what
bind()does twice and why nothing re-checks it afterwards, four kinds of name including the one you get by accident, why only the write bit lets anyone connect and why0755lets nobody, the abstract namespace and the container escape it caused, a socket you were handed rather than made, a peer identity the kernel fills in and never admits it cannot know, a queue ten datagrams deep — and the queue columns that can be fabricated zeros with exit status 0 - How a Name Is Actually Resolved — the layers between your program and a DNS server, and why
digcan disagree with everything else - DNS, All the Way Down — the long one. All eight stages between a program calling
getaddrinfoand an address coming back: the address sorting that happens before DNS is involved at all,nsswitch.confand what[!UNAVAIL=return]really means, systemd-resolved’s two listeners and three modes,ndotsand the search list, why TCP port 53 is now mandatory, recursion and negative caching, DNSSEC and the root key that changes this October, and the four independent caches that make a fix look like it did not take - What a TLS Certificate Actually Proves — the chain of trust, and why a site can work in a browser and fail from curl
- What Happens When You Type a URL — the long one. All six stages between a keystroke and a response: the fetch that may have already happened before you pressed Enter, a parser that deliberately disagrees with RFC 3986, the four separate mechanisms that upgrade
http://before any packet exists, connection racing across protocols rather than address families, TLS 1.3 and a revocation system that was replaced rather than adjusted, the dozen stores an answer can come from — and what a browser actually is on a Linux machine, sandbox and all
Isolation and access
- Users, Groups and Root — UIDs, why most accounts are not people, and the memberships that quietly hand out power
- The Life of a Login — the long one. All seven stages between a password and a shell: the three binaries
sshdhas become, PAM’s control flags and why the bracketed ones are the real syntax, thepasswdside ofnsswitch.conf, which hash your machine actually uses, theaccountstack and the nologin file that is not the one people name, session classes and the one sentence that explains every brokensystemctl --user, wherePATHreally comes from, and what survives your logout - Permissions and Privilege, Properly — the long one. Credentials, path traversal, the nine bits and the rule that stops at the first match, setuid and sticky, ACLs and the mask that quietly eats them, capabilities, SELinux and AppArmor, and user namespaces — all six layers in the order the kernel checks them
- SSH Keys — key pairs, host keys, the agent, and why the changed-host-key warning matters
- Containers — namespaces, cgroups and overlay filesystems, and why a container is not a small virtual machine
- Containers, All the Way Down — the long one. All nine stages between
docker runand a running process: the registry request, the manifest and what an image digest really is, layers and whiteouts, the overlay mount, the OCI bundle and the runtime, the eight namespaces, the cgroup,pivot_rootand seccomp and capabilities, and why your program ends up as PID 1 - What Ships in Your Container Image — the long one, and the page above it read backwards. All seven stages between the build that made an image and the moment nobody can account for it any more: the build context and the version-control history that rides in with it, the layer a deleted secret is still published in and the config blob no squash can reach, the tag anybody with write access can re-aim, the three digests one image has and the copy routes that silently change them, the signature and SBOM that are separate objects nothing is obliged to check, blob custody that is scoped to a repository rather than to the digest — and why you cannot build the same image twice
If you keep hitting one of these
| The thing that keeps happening | Read |
|---|---|
| A service disappears with nothing in its own log | Memory, Swap and the OOM Killer |
Something was killed and there is nothing in dmesg | Memory, All the Way Down |
free shows plenty available and the machine still stalls | Memory, All the Way Down |
| The page cache is evicted while swap sits unused | Memory, All the Way Down |
| A threaded service has an RSS nobody can explain | Memory, All the Way Down |
| RAM is in use and no process owns it | Memory, All the Way Down |
| Everything stalls intermittently with no process at fault | Memory, All the Way Down |
| Load average is high but the CPU looks idle | Processes and Memory |
| A script will not run, and the error names a file that exists | What Happens When You Run a Program |
error while loading shared libraries | What Happens When You Run a Program |
| “Too many open files”, and the limit is already enormous | The Life of a File Descriptor |
ulimit -n and the limit the service actually has disagree | The Life of a File Descriptor |
| You deleted a large log and the disk did not get any emptier | The Life of a File Descriptor |
| A file lock disappeared and nothing reported an error | The Life of a File Descriptor |
epoll keeps firing for a socket you closed | The Life of a File Descriptor |
| A child process inherited a file it should never have seen | The Life of a File Descriptor |
strace names a system call your program never made | The Life of a System Call |
| A program is obviously busy and its trace is nearly empty | The Life of a System Call |
| A daemon breaks the first time it receives a signal | The Life of a System Call |
ls lists the names and ls -l shows a row of question marks | How a Path Becomes a File |
O_NOFOLLOW or lstat did not stop a symlink | How a Path Becomes a File |
$PWD and pwd -P disagree, and every child sides with pwd -P | How a Path Becomes a File |
| A backup is far too large, or contains files from outside the tree | How a Path Becomes a File |
| A service insists a config lacks a value you can plainly read | How a Path Becomes a File |
| Two parts of one process disagree about a file’s contents | How a Path Becomes a File |
| Stopping something takes exactly ninety seconds | Signals |
| Ctrl+C does nothing, or a container will not stop | Signals |
| A file is empty or truncated after a power cut | What a Journal Does During a Crash |
| A filename with a space in it becomes two arguments | The Shell, in Depth |
| A script carries on after an obvious failure | The Shell, in Depth |
A variable set inside a while loop is empty afterwards | The Shell, in Depth |
A #!/bin/sh script works on Fedora and fails on Ubuntu | The Shell, in Depth |
| “Command not found” for something you can see is installed | The Shell, in Depth |
scp or rsync to a host fails for no visible reason | The Shell, in Depth |
| Writes are impossibly fast, then the machine freezes for a minute | The Life of a Write |
| Data is lost on a power cut although the write succeeded | The Life of a Write |
| A config file is empty, or a new file is missing, after a crash | The Life of a Write |
| An NVMe drive performs like a SATA one | The Life of a Write |
| Latency spikes whenever files are deleted | The Life of a Write |
| A benchmark gives a different answer the second time you run it | The Life of a Write |
Everything is stuck in D state and the CPU is idle | The Life of a Write |
| Random new connections fail while existing ones are fine | The Life of a Packet |
| Small requests work and large ones hang forever | The Life of a Packet |
| High system-interrupt CPU with no process responsible | The Life of a Packet |
| A firewall rule has no effect on container traffic | The Life of a Packet |
| Traffic leaves from the wrong address | The Life of a Packet |
| Every tool lists a socket file that clients cannot reach | The Life of a Unix Socket |
| Permission denied on a socket whose mode looks perfectly sensible | The Life of a Unix Socket |
Address already in use, and nothing is listening | The Life of a Unix Socket |
| A daemon quietly stopped being socket-activated | The Life of a Unix Socket |
| A service starts trusting the wrong caller once it is containerised | The Life of a Unix Socket |
| A service is intermittently slow, drops nothing and logs nothing | The Life of a Unix Socket |
ss -x shows an empty queue on a service that is clearly stalled | The Life of a Unix Socket |
| A service works by hand and fails at boot | systemd Beyond Services |
| A long job dies when your connection drops | What a Terminal Actually Is |
| A name will not resolve at all | How a Name Is Resolved |
dig works but the application cannot connect | DNS, All the Way Down |
| Every connection stalls for seconds before it succeeds | DNS, All the Way Down |
| A name resolves on the host and not inside the container | DNS, All the Way Down |
| A DNS record was changed hours ago and still has not taken effect | DNS, All the Way Down |
| SERVFAIL on an internal name while connected to the VPN | DNS, All the Way Down |
| Small lookups work and large DNS answers fail | DNS, All the Way Down |
| An Alpine container resolves names differently from a Debian one | DNS, All the Way Down |
| It works in a browser and fails from curl | What a TLS Certificate Proves |
| It works from curl and fails in the browser | What Happens When You Type a URL |
| You installed a corporate root and the browser still refuses it | What Happens When You Type a URL |
| A certificate error that only one browser shows | What Happens When You Type a URL |
http:// becomes https:// and you cannot tell what did it | What Happens When You Type a URL |
| A page loads with no network activity at all | What Happens When You Type a URL |
| Two sites load the same CDN file and neither one is cached | What Happens When You Type a URL |
| HTTP/3 works from one client and not another | What Happens When You Type a URL |
| The same script sorts differently on two machines | Text Encodings and Locales |
| An upgrade wants to remove half the system | The Life of a Package |
| A third-party repository stopped working and nobody changed anything | The Life of a Package |
| You imported a repository key and the packages still are not verified | The Life of a Package |
| An install failed and you cannot tell whether anything was changed | The Life of a Package |
dpkg --configure -a is suggested everywhere and does nothing | The Life of a Package |
| A configuration file you edited was replaced, or quietly was not | The Life of a Package |
| Two package tools disagree about why something is installed | The Life of a Package |
| Permission denied on a file you appear to own | Permissions and Privilege |
| You added yourself to a group and are still denied | Permissions and Privilege |
A file is 644 and readable by nobody | Permissions and Privilege |
| An ACL is set and does not seem to apply | Permissions and Privilege |
| The modes are obviously correct and it still says permission denied | Permissions and Privilege |
| Container files belong to a user that does not exist | Permissions and Privilege |
systemctl --user works on the console and fails over SSH | The Life of a Login |
| A PAM change you made has no effect on SSH logins | The Life of a Login |
sudo -i gives a shell with no XDG_RUNTIME_DIR | The Life of a Login |
| An account is locked out and nothing is configured to lock it | The Life of a Login |
| Background processes are killed at logout on one machine and not another | The Life of a Login |
PATH is different over SSH than it is on the console | The Life of a Login |
The disk is full but du disagrees | Filesystems |
| A container behaves differently from your machine | Containers |
| The same pinned image works on the old host and fails on the new one | Containers, All the Way Down |
| “Operation not permitted” from a program that is otherwise running | The Life of a System Call |
| A container is killed with exit code 137 | Containers, All the Way Down |
docker stop always takes exactly ten seconds | Containers, All the Way Down |
| An image stays enormous however many files you delete | Containers, All the Way Down |
| One container quietly consumes the whole machine | Containers, All the Way Down |
| A rootless container cannot set memory or CPU limits | Containers, All the Way Down |
A secret you deleted in a later RUN is still readable in the registry | What Ships in Your Container Image |
A credential is in the image and it is in no COPY you wrote | What Ships in Your Container Image |
| The same pinned digest gives different software on two machines | What Ships in Your Container Image |
| An image digest changed and nobody rebuilt or re-tagged anything | What Ships in Your Container Image |
cosign verify finds no signature and the image pulls perfectly | What Ships in Your Container Image |
ImagePullBackOff on a replaced node, weeks after a registry cleanup | What Ships in Your Container Image |
| Two builds of the same commit produce different images | What Ships in Your Container Image |
| Your settings load in one terminal and not another | What a Shell Actually Is |
| Hardware is detected but does not work | What the Kernel Does |
| SSH warns that the host key has changed | SSH Keys |
| The machine will not come back after a reboot | From Power-On to a Login Prompt |
| “No bootable device”, or it boots straight to firmware setup | From Power-On to a Login Prompt |
| A kernel parameter you set has no effect on Fedora or RHEL | From Power-On to a Login Prompt |
| It drops to an emergency shell asking for the root password | From Power-On to a Login Prompt |
| Boot stalls for exactly ninety seconds, then carries on | From Power-On to a Login Prompt |
| A third-party kernel module refuses to load with no error of its own | From Power-On to a Login Prompt |
| The clock is wrong immediately after a reboot | How Time Works on a Linux Machine |
timedatectl says the clock is synchronized and the time is still wrong | How Time Works on a Linux Machine |
| A certificate is refused as not yet valid and it is plainly valid | How Time Works on a Linux Machine |
| The clock was corrected hours ago and it is still wrong | How Time Works on a Linux Machine |
| Log timestamps from two machines cannot be lined up | How Time Works on a Linux Machine |
uptime -s and who -b disagree about when the machine booted | How Time Works on a Linux Machine |
| A service keeps printing the old timezone after you changed it | How Time Works on a Linux Machine |
make says “Clock skew detected. Your build may be incomplete” | How Time Works on a Linux Machine |
| A log line is in the journal and you cannot tell whether the program wrote it | The Life of a Log Line |
| A service was clearly failing and its log has a gap in the middle of the failure | The Life of a Log Line |
One machine’s log timestamps disagree with its own --since window | The Life of a Log Line |
A log message is shown as [N B blob data] | The Life of a Log Line |
journalctl -o json gives "MESSAGE": null | The Life of a Log Line |
journalctl -o json gives MESSAGE as a list of numbers | The Life of a Log Line |
journalctl -u NAME shows lines the service never wrote | The Life of a Log Line |
_COMM= never matches a daemon whose name is longer than fifteen characters | The Life of a Log Line |
| A log line is truncated at 48K, or 16K, or 8K, and nothing says so | The Life of a Log Line |
journalctl -p err is empty on a machine that is obviously failing | The Life of a Log Line |
The journal is persistent, /var/log/journal exists, and -b -1 still says no such boot | The Life of a Log Line |
--vacuum-time freed nothing, or took a whole boot | The Life of a Log Line |
| Logs are being shipped and the fields you filter on locally are not in the store | The Life of a Log Line |
| The central log store shows a login you can prove never happened | The Life of a Log Line |
Your logs say status=sent and the recipient never got it | The Life of an Outbound Email |
| Mail reaches some people at one company and is filed as spam for others | The Life of an Outbound Email |
| A bounce you never saw turns up in your relay provider’s dashboard | The Life of an Outbound Email |
| Your application sends as one address and the bounces come back to another | The Life of an Outbound Email |
dig shows your SPF record is correct and receivers report permerror | The Life of an Outbound Email |
| DKIM verifies for most receivers and fails for one network | The Life of an Outbound Email |
| A forwarded copy of your mail is rejected and the original was fine | The Life of an Outbound Email |
| A message sat in the queue for days and nothing told you | The Life of an Outbound Email |
| You switched relay providers and mail started failing DMARC | The Life of an Outbound Email |
| A recipient shows you a subject line you never sent | The Life of an Outbound Email |
| A scheduled job runs every night and nothing it was supposed to produce exists | The Life of a Scheduled Job |
systemctl --failed is clean and the backup has not run since March | The Life of a Scheduled Job |
A timer says active (waiting) and its service can never start | The Life of a Scheduled Job |
| A job runs correctly from cron and fails as a timer, or the reverse | The Life of a Scheduled Job |
OnFailure= is configured and has never fired, for either good reason | The Life of a Scheduled Job |
The NEXT column is - and nothing says why | The Life of a Scheduled Job |
| A daily job caught up once after a week of downtime | The Life of a Scheduled Job |
| A mount succeeded and the files are months out of date | The Life of a Block Device |
Two disks are attached and only one of them has a name under /dev/disk | The Life of a Block Device |
An fstab line reaches one disk at boot and a different one from a rescue shell | The Life of a Block Device |
findmnt --verify passes and the wrong filesystem is mounted | The Life of a Block Device |
| A cloned VM and the machine it was cloned from cannot both be attached to one host | The Life of a Block Device |
| A mount that was fine for a year started reaching the spare disk | The Life of a Block Device |
lsblk shows the disk and blkid shows nothing | The Life of a Block Device |
pvs warns about a duplicate PV and mount says nothing | The Life of a Block Device |