Linux is a kernel. Everything else on your machine — the shell, ls, systemd, your desktop — is separate software that a distribution assembles around it. That is not pedantry about naming; it is the single most useful fact for working out which half of your system is misbehaving.
The boundary
Your programs run in userspace. They cannot touch a disk, open a socket, or read a keystroke directly — the CPU physically prevents it. When a program needs anything outside its own memory, it makes a system call: it asks the kernel, the CPU switches into kernel mode, the kernel does the work and hands back a result.
That is the whole architecture. The kernel owns the hardware and arbitrates access to it; everything else queues up politely and asks. Around 350 system calls cover it all — open, read, write, fork, execve, mmap, socket, and a long tail.
The diagnostic: watch a program ask
strace prints every system call a process makes. It turns “it does not work and says nothing useful” into a list of exactly what the program tried to touch.
strace -c ls # summary: which calls, how many, how long
strace -f -e trace=openat,stat myprogram 2>&1 | grep -i config
strace -f -p 1234 # attach to something already running
strace -e trace=network curl -s https://example.com >/dev/nullThe most common real use: a program says it cannot find its configuration, and you want to know where it looked. Filter for openat and the answer is on screen in a second. The second most common: a permission error where the message does not say which file — EACCES in the strace output names it.
Attaching to another user’s process needs root, and on some systems ptrace is restricted even then — sysctl kernel.yama.ptrace_scope tells you.
What the kernel is responsible for
| Job | What that means in practice |
|---|---|
| Scheduling | Deciding which process gets the CPU next. Why load average is not CPU percentage. |
| Memory | Virtual addresses, page tables, the page cache, swapping, the OOM killer. |
| Filesystems | The VFS layer, so ext4, xfs, btrfs and NFS all answer the same calls. |
| Networking | The entire TCP/IP stack, routing, firewalling (netfilter). |
| Devices | Drivers. Every piece of hardware you own is talked to by kernel code. |
| Permissions | UID and GID checks, capabilities, SELinux and AppArmor hooks. |
| Namespaces and cgroups | The isolation and resource limits that containers are built out of. |
Notice what is not on that list: starting services, formatting a disk, resolving a hostname, running a shell. Those are userspace. If your service fails to start, the kernel is almost certainly not the problem — see systemctl.
Modules: the kernel changes shape at runtime
Linux is monolithic — drivers run inside the kernel rather than as separate processes — but it is modular: most drivers are separate .ko files loaded on demand when hardware appears. That is why a generic install disc boots on hardware nobody tested it on.
lsmod # what is loaded, and what depends on it
modinfo i915 # description, parameters, firmware needed
sudo modprobe vboxdrv # load one, with its dependencies
sudo modprobe -r pcspkr # unload
cat /proc/cmdline # what the bootloader passed to the kernelOptions go in /etc/modprobe.d/*.conf — options lines to set a parameter, blacklist to stop a module loading automatically. Blacklisting is the standard fix when the open-source driver for a device fights with a proprietary one.
Firmware is not a module. Many devices need a binary blob uploaded to them at initialisation, shipped in packages like linux-firmware. A wifi card that is detected but never works is usually missing firmware, and dmesg says so plainly.
Reading what the kernel says
sudo dmesg -T # the ring buffer, with human timestamps
sudo dmesg -T -l err,warn # only the parts worth reading
sudo dmesg -w # follow, then plug the device in
journalctl -k -b # kernel messages, this boot
journalctl -k -b -1 # kernel messages from the boot that crasheddmesg -w while attaching hardware is the fastest way to find out whether the machine saw the device at all, and what it decided to call it. Disk errors, OOM kills, filesystem remounts to read-only and segfaults all land here. The buffer is finite and wraps, which is why journalctl -k matters — it keeps the history across boots.
Talking back: /proc, /sys and sysctl
/proc and /sys are not directories of files. They are the kernel exposing its own state as if it were a filesystem — reading a file runs kernel code that generates the answer at that instant.
cat /proc/cpuinfo /proc/meminfo # what ps, top and free are reading
ls /proc/1234/fd # every file that process has open
cat /proc/1234/cgroup # which cgroup it is in
sysctl -a | grep somaxconn # every tunable and its current value
sudo sysctl -w net.ipv4.ip_forward=1 # change one now (lost at reboot)To make a sysctl change survive, write it to a file in /etc/sysctl.d/ ending in .conf and apply it with sudo sysctl --system. The two settings people most often need: net.ipv4.ip_forward for anything routing traffic, and vm.swappiness when a machine swaps more eagerly than you want.
The misreading: “my kernel version is ancient”
An enterprise distribution’s kernel version number tells you almost nothing about how old its code is. Red Hat and SUSE take a kernel, freeze the version, and then backport security fixes and drivers into it for a decade. A RHEL kernel calling itself 5.14 contains patches written years after 5.14 was released. Comparing that number against the latest mainline release, and concluding you are unpatched, is simply wrong — check your distribution’s advisories instead.
uname -r # the running kernel
uname -a # plus build date and architecture
cat /proc/versionThe number after the dash is the distribution’s own build — that is the one that increments when they ship fixes.
There is one case where the version genuinely matters: hardware newer than the kernel. Drivers for a laptop released this year may simply not exist in a kernel frozen two years ago, and no backport is coming. That is what Ubuntu’s HWE kernels are for, and one of the honest arguments for running Fedora on a desktop while your servers run something conservative.
When the kernel itself fails
Two words worth telling apart. An oops is an internal error the kernel survives — something is now broken, the machine limps on, and you should reboot at your convenience. A panic is unrecoverable and the machine stops.
Either way the useful evidence is the first few lines, not the hexadecimal stack trace: the message text and the module name in brackets. journalctl -k -b -1 after a reboot usually has it, if the machine got as far as writing to disk. A machine that panics reproducibly on one kernel and not on the previous one is telling you exactly what to do — boot the older entry from the boot menu and pin it until there is a fix.
Most “kernel” problems are not. Out of memory, a full disk, and a filesystem remounted read-only after I/O errors all produce alarming kernel messages while being ordinary operational faults — see Processes and Memory and Disk Space.
Related
- How Linux Boots — how the kernel gets loaded, and what an initramfs is for
- Processes and Memory — what the scheduler and memory manager are doing to your numbers
- Filesystems — the layer the VFS presents to everything above it
- systemctl — the userspace half, where most problems actually live