Two things about Linux memory confuse people in opposite directions. The first is that a healthy machine reports almost all its memory as used, which looks alarming and is not. The second is that a process can vanish without warning, leaving nothing in its own log, which looks like a crash and is actually the kernel making a decision.
Both come from the same design: the kernel treats free memory as wasted memory, and it will promise more than it has.
Reading the numbers
free -h total used free shared buff/cache available
Mem: 15Gi 4.2Gi 412Mi 310Mi 11Gi 10Gi
Swap: 2.0Gi 0B 2.0Gifree is memory doing nothing at all. On a machine that has been up for a week it is always small, and that is correct. Anything the kernel is not otherwise using gets filled with cached copies of files it has read — the page cache — because RAM sitting idle helps nobody.
available is the number that matters. It is the kernel’s estimate of how much a new process could get, counting the cache it would evict to provide it. Watch that column and ignore free entirely.
Cache is not a leak, and “clearing” it makes things worse. The advice to run echo 3 > /proc/sys/vm/drop_caches circulates endlessly and is almost always wrong outside benchmarking. Cached pages are already reclaimable — the kernel hands them back the instant a process needs the memory, without being asked. Dropping them does not free anything that was not already free; it just throws away work, and the machine slows down while it reads everything from disk again. The number to worry about is available, and cache is counted in it.
Why a process’s memory figure is misleading too
Per-process numbers have the same problem in miniature.
| Column | Means | Use it for |
|---|---|---|
| VSZ / VIRT | Address space reserved, most of it never touched | Almost nothing |
| RSS / RES | Physical pages currently resident | A rough answer |
| Shared | Pages also counted against other processes | Understanding double counting |
| PSS | Private pages plus a fair share of the shared ones | The honest per-process figure |
RSS double-counts: ten processes sharing one 50 MB library each report that 50 MB, so adding RSS across a machine gives a total far larger than the RAM installed. PSS divides shared pages between the processes using them, so the column does sum sensibly.
ps -eo pid,rss,comm --sort=-rss | head
grep -E '^(Rss|Pss):' /proc/1234/smaps_rollup
sudo smem -rs pss | head # if smem is installedOvercommit: the promise the kernel cannot keep
When a program asks for memory, the kernel grants the request without allocating anything. Physical pages appear only when the program first writes to them. This is why fork() is cheap, why a process can allocate 4 GB on a 1 GB machine, and why a successful allocation is not a guarantee.
It also means the machine can promise more than it owns. Usually fine — most programs never touch most of what they ask for. Occasionally the promises are all called in at once, and there is nothing left. At that point the kernel cannot fail the allocation politely, because the program is not asking for anything; it is writing to memory it was already told it had.
So something has to be killed.
The OOM killer
The kernel scores every process, roughly in proportion to how much memory it is using, and kills the highest scorer with an unblockable SIGKILL. The victim gets no chance to log anything, flush anything, or complain — which is why the first symptom is usually a service that is simply gone.
The evidence is in the kernel log, not the application’s:
sudo dmesg -T | grep -i -E 'out of memory|oom-kill'
sudo journalctl -k --since "2 hours ago" | grep -i oom
# systemd records it against the unit too
systemctl status myapp | head -20 # Main process exited... status=9/KILLA line reading Out of memory: Killed process 1234 (postgres) is conclusive. Exit status 9, or Killed with no other explanation, is the same story — signal 9 is SIGKILL, and the OOM killer is the usual source of one nobody sent by hand.
Note who dies. The biggest process is not always the guilty one: a leaking script that pushes a database over the edge usually gets the database killed, because the database is larger. You can bias the choice:
cat /proc/1234/oom_score # current score
echo -500 | sudo tee /proc/1234/oom_score_adj # make it less likely a target
# permanently, for a service
# [Service]
# OOMScoreAdjust=-500Protecting a process does not create memory. It only changes who dies instead, so use it to say which thing on the box matters most, not as a fix.
Swap, and what it is actually for
Swap is disk space the kernel can move rarely-used pages into, freeing RAM for things being used now. It is not “extra memory”, and it does not stop the OOM killer — it delays it.
The real value of a small amount of swap is that long-lived idle pages — a daemon’s startup code that will never run again — get moved out, leaving more RAM for page cache. A machine with a little swap and low swappiness often performs better than one with none.
swapon --show
cat /proc/sys/vm/swappiness # 60 by default; 10 suits a server
# per-process swap use
for p in /proc/[0-9]*; do
awk '/VmSwap/{print $2, FILENAME}' "$p/status" 2>/dev/null
done | sort -rn | headOne correction to the comment above, because it is the most repeated piece of stale advice in Linux tuning. vm.swappiness has had a range of 0 to 200 since Linux 5.8, and the documented meaning is now the relative I/O cost of swapping versus filesystem paging — not how eager the kernel is to swap. At 100 the two are treated as equally expensive; above 100 you are asserting that swap is cheaper, which is correct when swap is compressed memory rather than a disk. So on a machine using zram — the default on Fedora, among others — a value of 10 is actively wrong, and values well above 100 are the intended usage. The old “10 suits a server” rule only makes sense for swap on a spinning disk.
The failure mode to know is thrashing: memory is so tight that pages are written out and read back constantly, and the machine spends all its time on disk I/O. It responds to nothing, load average climbs into the dozens, and it never quite dies — which is worse than an OOM kill, because at least a kill ends it. zram, which compresses swap in RAM instead of writing to disk, avoids much of this and is a good default on small machines.
cgroups: limits with a smaller blast radius
A system-wide OOM kill is indiscriminate. Control groups let you cap a single service, so when it exceeds its limit only that service is affected — the rest of the machine never notices.
# in a systemd unit
[Service]
MemoryHigh=1G # throttle and reclaim hard above this
MemoryMax=1.5G # kill inside the cgroup above thissystemctl show myapp -p MemoryCurrent -p MemoryMax
systemd-cgtopThis is exactly what a container’s memory limit is, underneath — docker run -m 512m writes a cgroup limit. It is also why a container can be killed while the host has plenty of free RAM: the limit that was hit was the group’s, not the machine’s.
MemoryHigh is the more useful of the two in practice. It applies back-pressure rather than killing, so a service that briefly overshoots gets slowed down instead of destroyed.
Symptoms and what they mean
| Symptom | What is happening | Where to look |
|---|---|---|
free shows almost nothing free | Normal — page cache | The available column |
| Service disappears, nothing in its log | OOM kill | dmesg -T | grep -i oom |
Exit status 9 / Killed | SIGKILL, usually the OOM killer | Kernel log |
| Machine unresponsive, huge load, little CPU | Thrashing on swap | vmstat 1 — si/so columns |
| Container killed while host has free RAM | cgroup limit, not machine limit | MemoryMax, docker inspect |
| RSS across all processes exceeds installed RAM | Shared pages double-counted | PSS, via smaps_rollup |
| Allocation succeeded, crash later | Overcommit | vm.overcommit_memory |
| Memory use grows forever, never returns | An actual leak | RSS over time; the application |
The distinction worth holding on to is between the last row and everything above it. Most “memory problems” reported on Linux machines are misread numbers or a limit doing its job; genuine leaks show up as RSS climbing steadily over hours and never coming back down, which is a different shape from anything else here.
If you want the full picture — all eight stages between a program calling malloc and something being killed for it, including what your allocator does before the kernel is involved, why free is the wrong number to watch, huge pages and the database advice that has stopped being true, and the difference between background reclaim and the kind that stalls your own process — that is a separate, much longer article: Memory, All the Way Down: From malloc to the OOM Killer. It also lists the memory tuning advice that has quietly expired, including some repeated on this page.
Related reading
- Processes and memory — the address space these numbers describe
- Containers explained — cgroups from the other direction
- Processes: ps, top and kill — reading RSS in the wild
- journalctl — finding the kernel’s account of the kill
- Monitoring without a full stack — alerting before it gets this far