Processes and Memory: What the Numbers Actually Mean

Almost every performance number Linux shows you is misread the first few times. Load average looks like a percentage and is not. free -h reports almost no free memory on a healthy machine. top shows a process using more memory than the machine has. None of these are bugs, and understanding them turns “the server feels slow” into a specific answer.

What a process is

A process is a running program plus everything the kernel tracks about it: its memory, its open files, its user, its working directory, and a numeric ID. A program on disk is a file; a process is that file in motion. Run the same program three times and you have three processes.

Every process except the first has a parent, because the only way to create one is for an existing process to copy itself. That copy — fork() — produces a duplicate, which then usually replaces its own contents with a different program — exec(). Fork then exec is how everything on the system starts.

# The whole family tree
pstree -p

# Parent of a process
ps -o ppid= -p 1481

# PID 1 is systemd; it has no parent
ps -p 1

# The whole family tree
pstree -p

# Parent of a process
ps -o ppid= -p 1481

# PID 1 is systemd; it has no parent
ps -p 1

This structure explains two things that otherwise look strange. Killing a parent does not kill its children — they are adopted by PID 1 and keep running, which is why stopping a wrapper script sometimes leaves the real work behind. And a zombie process is simply one that has finished but whose parent has not yet collected its exit status; it uses no memory and no CPU. A handful of zombies is harmless. Thousands means a parent process has a bug, and the fix is to restart the parent.

Process states

The STAT column in ps aux is more useful than most people realise.

D is the one to watch for. A process stuck in uninterruptible sleep is waiting on the disk or the network and cannot be killed — not even with kill -9, because it is not in a state where it can receive a signal. Several processes in D at once usually means failing storage or a hung network mount, and it is the classic signature of a machine that feels frozen while showing low CPU use.

# Anything stuck in D state?
ps aux | awk '$8 ~ /D/ {print}'

# What is it waiting on?
sudo cat /proc/1481/stack

# Anything stuck in D state?
ps aux | awk '$8 ~ /D/ {print}'

# What is it waiting on?
sudo cat /proc/1481/stack

Signals: what kill really does

kill does not kill. It sends a signal, and what happens next is up to the process.

# Ask politely first - this is the default
kill 1481

# Force, only after TERM has failed
kill -9 1481

# Reload configuration without restarting
kill -HUP $(pidof nginx)

# Ask politely first - this is the default
kill 1481

# Force, only after TERM has failed
kill -9 1481

# Reload configuration without restarting
kill -HUP $(pidof nginx)

Reach for -9 last, not first. A process receiving TERM gets to flush its buffers, finish writing files and close database connections cleanly. KILL removes it mid-sentence, which is how you get corrupted files and stale lock files that block the next start. Give TERM a few seconds before escalating — inspecting processes covers finding the right PID in the first place.

Load average is not a percentage

uptime
#  14:32:01 up 12 days,  load average: 2.15, 1.80, 1.42

nproc        # how many cores you have

uptime
#  14:32:01 up 12 days,  load average: 2.15, 1.80, 1.42

nproc        # how many cores you have

Those three numbers are the average number of processes either running or waiting to run, over the last 1, 5 and 15 minutes. They are counts, not percentages, and they have no upper bound.

The only way to interpret them is against your core count. A load of 4.0 is a fully-busy four-core machine with nothing queuing — healthy. The same 4.0 on a single-core machine means four processes competing for one core, and everything is waiting.

Read the three numbers together, because the trend matters more than any one of them. 8.0, 4.0, 2.0 means something started recently and is getting worse. 2.0, 4.0, 8.0 means a spike that has passed. A brief high load is normal; a sustained one is a problem.

One Linux-specific quirk: unlike other Unix systems, Linux counts processes waiting on disk I/O in the load average, not just CPU. So a machine with load 15 and an idle CPU is not lying to you — it is telling you the bottleneck is storage. That combination is one of the most useful diagnostic signals available.

# Is the CPU busy, or is it waiting for disk?
top
# Look at the %wa figure on the CPU line. High wa = I/O bound.

# Per-device detail
iostat -x 2
sudo iotop -o

# Is the CPU busy, or is it waiting for disk?
top
# Look at the %wa figure on the CPU line. High wa = I/O bound.

# Per-device detail
iostat -x 2
sudo iotop -o

Why free -h looks alarming

free -h
#               total   used   free   shared  buff/cache   available
# Mem:           7.7Gi  2.1Gi  231Mi   410Mi       5.4Gi       4.9Gi

free -h
#               total   used   free   shared  buff/cache   available
# Mem:           7.7Gi  2.1Gi  231Mi   410Mi       5.4Gi       4.9Gi

231 MB free out of 7.7 GB looks like a machine about to fall over. It is not. Linux uses every spare byte of RAM as a disk cache, because unused memory is wasted memory — idle RAM does nothing, whereas cached file data makes everything faster. The moment a program needs that memory, the cache is dropped instantly.

Read the available column, and ignore free. Available is the kernel’s own estimate of what a new program could get, cache included. In the example above the machine has 4.9 GB to hand and is perfectly healthy.

Virtual versus resident

top shows two memory columns and they mean very different things. VIRT is address space the process has asked for, including memory it never touched and libraries shared with every other process. It can exceed physical RAM and routinely does. RES is what is actually in RAM right now.

RES is the number that matters, with one caveat: shared libraries are counted in full against every process using them, so adding up RES across all processes overcounts badly. For a real per-process figure, smem reports PSS, which divides shared memory fairly between the processes sharing it.

# Top ten memory consumers
ps aux --sort=-%mem | head -11

# A fairer accounting
smem -rs pss

# Detail for one process
cat /proc/1481/status | grep -i vm

# Top ten memory consumers
ps aux --sort=-%mem | head -11

# A fairer accounting
smem -rs pss

# Detail for one process
cat /proc/1481/status | grep -i vm

Swap, and why some is good

Swap is disk space used to hold memory pages that are not being touched. Its reputation is poor because heavy swapping is genuinely slow — disk is thousands of times slower than RAM — but a little swap in use is a sign of good housekeeping, not trouble.

A process that started at boot, did something once, and has been idle for three weeks does not deserve RAM. Moving it to swap frees memory for cache and for programs doing real work.

# How much swap, and is it being used?
swapon --show
free -h

# The real question: is it actively swapping RIGHT NOW?
vmstat 2 5
# Watch si and so. Steady non-zero numbers = thrashing.

# How eagerly the kernel swaps (0-100, default 60)
cat /proc/sys/vm/swappiness
sudo sysctl vm.swappiness=10

# How much swap, and is it being used?
swapon --show
free -h

# The real question: is it actively swapping RIGHT NOW?
vmstat 2 5
# Watch si and so. Steady non-zero numbers = thrashing.

# How eagerly the kernel swaps (0-100, default 60)
cat /proc/sys/vm/swappiness
sudo sysctl vm.swappiness=10

The distinction that matters: swap used is fine; swap activity is the problem. A gigabyte sitting in swap untouched costs nothing. Constant si/so traffic in vmstat means the machine is thrashing and needs more RAM or less running on it.

Lowering vm.swappiness to 10 on a database server is a common and reasonable tuning choice. Setting it to 0, or running with no swap at all, is usually a mistake — it removes the kernel’s ability to shed idle pages under pressure, so instead of slowing down slightly the machine goes straight to killing processes.

The OOM killer

When memory genuinely runs out and nothing can be reclaimed, the kernel picks a process and kills it. The alternative is the entire machine locking up, so this is the merciful option, but it is abrupt and it does not ask.

The symptom is distinctive and widely misdiagnosed: a service that disappeared with nothing in its own logs. No error, no stack trace, no shutdown message — because the process was never told. The evidence is in the kernel log instead.

# Did the OOM killer take something?
sudo journalctl -k | grep -i "killed process"
sudo dmesg -T | grep -i "out of memory"

# Which process is most likely to be chosen next?
# Higher score = more likely. Range -1000 to 1000.
cat /proc/1481/oom_score

# Protect a critical process
sudo sh -c 'echo -500 > /proc/1481/oom_score_adj'

# Did the OOM killer take something?
sudo journalctl -k | grep -i "killed process"
sudo dmesg -T | grep -i "out of memory"

# Which process is most likely to be chosen next?
# Higher score = more likely. Range -1000 to 1000.
cat /proc/1481/oom_score

# Protect a critical process
sudo sh -c 'echo -500 > /proc/1481/oom_score_adj'

Selection is roughly “whatever frees the most memory”, which means the database is a prime candidate on a machine where the database is the point. If a service keeps vanishing overnight, check for OOM kills before anything else — it explains a large share of unexplained restarts, and no amount of reading the application’s own log will reveal it.

A diagnostic order for “the server is slow”

1. Load against cores. uptime and nproc. Is anything actually queuing?
2. CPU or I/O? top — high %wa means storage, not processor.
3. Memory. free -h, reading the available column. Then vmstat 2 for swap activity.
4. Who is responsible? ps aux --sort=-%cpu | head and the same for -%mem.
5. Anything stuck? Processes in D state point at hardware or a hung mount.
6. Anything killed? Check for OOM kills in the kernel log.
7. Disk full? df -h and df -i. A full filesystem produces symptoms that look like anything.

Steps 1 to 3 take under a minute and rule out most of the possibility space. Resist the urge to start restarting services before you have done them — a restart destroys the evidence and often fixes the symptom just long enough for it to come back later.

Quick reference

Related reading

• Inspecting processes — ps, top and htop in practice
• Reading Linux logs — finding OOM kills and service failures
• The Linux filesystem — /proc, where all these numbers come from
• Checking disk space — step 7 of the diagnostic order
• systemctl — restarting what the OOM killer took
• Users, groups and root — who each process runs as, and why it matters