Sooner or later a Linux machine tells you No space left on device and something stops working. This page is the sequence for finding out what filled up and getting the space back — including the two cases where the obvious tools appear to lie to you.
The short version: df tells you which filesystem is full. du tells you what filled it. They answer different questions and are not interchangeable.
df: which filesystem is full
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 49G 46G 1.1G 98% /
tmpfs 3.9G 1.2M 3.9G 1% /dev/shm
/dev/sdb1 196G 28G 159G 15% /var/lib/docker-h means human-readable. Without it you get 1K blocks, which nobody wants. Useful variations:
df -h # everything, human units
df -h /var/log # just the filesystem holding this path
df -hT # include the filesystem type
df -i # inodes instead of bytes — read on
df -h --total # add a total rowThe column that matters is Use%, and the one people misread is Mounted on. A path being nearly full only matters for the filesystem it actually lives on — in the example above, filling /home would be a problem for /, but filling /var/lib/docker would not, because it is a separate disk.
The reserved blocks surprise
On ext4, roughly 5% of the filesystem is reserved for root by default. A disk showing 100% full to a normal user often still has gigabytes that only root can write. That is deliberate — it keeps a full disk from locking the administrator out. It also explains why Size minus Used rarely equals Avail.
Running out of inodes instead of bytes
This one wastes hours because df -h shows plenty of free space while writes still fail.
$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 3.2M 3.2M 0 100% /Every file consumes one inode regardless of size, and a filesystem is created with a fixed number of them. Millions of tiny files — a session directory, a mail queue, a cache that never expires — will exhaust inodes long before they exhaust space.
If df -h looks fine but you cannot create files, run df -i. To find the culprit directory:
for d in /*; do echo -n "$d "; find "$d" -xdev -type f 2>/dev/null | wc -l; done | sort -k2 -nYou cannot add inodes to an existing ext4 filesystem. The fix is deleting files, or recreating the filesystem with a higher inode count. See find for safely deleting large numbers of files.
du: what is taking the space
The single most useful invocation, drilling down one level at a time:
du -h --max-depth=1 /var | sort -hThat shows the size of each immediate subdirectory, largest last. Follow the biggest number down a level at a time and you will find the problem in three or four steps.
| Flag | What it does |
|---|---|
-h | Human-readable sizes |
-s | Summary only — one total per argument |
--max-depth=N | How many levels down to report |
-x | Stay on one filesystem — important on a machine with mounts |
-a | Include individual files, not just directories |
--apparent-size | Report file size rather than disk blocks used |
-c | Add a grand total |
Note sort -h rather than sort -n — it understands the K, M and G suffixes that du -h produces. Sorting those numerically without -h puts 9.9M above 2.0G.
du -sh /var/log # one directory's total
du -sh * | sort -h # everything here, ranked
du -h --max-depth=1 / -x | sort -h # top-level, ignoring other mounts
du -ah /var/log | sort -h | tail -20 # the 20 biggest individual filesAdd 2>/dev/null when running as a normal user to silence permission errors.
ncdu: the one to actually install
sudo apt install ncdu # Debian, Ubuntu
sudo dnf install ncdu # Fedora, RHEL, Rocky, Alma
sudo pacman -S ncdu # Arch
ncdu / # scan, then browse interactively
ncdu -x / # stay on one filesystemncdu scans once and gives you a sorted, browsable tree you navigate with the arrow keys. d deletes the highlighted item. It turns a ten-command investigation into thirty seconds of pressing Enter, and it is the first thing worth installing on a server you expect to maintain.
For a machine you cannot install anything on, ncdu -o scan.json / on one host and ncdu -f scan.json on another lets you analyse the results elsewhere.
When df and du disagree
df says the disk is 98% full. du -sh / adds up to far less. Neither is broken — and this is the second classic time-waster.
Deleted files still held open
When a process has a file open and someone deletes it, the directory entry disappears but the blocks are not freed until the process closes it or exits. du walks the directory tree, so it cannot see the file; df asks the filesystem, so it can.
This is overwhelmingly the cause when someone has just rm‘d a large log file that an application is still writing to.
sudo lsof +L1That lists open files with a link count of zero — deleted but still held. Restarting the owning process releases the space. See systemctl for restarting a service cleanly.
The lesson for next time: truncate a live log instead of deleting it.
sudo truncate -s 0 /var/log/huge.log # frees space immediately, keeps the handle validSomething is mounted over a directory
If a filesystem is mounted at /mnt/data, any files written to that path before the mount existed are still on the underlying disk, invisible and taking up space. Unmount and look underneath to check.
Sparse files
Virtual machine images and database files can report a large size while occupying far fewer blocks. du reports blocks by default and du --apparent-size reports the nominal size; a big gap between them means sparseness, not a bug.
The usual culprits
# systemd journal — frequently gigabytes
journalctl --disk-usage
sudo journalctl --vacuum-size=200M
# Package manager caches
sudo apt clean # Debian, Ubuntu
sudo dnf clean all # Fedora, RHEL
# Docker — often the single biggest win
docker system df
docker system prune -a # removes unused images: read the warning first
# Old kernels (Debian, Ubuntu)
sudo apt autoremove --purge
# Rotated and stale logs
sudo du -sh /var/log/* | sort -h | taildocker system prune -a deletes every image not backing a running container. On a build server that is exactly what you want; on a machine where images are the only copy of something, it is not. Read what it lists before confirming.
The sequence
df -h— which filesystem, and is it really the one you thinkdf -i— rule out inode exhaustion before anything elsedu -h --max-depth=1 /path -x | sort -h— walk down to the culprit, or usencdu -x- If the numbers do not add up,
sudo lsof +L1— deleted files held open - Clear the obvious caches: journal, package manager, Docker
- Fix the cause — log rotation, a retention policy — not just the symptom
Quick reference
df -h # free space per filesystem
df -i # free inodes — check this too
df -h /path # which filesystem holds this path
du -sh /path # total for one directory
du -h --max-depth=1 /path | sort -h # drill down one level
du -ah /path | sort -h | tail -20 # biggest individual files
ncdu -x / # interactive, best tool for the job
lsof +L1 # deleted files still held open
truncate -s 0 file.log # empty a live log safely
journalctl --vacuum-size=200M # shrink the systemd journalRelated commands
- find — locating and removing files by size or age.
- systemctl — restarting the service holding a deleted file.
- awk — filtering
dfoutput for alerting, e.g. everything above 80%. lsblk— shows the block devices and partitions behind the filesystems.logrotate— the actual fix for logs that grow without limit.