Something stopped working. The logs contain the answer, and they also contain forty thousand lines that have nothing to do with it. The skill worth learning is not “how to read logs” but how to cut the pile down to the few lines that matter, quickly, under pressure. This guide covers journalctl, the plain text log files that still exist alongside it, and a diagnosis method that works whether the failure is thirty seconds or three days old.
Two log systems, side by side
Modern distributions run systemd, which collects logs into a binary journal you query with journalctl. That covers the kernel, every systemd service, and anything writing to syslog. But plenty of software writes its own plain text files regardless, and those are not in the journal at all.
| Source | Where it goes | Read it with |
|---|---|---|
| systemd services, kernel, syslog | The journal | journalctl |
| Nginx and Apache | /var/log/nginx/, /var/log/apache2/ | tail, grep, awk |
| MySQL, PostgreSQL | /var/log/mysql/, /var/log/postgresql/ | tail, grep |
| Application logs | Wherever the app was told to write | tail, grep |
| Older Debian systems | /var/log/syslog, /var/log/auth.log | Either |
When you do not know where a service writes, ask the package: ls /var/log/ is usually enough, and sudo lsof -p PID | grep log will tell you which files a running process actually has open.
The journalctl commands worth memorising
Five options do almost all the work. Everything else is refinement.
| Option | Does |
|---|---|
-u NAME | That unit — and, with it, coredump records for the unit and anything PID 1 or another root daemon said about it |
-f | Follow, live, as new entries arrive |
-e | Jump to the end |
--since / --until | Restrict the time window |
-p | Only this priority and worse |
# Everything for one service, newest at the bottom
sudo journalctl -u nginx
# Jump straight to the end instead of paging through
sudo journalctl -u nginx -e
# Watch it live while you reproduce the problem
sudo journalctl -u nginx -f
# The last 100 lines only
sudo journalctl -u nginx -n 100
# Follow two services at once
sudo journalctl -u nginx -u php8.2-fpm -fGetting the unit name right matters. systemctl list-units --type=service lists what is loaded, and tab completion after -u works. See systemctl for managing the services themselves.
Filtering by time
This is the highest-value filter, because you almost always know roughly when the failure happened. journalctl accepts both absolute timestamps and plain English.
# Relative
sudo journalctl --since "10 minutes ago"
sudo journalctl --since "1 hour ago" -u nginx
sudo journalctl --since yesterday --until today
# Absolute
sudo journalctl --since "2026-08-27 14:00" --until "2026-08-27 14:30"
# Since the machine last booted
sudo journalctl -b
# The boot BEFORE this one, for diagnosing a crash
sudo journalctl -b -1
# List the boots the journal still has
sudo journalctl --list-bootsjournalctl -b -1 is the one to reach for after an unexplained reboot. It shows the previous boot’s log, right up to the last thing the machine managed to write before it went down.
Filtering by severity
Most entries carry a syslog priority from 0 to 7, and -p shows that level and everything more severe, which is what you want. PRIORITY is an ordinary field supplied by whatever sent the entry, though, and it is optional: an entry that arrives without one matches no -p level at all except debug, so it never appears in a filtered view.
| Level | Name | In practice |
|---|---|---|
| 0 | emerg | System is unusable |
| 1 | alert | Act immediately |
| 2 | crit | Hardware and critical failures |
| 3 | err | Start here |
| 4 | warning | Widen to here if err shows nothing |
| 5–7 | notice, info, debug | Normal chatter |
# Errors and worse, since boot
sudo journalctl -p err -b
# Errors in the last hour, one service
sudo journalctl -p err -u nginx --since "1 hour ago"
# Widen to warnings
sudo journalctl -p warning -bsudo journalctl -p err -b is the closest thing to a “what is wrong with this machine” command. On a healthy server it returns almost nothing — but so does a badly broken one whose applications log everything at info, record their severity inside a JSON message, or send entries carrying no PRIORITY at all. Read an empty result as nothing matched rather than as nothing wrong: The Life of a Log Line.
Other useful filters
# Kernel messages only (the dmesg equivalent, with timestamps that make sense)
sudo journalctl -k
# Everything from one executable, even if it is not a unit
sudo journalctl /usr/sbin/sshd
# One process ID
sudo journalctl _PID=1481
# One user
sudo journalctl _UID=1000
# Full untruncated lines, no pager
sudo journalctl -u nginx --no-pager -o cat
# Structured output, when you want to pipe it somewhere
sudo journalctl -u nginx -o json-prettyThe -o cat format strips the timestamp and hostname prefix, leaving only the message. It is the right format when you are piping into grep or awk and the prefix is just noise. It is not a way to see more of a long message — journalctl prints messages in full by default, and the option that shows long fields in full is --all (-a). Be careful pointing it at a terminal: -o cat writes control bytes from the message straight through, where the default format would show [33B blob data] instead.
Plain text log files
For everything not in the journal, the tools are older and simpler.
# Watch a log as it is written
sudo tail -f /var/log/nginx/error.log
# Watch several at once; tail labels each with a header
sudo tail -f /var/log/nginx/*.log
# Last 200 lines
sudo tail -n 200 /var/log/nginx/error.log
# Follow, but only show lines matching something
sudo tail -f /var/log/nginx/error.log | grep --line-buffered "upstream"--line-buffered matters in that last one. Without it, grep buffers its output when writing to a pipe and you get nothing for a long time and then a block of lines. With it, matches appear as they happen.
Rotated logs are the other thing to know about. logrotate renames yesterday’s file to error.log.1 and compresses older ones to error.log.2.gz. A plain grep across the directory silently skips the compressed ones:
# Searches only the uncompressed files
grep "error" /var/log/nginx/error.log*
# Searches compressed files too
zgrep "error" /var/log/nginx/error.log*
# Read one compressed file
zcat /var/log/nginx/error.log.3.gz | lessIf a search over “the last week” comes back suspiciously empty, rotation is the usual reason. Reach for zgrep by default and the problem never arises.
A method for diagnosing something that just broke
Under pressure it is tempting to scroll and hope. A fixed sequence is faster and it stops you missing the obvious.
- Ask what the service thinks.
systemctl status nginxshows the state, the exit code, and the last few journal lines in one screen. Often that is the whole answer. - Fix the time window. Decide when it last worked and when it did not. Everything outside that window is noise:
--since "30 minutes ago". - Look at errors across the whole machine first, not just the service.
sudo journalctl -p err --since "30 minutes ago". The cause is often somewhere else — a full disk, a failed mount, an OOM kill. If that comes back empty on a machine that is plainly broken, treat the emptiness as a result rather than an all-clear, and run the same window again without-p. - Then narrow to the unit.
sudo journalctl -u nginx --since "30 minutes ago", and read the first error, not the last. Later errors are usually consequences. - Reproduce it with a follow open.
journalctl -u nginx -fin one terminal, trigger the failure in another. Nothing beats seeing the line appear as you cause it. - Check the boring physical causes. Disk full and out of memory produce strange, unrelated-looking failures everywhere.
Step six deserves its own commands, because it explains a surprising share of “it just stopped working”:
# Disk full? Check inodes too, they run out separately
df -h
df -i
# Did the kernel kill something for using too much memory?
sudo journalctl -k --since today | grep -i "out of memory"
sudo dmesg -T | grep -i oom
# Did a service fail to start at boot?
systemctl --failedsystemctl --failed is a good habit on any machine you have just logged into. Checking disk space covers the df and du side, including the inode trap.
Reading web server logs
Access logs are structured enough to answer real questions with awk. The default combined format puts the client address in field 1, the request in fields 6 to 8, and the status code in field 9.
# How many of each status code?
awk '{print $9}' /var/log/nginx/access.log | sort | uniq -c | sort -rn
# Which URLs are 404ing?
awk '$9 == 404 {print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20
# Busiest client addresses
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20
# Requests per hour, to find a traffic spike
awk '{print substr($4, 2, 14)}' /var/log/nginx/access.log | uniq -cThe sort | uniq -c | sort -rn pattern is worth committing to muscle memory: count each distinct value, then order by count descending. It answers “what is happening most?” against any column of any log.
Errors go to a different file. Nginx’s error.log is where you find upstream timeouts, permission denials and configuration problems, and it is the one to watch when the site is broken rather than just busy. Setting up Nginx covers where those files live and how to change them.
Failed logins and access attempts
# Debian and Ubuntu
sudo grep "Failed password" /var/log/auth.log | tail -20
# Any systemd machine
sudo journalctl -u ssh --since today | grep -i "failed"
sudo journalctl _COMM=sshd --since "1 week ago" | grep -c "Failed password"
# Successful logins, which is the more important list
sudo journalctl _COMM=sshd | grep "Accepted"
# Who is logged in now, and who was recently
who
last -n 20A public-facing server will show thousands of failed password attempts. That is background noise from automated scanners, not a targeted attack, and the correct response is to make passwords irrelevant by using keys and disabling password authentication — covered in securing a new server. Two notes on the commands above. _COMM= matches the kernel’s short process name from /proc/PID/comm, which is capped at fifteen characters: it works for sshd, but the same idiom silently matches nothing for a daemon with a longer name, so prefer -u or _EXE= as a general recipe. And what deserves attention is an accepted login you cannot account for — before you act on one, check who wrote it. A convincing Accepted publickey for root line can be produced by any unprivileged local account, and in the default view it is byte-identical to a real one; journalctl -o verbose shows the fields journald stamped itself, which is what separates the claim from the finding: The Life of a Log Line.
Keeping the journal a sensible size
By default the journal is capped at 10% of the filesystem it lives on or 4 GiB, whichever is smaller — so on a small VPS the percentage is what bites, and above roughly a 40 GB filesystem it is a flat 4 GiB however large the disk gets.
# How much space is it using?
journalctl --disk-usage
# Trim to a size or an age
sudo journalctl --vacuum-size=500M
sudo journalctl --vacuum-time=30dVacuuming deletes whole archived journal files and never touches the active one, which makes it coarser than it looks: the same --vacuum-time can free nothing at all on one machine and a whole boot at once on another, depending on where the file boundaries happen to fall. To make a limit permanent, set SystemMaxUse=500M in /etc/systemd/journald.conf and run sudo systemctl restart systemd-journald. One more thing worth checking on a fresh machine: if /var/log/journal/ does not exist, the journal is in memory only and everything is lost on reboot — which makes diagnosing a crash impossible. sudo mkdir -p /var/log/journal is the first of two steps, though: journald goes on writing to /run until it is told to move across, and the flag it checks is /run/systemd/journal/flushed. Create the directory, then run sudo journalctl --flush or reboot, and check for that file rather than for the directory.
Quick reference
| You want | Command |
|---|---|
| What is wrong with this machine | journalctl -p err -b |
| Watch a service live | journalctl -u NAME -f |
| What happened at 3pm | journalctl --since "15:00" --until "15:15" |
| Why did it reboot | journalctl -b -1 -e |
| Kernel and hardware messages | journalctl -k |
| Which services failed | systemctl --failed |
| Search rotated text logs | zgrep PATTERN /var/log/name.log* |
| Count values in a column | awk '{print $N}' FILE | sort | uniq -c | sort -rn |
Related reading
- The Life of a Log Line — the long one, underneath everything on this page. What each field in an entry actually is, which of them the program chose and which journald stamped, why a line can be missing with nothing reporting an error, and where the guarantee that makes any of it trustworthy stops
- journalctl — the command reference, option by option
- systemctl — the services these logs come from
- grep — searching text, in logs and everywhere else
- awk — pulling columns out of structured logs
- Inspecting processes — what is running, and what is using the memory
- Checking disk space — the cause behind a lot of odd failures
- Setting up Nginx — access and error log configuration