Monitoring advice usually starts with Prometheus, Grafana and an alertmanager. That is the right answer at a certain scale, and it is a weekend of work plus a permanent maintenance commitment for one or two servers.
For a small setup, three things cover nearly everything that actually goes wrong:
- A service died — and nothing told you.
- The disk filled up — which breaks things in ways that look unrelated.
- The whole machine is unreachable — which nothing running on it can report.
This sets up all three in about an hour, using tools already on the server.
What this does not give you: history, graphs, or trend analysis. It tells you something is wrong now; it will not tell you memory has been creeping up for three weeks. If you need that, you have outgrown this page and should build the real thing.
1. Get alerts out of the machine
Everything below depends on being able to notify you, so do this first. Do not use email from the server itself — mail from a random VPS is quietly discarded by every major provider, and you will believe alerting works when it does not.
Use a webhook to something you actually look at. This example uses ntfy, which needs no account, but the shape is the same for Slack, Discord, Telegram or Gotify:
sudo nano /usr/local/bin/notify#!/bin/bash
# notify "Title" "Message body"
set -euo pipefail
TOPIC="https://ntfy.sh/your-secret-topic-name"
HOST=$(hostname -s)
curl -fsS \
-H "Title: [${HOST}] ${1}" \
-H "Priority: high" \
-d "${2}" \
"${TOPIC}" > /dev/nullsudo chmod +x /usr/local/bin/notify
# Test it NOW - everything below is worthless if this fails
/usr/local/bin/notify "Test" "If you can read this, alerting works"Verify: the notification arrives on your phone or in your chat. Do not continue until it does.
Two notes. Put the hostname in the title — with three servers, an alert that does not say which machine is only half an alert. And on a public ntfy topic, choose a long unguessable name, since anyone who knows it can read your alerts.
2. Tell me when a service dies
systemd already knows when a unit fails. It just does not tell anyone by default. OnFailure= fixes that, and it is the single highest-value thing on this page.
Create one reusable alerting unit that takes the failed service’s name as a parameter:
sudo nano /etc/systemd/system/alert@.service[Unit]
Description=Alert that %i failed
[Service]
Type=oneshot
ExecStart=/bin/bash -c '/usr/local/bin/notify "%i FAILED" \
"$(systemctl status %i --no-pager -n 20 | tail -20)"'Then attach it to any service you care about, using a drop-in rather than editing the packaged unit:
sudo systemctl edit nginx.service[Unit]
OnFailure=alert@%n.servicesudo systemctl daemon-reload
# Prove it works - kill the service uncleanly
sudo systemctl kill -s SIGKILL --kill-who=main nginx
Verify: an alert arrives containing the last twenty log lines — which is usually enough to know what happened without logging in.
systemctl edit writes to /etc/systemd/system/nginx.service.d/override.conf, so package updates will not overwrite it. Repeat for each service that matters. Worth doing for the same list you would restart in a panic: the web server, the database, the application itself, and any backup timer.
While you are here, make sure important services come back on their own:
# In the same drop-in
[Service]
Restart=on-failure
RestartSec=10sA service that restarts itself and tells you it did is a far better position than one that does either alone. systemctl covers units in more depth.
3. Watch the disk
A full disk causes failures that look like everything except a full disk — databases refusing writes, sessions failing, logs stopping, updates breaking. It is worth its own check, and it must run before things break rather than after.
sudo nano /usr/local/bin/check-disk#!/bin/bash
set -euo pipefail
THRESHOLD=85
df -h --output=pcent,ipcent,target -x tmpfs -x devtmpfs \
| tail -n +2 | while read -r used inodes mount; do
u=${used%\%}
i=${inodes%\%}
if [ "$u" -ge "$THRESHOLD" ]; then
/usr/local/bin/notify "Disk ${used} on ${mount}" \
"$(df -h "$mount"; echo; du -h -d1 "$mount" 2>/dev/null | sort -h | tail -5)"
fi
if [ "$i" -ge "$THRESHOLD" ]; then
/usr/local/bin/notify "Inodes ${inodes} on ${mount}" \
"Running out of inodes, not space. df -i for detail."
fi
donesudo chmod +x /usr/local/bin/check-disk
sudo /usr/local/bin/check-disk # test it
# Run it hourly
sudo crontab -e
# 17 * * * * /usr/local/bin/check-diskChecking inodes as well as space is the part most scripts miss. A filesystem can be 40% full and completely unable to create a file — filesystems explains why. The alert also includes the five largest directories, so you often know the cause before opening a terminal.
Note the odd minute in the cron line. Scheduling everything at 0 means every check on every server fires simultaneously; spreading them out is free.
Remember cron’s environment is small — five variables, no aliases, no LANG, and nothing your shell startup files set. Its PATH is not the problem people expect, though: on Debian and Ubuntu it is the full login PATH from /etc/environment, because cron runs as a PAM application. Use absolute paths in anything it runs anyway — which is why the script calls /usr/local/bin/notify in full — because the next machine may not be Debian-family. The Life of a Login covers where a PATH comes from.
4. Watch from outside
Everything so far runs on the server, which means none of it works when the problem is the server. A machine that has run out of memory, lost its network or been shut down cannot report that.
You need one check from somewhere else. Two reasonable options:
| Approach | Good for |
|---|---|
| A hosted uptime service | Zero maintenance; free tiers are usually enough |
| A dead-man’s switch | Cron jobs and backups — alerts when something stops reporting |
The second is the more interesting one and it is underused. Instead of alerting when a job fails, you alert when a job stops checking in — which catches the case where the job never ran at all, the server was off, or cron itself was broken.
# In your backup script, at the very end
/usr/local/bin/backup.sh && curl -fsS https://hc-ping.com/your-uuid-here
# Or report failure explicitly
if /usr/local/bin/backup.sh; then
curl -fsS https://hc-ping.com/your-uuid
else
curl -fsS https://hc-ping.com/your-uuid/fail
fiIf the ping does not arrive within the window you configured, you get an alert. This is the only way to detect a backup that silently stopped running, which is a genuinely common and genuinely serious failure — see automated backups.
If you would rather self-host the whole thing, Uptime Kuma runs happily in a container behind the reverse proxy — on a different machine from the one it is watching, which is the entire point.
5. A morning summary
Alerts tell you about emergencies. A daily digest catches the slow problems that never trip a threshold.
#!/bin/bash
# /usr/local/bin/daily-report
set -uo pipefail
# systemctl --failed exits 0 when nothing has failed, so `|| echo none`
# after it never runs. Capture the output and test that instead.
failed=$(systemctl --failed --no-legend)
{
echo "Uptime: $(uptime -p)"
echo "Load:$(cut -d' ' -f1-3 /proc/loadavg) on $(nproc) cores"
echo
echo "Disk:"; df -h -x tmpfs -x devtmpfs | tail -n +2
echo
echo "Memory:"; free -h | head -2
echo
echo "Failed units:"; echo "${failed:- none}"
echo
echo "Errors in the last 24h: $(journalctl -p err --since '24 hours ago' --no-pager | wc -l)"
echo
echo "Reboot required: $([ -f /var/run/reboot-required ] && echo YES || echo no)"
} | /usr/local/bin/notify "Daily report" "$(cat)"Run it once a day from cron. Thirty seconds of reading catches a disk at 70% and climbing, a unit that has been failing quietly, or a pending reboot from a kernel update — none of which are urgent, and all of which become urgent if ignored. Read the error count as a rough tripwire rather than a health metric, because it is wrong in three directions at once: it counts lines, so a single stack trace inflates it by its own depth; it misses every entry that arrived with no PRIORITY and every application that records its severity inside a JSON message; and on the day journald starts discarding messages the number falls, because the notice explaining why is filed at info under systemd-journald. A morning when it drops to zero deserves as much attention as one when it spikes: The Life of a Log Line.
What to actually alert on
The failure mode of monitoring is too many alerts, not too few. An alert you routinely ignore is worse than no alert, because it trains you to ignore the real one.
| Condition | Alert? |
|---|---|
| Service failed | Yes, immediately |
| Site unreachable from outside | Yes, immediately |
| Backup did not check in | Yes |
| Disk above 85% | Yes |
| Certificate expiring in under 14 days | Yes |
| The daily error count drops to zero, or falls sharply | Yes — that is usually throttling or a broken logging path rather than a good day |
| Brief CPU spike | No |
| Memory looking full | No — that is cache |
| Failed SSH login attempts | No — constant background noise |
| Load above some number | No, unless sustained and compared to core count |
Two of those “no” rows are things people alert on and then regret. Memory appearing full is normal — Linux uses spare RAM as cache, and processes and memory explains which number to read instead. And a public server sees thousands of failed SSH attempts a day; if that bothers you, the fix is key-only authentication, not an alert. If you redirect that attention to accepted logins instead — which is the usual advice, and better advice — check who wrote the line before you act on it, because a convincing accepted-login message can be produced by any unprivileged local account: The Life of a Log Line.
The useful test: would this wake you up, and would you do something about it? If not, it belongs in the daily report rather than an alert.
Things that catch people out
| Symptom | Cause |
|---|---|
| Alerts never arrive | Sending email from the server — use a webhook |
| Script works by hand, not in cron | Missing environment; use absolute paths |
OnFailure never fires | Forgot daemon-reload; the service exits 0 having done nothing; or the timer never fired, which OnFailure= on the service cannot see — The Life of a Scheduled Job |
| Server died, nothing alerted | All monitoring was on that server |
| Disk alert fired too late | Threshold too high, or check too infrequent |
| Backup broken for weeks | No dead-man’s switch |
| Everyone ignores the alerts | Too many, or too many are not actionable |
Whatever you set up, break something on purpose once and confirm the alert arrives. Untested monitoring is not monitoring — it is a belief about monitoring, and it is usually wrong.
Quick reference
| You want | Command |
|---|---|
| Anything failed right now? | systemctl --failed |
| Errors since boot | journalctl -p err -b |
| Alert when a service dies | systemctl edit NAME, add OnFailure= |
| Restart it automatically too | Restart=on-failure |
| Space and inodes | df -h and df -i |
| Test an alert path | Run the notify script by hand |
| Test failure alerting | sudo systemctl kill -s SIGKILL NAME |
| See the drop-in you created | systemctl cat NAME |
Related reading
- systemctl — units, drop-ins and restart policies
- The Life of a Scheduled Job — the long one: why every alarm on this page is keyed on a non-zero exit, and what to do about the failure that never produces one
- Reading Linux logs — what to do once an alert arrives
- The Life of a Log Line — why a count of errors is a weaker signal than it looks, and what a gap in a log actually means
- Automated backups — and why they need a dead-man’s switch
- Processes and memory — which numbers are worth alerting on
- Filesystems — inode exhaustion, the invisible disk-full
- cron — scheduling the checks, and what its environment really is