Learn

The Life of a Scheduled Job: Eight Stages, and the Failure You Should Fear Exits Zero

ByLinuxtastic

Sep 5, 2026

A scheduled job is the only routine thing on a Linux machine that destroys the evidence it ran by succeeding.

Every scheduled job on a modern system is a Type=oneshot unit and a timer, or a line in a crontab, and both of them arrange for a process to start, do something, and go away. When it goes away cleanly, the machine tidies up after it. systemd returns the unit to inactive (dead) and drops the invocation ID in the same instant. cron writes one line naming the command it launched and forgets the rest. Ten seconds after a successful run, a unit that did its work, a unit that ran and did nothing, and a unit that has never run in its life are identical across ActiveState, SubState, Result, ExecMainCode, ExecMainStatus and InvocationID. There is no property you can ask for that separates them.

That would be tolerable if the things you were told to watch with pointed at the gap. They do not. systemctl --failed, OnFailure=, cron’s mail, journalctl -p err — every one of them is keyed on a non-zero exit status, and the failure that ruins your month never produces one. A backup that runs every night, exits zero, and copies nothing is green on every instrument the operating system offers.

This page follows a scheduled job through the eight stages between the line you wrote and the person who needed last night’s output. The mechanism stops at stage seven. The reader’s problem does not.

The eight stages, and where each one can lose you.

  1. The schedule as written — a crontab line or a .timer file. Accepted is not the same as understood.
  2. Registration — enabled, loaded, parsed. Most of the ways a job never runs at all die here, invisibly.
  3. The elapse — the calendar computation, accuracy, randomisation, catch-up, daylight saving. The schedule that runs is not the schedule you wrote.
  4. Exec — a process starts, in an environment owned by PAM or by PID 1 and by neither of you.
  5. The work — or, in the case this page exists for, not the work.
  6. The record — kept at a priority and under an identifier you did not choose, or discarded because no mail transport exists.
  7. The reaction — OnFailure=, a mail, a webhook. All keyed on a non-zero exit. Nothing here reacts to a silent success.
  8. The party outside the machine — the only stage whose test is inverted, because it asks whether something arrived rather than whether something failed.

The reading, if your job is in a crontab. One command, against a fixed path:

readlink -e /usr/sbin/sendmail; echo "exit $?"

It must be -e. readlink -f prints the path back at you and exits 0 when nothing is there, which is exactly the case the reading exists to catch.

What comes backWhat it meansGo to
A path under /usr/sbin — postfix, exim4, msmtp, nullmailerEvery line your jobs have ever printed went to a local mailbox chosen by whoever installed that packageStage 6. The record exists; read that mailbox, then ask whether anything forwards it off the machine
Nothing, exit 1There is no record of your jobs’ output and there never was one. cron logged No MTA installed, discarding output at info, without naming which jobStage 8. Stage 6 never happened for you, so nothing on this machine can answer the question. Go to the last stage

If your job is a systemd timer, there is no equivalent reading, and that is the first thing this page has to tell you. The reason is structural rather than an oversight. In the case this page is about, everything upstream of the work is correct, so every reference upstream reads healthy; and the only reference downstream is OnFailure=, which you wrote yourself and which is keyed on the wrong event. Run this against a job that succeeded a minute ago and against one that has never run:

systemctl show backup.service \
  -p ActiveState -p SubState -p Result \
  -p ExecMainCode -p ExecMainStatus -p InvocationID

The two answers are byte-identical: inactive, dead, success, 0, 0, and an empty invocation ID. The state machine deletes the evidence at the moment of success, and it deletes it for the healthy case and the symptom case alike. On the timer side the first stage is not a reading. It is the absence of one.

Two limits, stated up front. The sendmail reading is about Debian-family packaging — a two-hop update-alternatives symlink at a path Red Hat systems arrange differently, and a cron that is Vixie rather than cronie. And it tells you where your job’s output went. Nothing on the machine will tell you whether the job did its work; that is stage eight, and it is off the machine by construction.

Stage 1 — the schedule as written

Both schedulers accept expressions they do not interpret the way the expression reads. Neither warns you, because in both cases the expression is valid; it just does not mean what a person means by it.

OnCalendar=*-*-31 03:00:00 is not a monthly job. It runs on every month that has a thirty-first, which is seven times a year, at intervals of one, two or three months. *-02-29 runs once every four years and is accepted without comment. Meanwhile 0/15 on its own does not parse at all and *:0/15 is what you meant. Mon-Fri is accepted and quietly normalised to Mon..Fri, so the form the documentation discourages works fine.

There is a command that will tell you, and it is the only one on this page you should run before you enable anything:

systemd-analyze calendar --iterations=5 '*-*-31 03:00:00'

It prints the normalised form and the next five elapses. Five is the useful number: one tells you the expression parses, and five tells you what it means. Anything whose iterations are not evenly spaced is a different job from the one you thought you were writing.

cron has its own accepted-but-surprising forms, and they are old enough that the surprising ones are documented. The one that catches people is that day-of-month and day-of-week are combined with or rather than and when both are restricted, so 0 3 13 * 5 runs on the thirteenth and on every Friday, not on Friday the thirteenth. That is stated in crontab(5). Two others are more practical: a bare % in a command is a newline, so date +%F in a crontab runs date + and produces nothing, and you must write \%F. And files dropped into /etc/cron.daily are run by run-parts, which skips any filename containing a dot and any file without the executable bit — so backup.sh and backup.dpkg-new are both silently ignored where backup runs.

The last item in this stage is not in your file at all. On Debian and Ubuntu, /etc/crontab contains this line:

25 6	* * *	root	test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.daily; }

Whether everything in /etc/cron.daily runs at 06:25 by cron, or at some other time by anacron, is decided by whether a package is installed. anacron is not present on a minimal Ubuntu 24.04 image. Install it on a server for an unrelated reason — it arrives as a recommendation of several desktop-adjacent packages — and the schedule of every daily job on that machine changes, with nothing written down anywhere to say so. The schedule you wrote is not the only input to the schedule that runs.

Stage 2 — registration

Between writing a schedule and having one, something has to load it. This is where most of the ways a job never runs at all live, and it is the stage the standard diagnostics are worst at, because almost everything that dies here dies without becoming a failure.

Set up eight scheduled jobs on one machine — one healthy and seven broken in different ways — and run every command normally recommended for finding scheduling problems. A control that works; one whose timer was written and never enabled; one whose Unit= names a service that does not exist; one whose service is masked; one with a fatal setting in the timer file; one whose service has a dependency that will fail; one guarded by RequiresMountsFor= on a path that is not mounted; and one where somebody enabled the .service instead of the .timer. Then:

$ systemctl list-timers --all
NEXT                         LEFT  LAST PASSED UNIT                 ACTIVATES
Sun 2026-09-06 04:00:00 UTC  9h    -    -      A-control.timer      A-control.service
Sun 2026-09-06 04:00:00 UTC  9h    -    -      D-masked.timer       D-masked.service
Sun 2026-09-06 04:00:00 UTC  9h    -    -      F-depfails.timer     F-depfails.service
Sun 2026-09-06 04:00:00 UTC  9h    -    -      G-mountguard.timer   G-mountguard.service
-                            -     -    -      C-noservice.timer    C-noservice.service
-                            -     -    -      E-badunit.timer      E-badunit.service
-                            -     -    -      fstrim.timer         fstrim.service

Two of the eight are not in that listing. The timer that was written and never enabled is absent, and so is the one where the .service was enabled instead of the .timer. --all does not mean every timer on the machine; it means every timer that is loaded, and a unit that was never started is not loaded. The single most common scheduling mistake there is — enabling the wrong half of a pair, or forgetting to enable at all — is invisible in the command everybody recommends for auditing a machine they have inherited.

The second finding is in the first column. NEXT = - covers at least four unrelated conditions: the service unit does not exist; the timer file has a fatal setting; the service is masked; and a completely healthy timer whose condition is not currently met. fstrim.timer in that listing is the fourth case and is fine. One glyph, four causes, one of them benign, and nothing in the table to tell them apart.

systemctl --failed — the first command most people run on a machine that is behaving oddly — shows none of the eight. It lists units that have run and exited non-zero. None of these has run. And systemctl status on the four that appear healthy prints the same four lines for each: a green dot, active (waiting), a trigger time, and the unit it activates. The masked one included.

You will be told to check a unit before enabling it with systemd-analyze verify, described as a syntax and reference check. It is a syntax check. A timer whose Unit= names a service that does not exist verifies silently and exits 0. A timer with no matching .service anywhere verifies silently and exits 0. A misspelt directive is reported on stderr and exits 0. Exit 1 is reserved for a unit that will not load at all. Read its output; do not test its exit status, and do not treat a clean run as evidence that the units it names exist. It has one further quirk worth knowing: run against a clean file, it prints errors belonging to other units that happen to be enabled on the machine, so its output is not scoped to the file you named.

cron’s version of this stage is smaller and has one trap. A malformed file in /etc/cron.d — a missing user field, say — never runs, and produces nothing at all under the identifier its jobs use. It is logged:

cron[1879]: Error: bad username; while reading /etc/cron.d/nouser
cron[1879]: (*system*nouser) ERROR (Syntax error, this crontab file will be ignored)

Per-job lines are logged as CRON; the daemon’s own complaints are logged as cron. So journalctl -t CRON misses this entirely, and grep CRON /var/log/syslog is case-sensitive and misses it too. journalctl -u cron catches both, and is the one to use. The evidence exists; it is filed where two of the three commonly recommended commands do not look.

One genuine piece of good news, for once. crontab somefile where the file has no trailing newline does not silently drop the last job. It refuses the file — new crontab file is missing newline before EOF, can't install. — and leaves your previous crontab in place. The silent version of that folklore applies to files you write into /etc/cron.d yourself, not to crontab.

Stage 3 — the elapse

OnCalendar=*-*-* 02:30:00 does not run at 02:30:00. The default AccuracySec= is one minute, which means systemd is free to fire anywhere in the following sixty seconds so it can coalesce your wakeup with somebody else’s. Two isolated timers, one with the default accuracy and one at AccuracySec=1us, on consecutive minutes:

default (nominal :17)        tight (nominal :43)
  18:27:24.492   +7.5s         18:26:43.012   +12ms
  18:28:39.904  +22.9s         18:27:43.016   +16ms
  18:29:31.209  +14.2s         18:28:43.012   +12ms

A different delay every time, between seven and twenty-three seconds. systemctl list-timers prints the nominal time; the journal records the actual one; nothing anywhere prints the difference. And the slip is not a constant — when two timers were both nominally at :00, both fired within thirty-one milliseconds. It depends on what else is due nearby, which is why “my timer is late” is unreproducible on a quiet machine and infuriating on a busy one. RandomizedDelaySec= stacks on top of this and re-randomises on every elapse by default.

The interval is not safe either, if you asked for one with OnUnitActiveSec=. That directive measures from the last activation, not from the last completion. A thirty-second timer against a ninety-second job:

18:20:22.084  start pid=2818
18:21:52.089  end   pid=2818
18:21:52.104  Starting slow90.service ...     <- immediately
18:21:52.107  start pid=2877
18:23:22.111  end   pid=2877
18:23:22.132  start pid=2936

Back to back, with no gap at all. By the time a ninety-second job deactivates, its next elapse is already sixty seconds in the past, so it fires instantly. Three elapses were dropped in that window and the journal contains no record that any elapse was missed — no skipping, no already running, nothing. This is what “prevents overlapping runs” costs: your stated interval is silently replaced by the job’s duration, forever, and the only way to see it is to subtract timestamps yourself. The cron equivalent is loud — four copies of a 150-second job started in four minutes, three alive at once, and the machine falls over — which is a worse night and a much easier diagnosis. Use OnCalendar= for anything whose period matters; it recomputes from the wall clock and does not drift.

Persistent=true is two surprises rather than one. Its entire state is a zero-byte file at /var/lib/systemd/timers/stamp-NAME.timer whose mtime is the record, which means touch on that file silently changes catch-up behaviour. A freshly enabled persistent timer does not fire on enable — starting it creates the stamp and waits — so the fear that it runs immediately on every boot is only true once a stamp exists and is old. And a three-day gap on a daily schedule produces exactly one catch-up run, not three. Persistent= means “if the last elapse was missed, run once now”; it does not replay the schedule. For a nightly backup that is correct and benign. For anything that does per-period work — a daily report, a rotation that names its output by date — six missed days produce one run and five absent outputs, and nothing anywhere counts them.

Then there is the clock, which How Time Works on a Linux Machine owns and this page will not re-derive. Two results specific to scheduling. At a spring-forward, a job in the hour that does not exist loses that day entirely, on both schedulers. At a fall-back, when the hour happens twice, a systemd timer fires once and moves to the next day — the widely repeated “your job runs twice” is a cron result, not a timer one. The timer hazard is different and nobody writes it down: consecutive runs of a “daily” job are twenty-five hours apart in the autumn and twenty-three in the spring. Anything that assumes a fixed period — a rotation window, a rate limit, an incremental cursor with a hardcoded lookback — has a day of the wrong length twice a year, and no log line marks either.

The advice you will be given for this is to pin the job to UTC, like so:

[Timer]
OnCalendar=*-*-* 02:30:00
Timezone=UTC

There is no Timezone= directive in [Timer]. systemd 255 logs Unknown key name 'Timezone' in section 'Timer', ignoring, starts the timer successfully, and computes the elapse in system local time. A reader on a London server who follows that advice believes they have pinned the job, has not, and gets exactly the daylight-saving behaviour the advice claims to prevent — from a [Timer] section that looks like it addresses the problem. The working form is a suffix inside the expression, which normalises and shows up in systemd-analyze calendar:

OnCalendar=*-*-* 02:30:00 UTC

Stage 4 — exec, and whose environment it is

Everybody knows that a cron job has a stripped-down environment and that a systemd timer is the more predictable option. On Ubuntu 24.04 that is backwards, and it is backwards in a way that changes which scripts break.

One script, reading its own /proc/$$/environ so the record is what the kernel handed the interpreter, invoked three ways on the same machine at the same time.

--- a systemd service, root, no User=, no Environment=  (six variables)
INVOCATION_ID=80ba0e7981df47b98637b00e0843d67b
JOURNAL_STREAM=9:24300
LANG=C.UTF-8
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
SYSTEMD_EXEC_PID=813
USER=root

--- a cron job, root  (five variables)
HOME=/root
LOGNAME=root
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
PWD=/root
SHELL=/bin/sh

The cron job’s PATH is the full interactive login PATH, /snap/bin included. The systemd service’s is the shorter of the two: no /sbin, no /bin, no /snap/bin. And the systemd service has no HOME at all, while the cron job does. A script ending cd "$HOME", or one that calls a snap, or one that calls something in /sbin by bare name, works under cron and fails under a timer — which is the exact inverse of what you will be told.

The mechanism is not cron. Ubuntu’s unit is ExecStart=/usr/sbin/cron -f -P $EXTRA_OPTS, and /etc/pam.d/cron includes pam_env.so. cron is a PAM application. pam_env reads /etc/environment, and -P tells cron not to overwrite the result with its own compiled-in /usr/bin:/bin. You can prove it in one edit: add a line to /etc/environment and the cron job sees it on its next run, while a systemd service on the same machine at the same instant does not. The same is true of /etc/default/locale, which gives the cron job a real LANG and leaves the service on C.UTF-8, and of /etc/security/limits.conf, which moves the cron job’s ulimit -n and has no effect at all on a service — a service needs LimitNOFILE= in the unit. Remove -P and the folklore comes back exactly: PATH=/usr/bin:/bin. The traditional advice was right for about thirty years and stopped being right on Debian-family systems when -P became the shipped default.

Check yours rather than believing either of us:

systemctl cat cron.service | grep ExecStart
printf '* * * * * /usr/bin/env > /tmp/cronenv\n' | crontab -

The one-sentence version, and it is specific to scheduling: a cron job’s environment is owned by the PAM stack and a systemd service’s is owned by PID 1’s compiled-in defaults, and those two parties have never agreed on anything. Neither is owned by the schedule you wrote. Where a PATH comes from in general, and the one sentence that explains every broken systemctl --user, belongs to The Life of a Login; what is specific here is that scheduling gives you two of them and lets you pick without telling you they differ.

Two consequences worth carrying. First, every cron job on the machine shares one cgroup — root’s job and a deploy user’s job both report 0::/system.slice/cron.service. Per-job accounting, MemoryMax=, CPUQuota= and any attribution in systemd-cgtop are unavailable for cron jobs as a class; they are one blob. Each timer’s service gets its own. Second, systemctl --user fails from both, with Failed to connect to bus: No medium found, and loginctl enable-linger is one clause short of fixing it. Lingering starts the user manager and creates /run/user/1000, but a process with no XDG_RUNTIME_DIR in its environment still cannot reach it. A cron job needs the variable set explicitly:

XDG_RUNTIME_DIR=/run/user/$(id -u) systemctl --user list-timers

Stage 5 — the work, or not the work

This is the stage the whole page is about, and the shortest way to describe it is to run it. Seven scripts under identical timers. One control that copies a file. Six that run, exit zero, and copy nothing — a wrong working directory; an empty glob with nullglob unset, so the pattern is passed through literally and the copy of a nonexistent file is inside an if that swallows it; a for loop over nothing with nullglob set; a lock it could not take and treated as “nothing to do”; a set -e script whose early exit was masked by a trailing true; and a pipeline whose failure was swallowed because pipefail was unset.

UNIT            RESULT    EXIT  ACTIVE    JOURNAL LINES
s0-control      success   0     inactive  4
s1-wrongcwd     success   0     inactive  3
s2-emptyglob    success   0     inactive  3
s3-nullglob     success   0     inactive  3
s4-lock         success   0     inactive  3
s5-seteearly    success   0     inactive  3
s6-pipe         success   0     inactive  4

The three journal lines are the same three for all of them: Starting, Deactivated successfully, Finished. systemctl --failed lists none. OnFailure= was configured on all seven and fired for none — the file it would have written was never created. Under cron the same seven produce seven identical (root) CMD (...) lines. The control is distinguishable from the other six by exactly one thing: it prints the word copied. That is, it is distinguishable only because its author instrumented it. Nothing the scheduler provides separates a job that did its work from a job that completed.

That is the distinction the rest of this page turns on, so it is worth naming. A scheduled job has three outcomes, not two. It can fail, which every instrument on the machine is built to catch. It can work. And it can complete — run to the end, exit zero, and produce nothing — which no instrument on the machine distinguishes from working. Completing is the common one, because almost every way a script can quietly do nothing ends in a successful exit.

One of the seven is worth its own paragraph, because it is the sharpest thing in the experiment. The pipeline job did leave evidence:

18:26:54.793  s6-pipe.sh[3533]: cat: /lab/src/DOES-NOT-EXIST.txt: No such file or directory
18:26:54.794  systemd[1]: s6-pipe.service: Deactivated successfully.

An error message and a success verdict, 1.3 milliseconds apart, in the same unit’s journal. The error is at priority 6 under the identifier s6-pipe.sh; the verdict is at priority 6 under systemd. The evidence exists. systemd kept it. And journalctl -u s6-pipe.service -p err returns nothing at all, which is the next stage’s problem.

Stage 6 — the record

Five jobs, identical except for what they printed and what they exited with, run under both schedulers on a machine with no mail transport installed — which is the default state of every minimal cloud image. Here is everything cron recorded about that minute:

CRON[2506]: (root) CMD (/lab/j/silent3.sh)
CRON[2509]: (root) CMD (/lab/j/out0.sh)
CRON[2508]: (root) CMD (/lab/j/silent0.sh)
CRON[2511]: (root) CMD (/lab/j/fail3.sh)
CRON[2512]: (root) CMD (/lab/j/err0.sh)
CRON[2505]: (CRON) info (No MTA installed, discarding output)

cron never records an exit status. Not for any job, ever. The job that exited 3 having written to both streams and the job that exited 0 having written nothing produced one identical line each. The only thing cron logs is the command it launched. Every marker the jobs printed appears zero times anywhere in the journal, and /var/mail is empty.

The discard is logged, and the way it is logged is the finding. It is at priority 6 — info, the same priority as “job started”, not a warning and not an error. And it does not name the job. Three of those lines appeared for three jobs that produced output, and the PIDs on them are the PAM session parents rather than the PIDs on the CMD lines, so you cannot tell from the log whose output was thrown away. If you have never seen that line, it is because nothing you run looks at info.

systemd keeps everything cron loses. The priorities are the story:

p6 info     fail3.sh   STDOUT-from-fail3 marker-C
p6 info     fail3.sh   STDERR-from-fail3 marker-D
p5 notice   systemd    j-fail3.service: Main process exited, code=exited, status=3/NOTIMPLEMENTED
p4 warning  systemd    j-fail3.service: Failed with result 'exit-code'.
p3 err      systemd    Failed to start j-fail3.service - job fail3.

Three things follow. stderr is not promoted — both streams arrive at priority 6, because the default is StandardError=inherit and the stream’s SyslogLevel is 6. Anything your script says on stderr is filed as information. The identifier is the executable’s basename, not the unit name: fail3.sh, not j-fail3.service. So journalctl -t backup finds nothing on a machine where backup.service runs backup.sh; you want -u backup.service, or -t backup.sh. And therefore:

$ journalctl -u j-fail3.service -p err
Failed to start j-fail3.service - job fail3.
Failed to start j-fail3.service - job fail3.

-p err is sold everywhere as “only the failures”. It is the one command guaranteed to show you systemd’s verdict and hide your script’s explanation of it. Drop the flag. The general form of this — that priority is a property of who wrote the line rather than of how bad the news is, and every field in a journal record has a different author — belongs to The Life of a Log Line. Scheduling is where it costs the most, because a scheduled job is the case where nobody was watching the terminal.

Stage 7 — the reaction

This is where the mechanism ends, and it is one stage before the reader’s problem does.

Everything the operating system offers to react on your behalf is keyed on a non-zero exit status. OnFailure= activates a unit when this one enters a failed state. systemctl --failed lists units that ran and exited non-zero. cron’s mail is triggered by output, and its absence by silence. journalctl -p err filters on a priority that only systemd’s own verdict reaches. Four mechanisms, four different implementations, one shared assumption: that a job which did not do its work will say so by exiting badly. The seven scripts in stage 5 exited zero, and all four mechanisms were quiet for all seven.

The most-recommended guard at this stage does not work at all when it matters most:

[Unit]
RequiresMountsFor=/backup

This is described as stopping the job if /backup is not mounted. It creates a dependency on whichever mount unit currently covers that path. If /backup is an ordinary directory on the root filesystem — which is what you have when the fstab line was never added, or when the disk went away and the mountpoint directory remained — it resolves to After=-.mount, the root filesystem, which is always active. The job runs:

$ mountpoint /backup
/backup is not a mountpoint
$ systemctl show G-mountguard.service -p After | tr ' ' '\n' | grep mount
After=-.mount
$ journalctl -u G-mountguard.service | tail -3
Starting G-mountguard.service ...
echo[3270]: ran-G-EVEN-THOUGH-BACKUP-IS-NOT-MOUNTED
Finished G-mountguard.service.

The difference between “the backup disk was not mounted, so nothing ran” and “the backup ran, wrote a full copy to the root filesystem, and reported success” is the whole subject of this page, and the directive lands on the wrong side of it. A mountpoint -q /backup || exit 1 at the top of the script is not belt and braces alongside RequiresMountsFor=; it is the only one of the two that does the job.

One more, because it appears in more daily-report scripts than anything else in this list:

systemctl --failed --no-legend || echo "  none"        # never prints none
systemctl --failed --no-legend | grep -q . || echo "  none"   # does

systemctl --failed exits 0 when nothing has failed. The first line prints an empty section every morning and the reader reads the emptiness as the report working. Both of them, note, are reporting on the wrong event anyway.

Stage 7 is where most writing on this subject stops, usually at OnFailure=alert@%n.service, described as the real payoff of moving from cron to timers. It is a real payoff. It closes the case where a job runs and fails. It cannot close the case where a job runs and completes, and it cannot close the case where a job never ran at all — OnFailure= is a property of the service, and a timer that never fired never activated the service, so there is nothing to fail.

Stage 8 — the party outside the machine

Stages 1 through 7 all ask the same question: did something go wrong? Every one of them answers “no” for a job that ran and did nothing. Stage 8 is the first stage in the chain that asks a different question — did something arrive? — and it is the only question in this whole sequence whose answer changes when the job completes without working.

That inversion is the entire content of the stage, and it is why the mechanism cannot supply it. A watcher that lives on the machine and is started by the same scheduler shares the scheduler’s failure modes. A watcher keyed on absence has to be somewhere that notices when nothing turns up, which means it has to be somewhere the job is not. There are two shapes of it and both are older than systemd:

  • A dead-man’s switch. The job checks in when it has done its work; something off the machine alerts when the check-in does not arrive inside a window. This catches the job that completed, the job that never fired, the machine that was off, and the scheduler that was broken — because it is not watching any of them, it is watching for an arrival.
  • A freshness check on the output. The job writes a timestamp file, or you check the age of the newest artefact against the schedule. Cheaper, works without a third party, and answers the question directly: not “did it run” but “is there anything new”.

Neither is new and neither is mine. Monitoring a Server Without a Full Observability Stack has both, and calls the first the only way to detect a backup that silently stopped running, which is the correct claim and was published before this page existed. The Life of a Certificate states the sharpest version of the whole thesis in six words — months of clean cron mail can mean nothing was ever attempted — and that sentence is the reason this page is about the mechanism rather than about the remedy. What is left to add is the reason the remedy has to live where it does: not because off-machine monitoring is generally good practice, but because stages 1 through 7 are structurally incapable of it. Every one of them is downstream of an exit status, and the exit status is zero.

The practical form, if you take nothing else from this page: have the job write a timestamp file on success, and have something check its age. That is two lines, it works identically for cron and for timers, and its failure mode is that you get told about a problem you do not have.

A worked diagnosis

A nightly database backup has been running for two years. The storage box it writes to has nothing newer than March. Nobody was alerted, and the daily report has been clean every morning.

$ systemctl status backup.timer
● backup.timer - nightly database backup
     Active: active (waiting) since Mon 2026-01-06 09:12:41 UTC
    Trigger: Sun 2026-09-06 02:30:00 UTC; 8h left
$ systemctl --failed
0 loaded units listed.

Stage 7 answers, and its answer is worth nothing. Green, waiting, nothing failed. Every one of the eight registration faults in stage 2 also prints a green active (waiting), so this rules out nothing on its own — but the Trigger line is real information: it means the timer is loaded and has a next elapse, which eliminates the two faults that would have kept it out of list-timers entirely.

$ journalctl -u backup.service --since "3 days ago" | grep -c Finished
3
$ journalctl -u backup.service -n 3 --no-pager
Starting backup.service - nightly database backup...
backup.service: Deactivated successfully.
Finished backup.service - nightly database backup.

Stage 5, and the three-line signature. It ran, on schedule, three nights out of three. Three lines and no fourth — the job printed nothing at all, which is what a well-behaved script does when it thinks everything is fine. Note what was not run here: -p err, which would have returned nothing on a night the script screamed, and would have looked exactly like this.

$ systemctl show backup.service -p InvocationID -p Result -p ExecMainStatus
InvocationID=
Result=success
ExecMainStatus=0

This is the reading that is not a reading. Identical to a unit that has never run. It tells you nothing, and knowing it tells you nothing is what stops you spending an evening on systemctl show. The evidence is not on the unit; it is in the filesystem the job wrote to.

$ ls -t /backup | head -2
db-2026-09-05.sql.gz
db-2026-09-04.sql.gz
$ mountpoint /backup
/backup is not a mountpoint
$ df -h /backup /
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1       900G  611G  243G  72% /
/dev/vda1       900G  611G  243G  72% /

Stage 7 has the cause after all, in the guard rather than the alarm. The backups exist and are current — they are on the root disk. /backup stopped being a mount point in March, when the volume was detached and reattached during a reboot and the fstab entry did not survive. The empty directory underneath was still there, so every night since, the job has written a perfectly good backup to the system disk of the machine it is backing up. RequiresMountsFor=/backup was in the unit, resolved to -.mount, and let it through. Six months at roughly a gigabyte a night is most of the 611G.

The fix is three lines and only one of them is the mount. mountpoint -q /backup || exit 1 at the top of the script, so tomorrow it fails loudly instead of succeeding quietly. The fstab entry restored. And a check on the age of the newest file on the storage box, run from the storage box, because that is the only one of the three that would have caught this in March — and it would have caught the empty-glob version, and the wrong-directory version, and the night the machine was off, none of which the other two see.

What everybody says, and what it actually matches

Each of these is repeated constantly and each was measured on Ubuntu 24.04, systemd 255.4 and Vixie cron 3.0pl1-184.

The adviceWhat is actually trueStage
cron gives you a minimal PATH, usually /usr/bin:/binOn Debian and Ubuntu, cron runs as cron -f -P and PATH comes from /etc/environment via PAM. It is the full login PATH, /snap/bin included4
Use a timer, the environment is more predictableMore predictable and more restrictive. A service gets no HOME at all and a shorter PATH than a cron job on the same machine4
Timezone=UTC in [Timer] pins a job across DSTNot a directive. Logged as Unknown key name, ignored, timer runs in local time. The suffix form OnCalendar=… UTC is what works3
At a fall-back your job runs twiceTrue on cron. False on systemd timers, which fire once. The timer hazard is a 25-hour gap between consecutive runs3
Persistent=true catches up the runs you missedIt runs once, not once per missed occurrence. Three days of downtime on a daily timer produced one run3
Timers prevent overlapping runsBy silently replacing your interval with the job’s duration. A 30-second timer on a 90-second job runs back to back, and no line marks the dropped elapses3
systemd-analyze verify checks a unit’s references before you enable itIt does not check references, and exits 0 for a Unit= naming a service that does not exist and for a misspelt directive2
systemctl list-timers --all shows every scheduled jobEvery loaded timer. One you wrote and never enabled is absent, which is the mistake the command is recommended for finding2
journalctl -u X -p err shows the failuresShows systemd’s verdict only. Your script’s stdout and stderr are priority 6 and are filtered out6
RequiresMountsFor= stops the job if the disk is not mountedOnly if the path is a mount point. On an ordinary directory it resolves to -.mount and the job runs, writing to the system disk7

The one error, and everything it generates

Every mistake in that table comes out of a single sentence nobody says out loud, because it seems too obvious to state: a scheduled job either works or fails.

It has three outcomes, not two. It can fail. It can work. And it can complete — run to the end, exit zero, produce nothing — and no instrument the operating system offers distinguishes completing from working. Believe there are two outcomes and the rest writes itself. If there are two outcomes, Result=success means the work was done, so an empty systemctl --failed is an all-clear. If there are two outcomes, OnFailure= covers everything that can go wrong, so configuring it is finishing the job. If there are two outcomes, the absence of errors is the absence of problems, so -p err returning nothing is good news. If there are two outcomes, a green active (waiting) is a job that will run, so list-timers is an inventory. And if there are two outcomes, the interesting question is always did it fail, so nobody ever builds the thing that asks did it arrive. Each of those follows perfectly from the premise. Each of them is wrong, and each of them is wrong in the same direction — towards reassurance.

The one-sentence test for anything else written on this subject: ask what the page says a cron job’s PATH is, and then whether it says where that PATH came from. A page written from memory says “usually /usr/bin:/bin, so use absolute paths”. A page written from a machine says the PATH is whatever /etc/environment contains, because cron is a PAM application and -P tells it not to overwrite what PAM set — and then says that this makes a cron job’s PATH longer than a systemd service’s. The wrong answer is the most-copied sentence in the genre, the right one cannot be reasoned to, and it takes one minute to check on any machine.

Three things nothing on your machine can tell you, which is the honest shape of this subject:

  • Whether last night’s run did its work. Not “whether it failed” — whether it did the thing. No property of the unit, and no line cron writes, is different between the two.
  • How many elapses were skipped. Overrun drops them silently, Persistent= collapses them to one, and nothing counts either.
  • Which job’s output was discarded. cron’s No MTA installed line does not name it, and the PID on that line is not the PID of the job.

And one part of this system has already noticed the problem and half-fixed it, which is worth knowing because almost nobody uses it. Every reaction on this page is triggered by failure, which is what makes stage 8 have to live off the machine. systemd 249 added the mirror of OnFailure=:

[Unit]
OnSuccess=backup-checkin.service

A unit activated when this one succeeds. It is the only hook in the whole sequence keyed on something happening rather than on something going wrong, and it is what a dead-man’s switch needs: a signal that arrives every night, whose absence is the alarm. That turns stage 8 into something you can wire from inside the machine, pointing outwards. It is half a fix rather than a whole one, and the half it is missing is the honest part — OnSuccess= fires on exit zero, so a job that completed and did nothing checks in as cheerfully as one that worked. It moves the burden of honesty onto the script, where a mountpoint -q or a “did I actually copy anything” check has to live anyway. It does not remove the burden. Nothing does.

Before you call it done

  • systemd-analyze calendar --iterations=5 on the expression, and the five elapses evenly spaced unless you meant otherwise
  • The .timer enabled, not the .service — and the unit’s name in systemctl list-timers, because absence there is not a display problem
  • The script’s own guard on the thing it cannot do without: mountpoint -q, a non-empty source, a lock it either takes or fails on. Not RequiresMountsFor= alone
  • set -euo pipefail, and no trailing true that undoes it
  • The job printing one line saying what it did — a count, a size, a filename. It is the only thing that will distinguish it from the six failures in stage 5
  • The environment checked the way it will run, not the way you tested it — systemd-run --unit=t1 /path/to/script, or one crontab line writing env to a file
  • Something off the machine that alerts on absence: a check-in that stops arriving, or the age of the newest output checked from where the output lands

Related reading

By Linuxtastic

Related Post

Learn

The Life of a Block Device: Seven Names for One Disk, and Not One of Them Is Both Stable and Unique

Sep 6, 2026 Linuxtastic
Learn

The Life of a Log Line: Every Field Has an Author, and the Record Does Not Say Who

Sep 4, 2026 Linuxtastic
Learn

How Time Works on a Linux Machine: When the Clock Moves, Nothing Writes It Down

Sep 4, 2026 Linuxtastic

You missed

Learn

The Life of a Block Device: Seven Names for One Disk, and Not One of Them Is Both Stable and Unique

September 6, 2026 Linuxtastic
Learn

The Life of a Scheduled Job: Eight Stages, and the Failure You Should Fear Exits Zero

September 5, 2026 Linuxtastic
Guides

The Life of an Outbound Email: Four Addresses, and the One You Set Is the One Nobody Checks

September 5, 2026 Linuxtastic
Learn

The Life of a Log Line: Every Field Has an Author, and the Record Does Not Say Who

September 4, 2026 Linuxtastic