There is one restore drill nearly every backup guide publishes, including ours, and the instinct behind it is right — restore into a scratch directory now, not at three in the morning. It even names what it is protecting you from:
the first restore is not the moment to discover that the excludes were too aggressive, the database dump was empty, or the permissions did not survive
The verification almost everybody reaches for at that point is diff. Here is that drill, run against a tree restored with rsync -aH — the flags most backup scripts are built on:
live restored
owner:group and mode identical identical
named ACL entry user:www-data:r-- (none)
user.deployed xattr 2026-08-31 (none)
helper capability cap_net_bind_service=ep (none)
store.img blocks 0 524288
du -sh of the tree 36K 257M
$ diff /tmp/restore-test/index.html live/index.html
identical
$ diff -r /tmp/restore-test/ live/ ; echo "exit=$?"
exit=0An ACL gone, an extended attribute gone, a file capability gone, and a directory that grew from thirty-six kilobytes to two hundred and fifty-seven megabytes. The verification exits 0 on all of it — not because diff is broken, but because diff answers a question about bytes, and every one of those losses is about something that is not a byte.
This page is the hour after the copy comes back. The surprising part, and the reason most accounts of it go wrong, is where the loss happens. Almost nothing here was missing from the archive. The capability bytes are in the tar file — you can grep for them. An unprivileged nightly cron records other users’ ownership and the setuid bit perfectly. restic carries ACLs, extended attributes, capabilities and hard links with no flags at all. The archive is usually complete and truthful. What throws half of it away is the restore: a flag on the extraction side, the account the command ran as, and a file on the target machine that was never in the backup.
One thing to settle before any of it, because it is load-bearing and almost universally believed backwards. When you copy files preserving ownership, it is the name that travels, not the number. Both tar and rsync put the name and the number in the archive, and both resolve by name by default; --numeric-owner and --numeric-ids are what make the numbers travel, and almost nobody passes them. Stage 2 is that sentence, measured both ways.
What this page does not cover. Choosing a tool, schedules and retention are Automated Backups‘s. PostgreSQL’s own backup and restore is Running PostgreSQL Properly‘s, and etcd’s is etcd, Honestly‘s. What ACLs and capabilities are is Permissions and Privilege, Properly‘s. This page is the hour in which somebody’s backup becomes a working machine.
A restore has its own order, and it is not the backup’s order reversed. Seven stages, each deciding something the backup did not: 1. what the archive recorded, and what it left you to decide · 2. identity — the names in the archive against the numbers on the new machine · 3. the account that ran the restore, and its umask · 4. the metadata beside the bytes · 5. the shape of the file: links, holes, and the things that are not files · 6. what the walk could not see · 7. the other copy, and everyone who cannot tell them apart.
The hinge: compare the two things you are holding, with a tool that can see more than bytes. A restore is the one moment when both halves are in front of you. Every command exited 0 and diff says they are identical; ask something that sees further:
rsync -n -i -aHAX --checksum /path/to/the/backup/ /path/to/the/restored/tree/-n changes nothing. Read which letter appears, not how many lines there are. Each column is one attribute, and each attribute is a reference to something that is not in the file:
| What you see | What differs | Who owns the answer | Stage |
|---|---|---|---|
>f+++++++++ | on one side and not the other | the exclude list somebody wrote months ago | 1 |
.f....og... | owner or group | the target’s /etc/passwd, written by the package manager in whatever order it installed | 2 |
.f...p..... | the mode bits | the umask of the account that ran the restore | 3 |
.f.......a. | an ACL | whoever set it, and the flag the extraction did not pass | 4 |
.f........x | an extended attribute | the kernel — it reads security.capability, and it refused to let an unprivileged restore write it | 4 |
hf … => … | one side is a hard link, the other a copy | the filesystem’s link count | 5 |
.f..t...... | the timestamp | the clock on the machine that did the restoring | 6 |
| nothing at all | nothing rsync can compare differs — the branch the page exists for | everybody who is not on this machine | 6 and 7 |
The silence is the important branch, and it has two causes. Either the restore carried everything rsync can carry — in which case what is left is stages 6 and 7, none of which is inside the tree. Or the backup never had it either, and you have compared a copy with a copy of the copy. One command separates them, and no output means the loss happened at stage 1, months ago, on a machine you may no longer have:
getfattr -Rd -m 'security|system' /path/to/the/backup/ 2>/dev/null | headTwo limits, both measured. If your backup is a tar archive rather than a tree, the nearest equivalent is tar -df <archive> -C <tree>, and even with --acls --xattrs --xattrs-include='*' its output is byte-for-byte identical with the attribute flags and without them: tar’s verification cannot see the attributes tar’s own flags exist to carry. It answers stages 1, 2, 3, 5 and 6 and hands stage 4 to getfacl -R and getcap -r. And the hinge is a localiser, not an oracle: it compares the backup with the restore, and says nothing about whether the backup was true.
Stage 1: What the archive recorded, and what it left you to decide
Start by disproving what everybody assumes, because the rest of the page rests on it. One tar archive, two files owned by two service accounts, listed twice:
$ tar -tvf plain.tar
-rw-r--r-- metrics/metrics 9 2026-09-04 17:48 ./metrics.db
-rw------- appsvc/appsvc 7 2026-09-04 17:48 ./app.conf
$ tar -tvf plain.tar --numeric-owner # the same archive
-rw-r--r-- 1600/1600 9 2026-09-04 17:48 ./metrics.db
-rw------- 1500/1500 7 2026-09-04 17:48 ./app.confThe ustar header carries both. The archive was never a claim about one namespace; the column you are shown is decided by the flags you typed. Which is also why the obvious diagnostic for this subject — list the archive and see whether it says names or numbers — is not one: it reads its own input.
The same is true of the attribute everybody believes is simply absent. tar --xattrs writes the file capability into the archive, and the bytes are findable with grep -a 'SCHILY.xattr.security.capability'. It is in there. It does not come back. Stage 4 is why, and the reason is printed in GNU tar’s own manual.
Three things really are absent, and all three are stages 5 and 6: what was never a file, what was moving while the walk went past, and sockets. Everything else on this page was in the backup.
Verify: run getfattr -Rd -m 'security|system' over the backup, not the machine. If it prints nothing, stop reading about restores and go and fix the backup.
Stage 2: The names in the archive, and the numbers on the new machine
A service account’s number is not a property of the service. It is whatever was next when the package was installed, on the machine where it was installed. Install the same packages in a different order on the rebuilt box and the same names hold different numbers, with nothing recording that this has happened. The same archive, restored three ways onto a target where two service accounts hold each other’s numbers:
# source: appsvc=1500 metrics=1600 target: appsvc=1600 metrics=1500
--- plain.tar, default flags (tar matches by NAME) ---
-rw------- 1 appsvc appsvc 7 app.conf # right owner, different number
-rw-r--r-- 1 metrics metrics 9 metrics.db
--- the same archive with --numeric-owner ---
-rw------- 1 metrics metrics 7 app.conf # right number, wrong owner
-rw-r--r-- 1 appsvc appsvc 9 metrics.db
--- default flags, names absent from the target altogether ---
-rw------- 1 1500 1500 7 app.conf # falls back to the number, silently
-rw-r--r-- 1 1600 1600 9 metrics.dbThree outcomes, all three exit 0, none of them warned — and the default, the one you get by not thinking about it, is the one that keeps the ownership meaningful. The name is the durable identifier; the number is the local accident. rsync is the same: the name is on the wire, and --numeric-ids is what takes it off.
Stage 2’s own question: does the number on the restored file resolve, and to whom? find‘s %u prints the user’s name — or the bare number, if that number has no name here. Read which kind of token comes back, not its value.
find /srv /home /var/www -xdev -printf '%u %g\n' 2>/dev/null | sort | uniq -c | sort -rn| The rows show | Who owns the answer | Stage |
|---|---|---|
| digits | nobody — the number is unclaimed, and the next useradd -r will take it | 2 |
| a service account’s name | the target’s /etc/passwd, and the order the packages were installed in | 2 |
root root and nothing else | either the source really was all-root, or an archive whose ownership was flattened before it got here | 1 |
| one non-root name, everywhere | the account the restore ran as | 3 |
The half-restore that authenticates everybody and gives them the wrong files
Human accounts are sharper, because their identity is split across files that are keyed differently. /etc/passwd maps name to number; /etc/shadow is keyed by name alone and holds no number at all. Restoring one without the other does not fail. It succeeds in a way you have to go looking for. Three users on a rebuilt machine whose numbers were allocated in a different order, old passwd restored over the new one:
$ ls -ldn /home/* $ ls -ld /home/*
drwxr-x--- 2 3002 3002 /home/alice drwxr-x--- 2 bob alice /home/alice
drwxr-x--- 2 3003 3003 /home/bob drwxr-x--- 2 carol bob /home/bob
drwxr-x--- 2 3001 3001 /home/carol drwxr-x--- 2 alice carol /home/carolSensible modes, every owner wrong and every group right. Everybody can log in; nobody can read their own mail. The other half-restore — old shadow, new passwd — is quieter still: because shadow is keyed by name, everyone authenticates correctly with their old password and the file ownership on disk is what is broken. The tool you would reach for has nothing to say: pwck prints no changes and exits 0, because it checks those files against each other rather than against the filesystem. Restore passwd, shadow, group and gshadow together or not at all — they are one document in four files.
Which is why the reflex to follow every restore with a chown is worth resisting. Given that the default is name mapping, a blanket chown -R www-data:www-data /var/www/example.com re-asserts something that is already right, over one directory out of a machine — and the recursive chmod that usually follows it is the worked diagnosis of Permissions and Privilege, Properly: an ACL mask lowered by something asserting a mode. Restore as root, keep the names, then ask find which numbers failed to resolve.
Verify: run the find … -printf '%u %g\n' line above on the restored machine. Every row that is digits rather than a name is a file with no owner, and a number waiting to be reissued.
Stage 3: The account that ran the restore
This stage is in nobody’s article and owns more measured failures than any other here. Every guide is careful about which account takes the backup; almost none says which account performs the restore. The restoring account rewrites two things, and cannot write a third.
Ownership. Only a process privileged enough to give a file away can restore somebody else’s ownership. Everybody else gets their own, on every file, with no message. The same account also cannot write security.capability at all, or create device nodes; both are skipped rather than refused.
The mode, through the umask. GNU tar’s -p is the default for root and is not the default for anybody else; non-root extraction subtracts the umask from every mode in the archive. One archive, one non-root account, three ways:
archive contains: -rwsr-xr-x root/root suidprog
umask 022, no -p -rwxr-xr-x backupop backupop # setuid silently dropped
umask 077, no -p -rwx------ backupop backupop # and every other file 0600
umask 077, with -p -rwsr-xr-x backupop backupop # a setuid-BACKUPOP binary
restored by root -rwsr-xr-x root root # correctRead the third row twice. -p as a non-root user does not restore a setuid-root binary; it creates a setuid binary owned by whoever ran the restore. If that is a deploy account with a key on three other machines, the restore has manufactured a privilege escalation out of a backup, and every command in it exited 0.
And partial success is the dangerous shape, on the recipe this site itself recommends. Automated Backups tells you to give the remote backup account only what it needs — good advice. Here is that account backing up /etc/ssh:
rsync: [sender] send_files failed to open "/etc/ssh/ssh_host_ed25519_key": Permission denied (13)
rsync error: some files/attrs were not transferred (see previous errors) (code 23)
$ ls /backup/etc/ssh/
moduli ssh_config sshd_config ssh_host_ecdsa_key.pub
ssh_host_ed25519_key.pub ssh_host_rsa_key.pubThree public host keys copied, three private ones omitted, and the directory looks populated. The signal is exit 23 in the middle of a nightly cron job that has been “working” for a year. It means some files were not transferred, and it is the most under-read number in this subject: a job exiting 23 every night has a hole in it that is the same shape every night.
Verify: id, before you type the restore command. Then check your backup job’s recorded exit status, not its log. If it is 23, something has been skipped every night, and the two commonest candidates are private keys and device nodes.
Stage 4: The metadata beside the bytes
Beside a file’s contents sit its ACL, its extended attributes, its capabilities and, on a labelled system, its SELinux context. Every tool has to be told to carry them, and the flags are asymmetric in a way almost no documentation records. Here is the answer everybody gives, put through a binary with cap_net_bind_service and then run (cp -a keeps it too):
as deployed: bind80 cap_net_bind_service=ep
run as an ordinary user: listening on 127.0.0.1:80 exit=0
(a) tar -cf / tar -xf cap: lost bind(127.0.0.1:80) failed:
Permission denied (errno 13)
(b) tar --acls --xattrs --selinux, both ends cap: LOST Permission denied (errno 13)
(c) tar … --xattrs-include='*' on EXTRACT cap: kept listening on 127.0.0.1:80
(d) rsync -aH (the usual backup flags) cap: lost Permission denied (errno 13)
(e) rsync -aHAX cap: kept listening on 127.0.0.1:80
every one of those five restores exited 0Row (b) is the finding. tar --acls --xattrs --selinux on both ends restores the ACL, restores the user.* attribute, and loses every file capability on the machine. Not because the capability is missing — stage 1 showed the bytes are in the archive. GNU tar’s own manual says why, and says it as a feature:
By default, when
--xattrsis used, all names are stored in the archive (with--create), but onlyusernamespace is extracted (if using--extract).extracting those (often security related) attributes on a different system than originally archived can lead to extraction failures, or even misinterpretations.
GNU tar deliberately declines to restore security attributes onto a different system — which is the definition of a restore. The flag that overrides it lives on the extract: --xattrs-include='*'. This is not news and not a regression: Red Hat has had it open since 2012, and Relax-and-Recover — a dedicated bare-metal restore product — shipped the wrong flags for thirteen months. The same rule covers SELinux labels, since they live in the same security.* namespace; that is documented rather than measured here, on a lab machine running AppArmor with no label to lose.
The preservation matrix
Eight round trips, one tree, measured on GNU tar 1.35, rsync 3.2.7, coreutils 9.4 and restic 0.16.4. Find your own row.
| Round trip | Owner by name | Mode + setuid | ACL | xattr + capability | Hard links | Sparseness |
|---|---|---|---|---|---|---|
tar --acls --xattrs --selinux | yes | yes | yes | user.* only — capability lost | yes | no |
tar + --xattrs-include='*' on extract | yes | yes | yes | yes | yes | with -S |
rsync -a | yes | yes | no | no | no | no |
rsync -aH | yes | yes | no | no | yes | no |
rsync -aHAX --sparse | yes | yes | yes | yes | yes | yes |
cp -p | yes | yes | yes | no | n/a | yes |
cp --preserve=xattr | no | no | no | yes | n/a | yes |
restic | yes | yes | yes | yes | yes | no — needs restore --sparse |
Two rows of that table invert what the flag names suggest, and both cost people ACLs. cp -p preserves an ACL and cp --preserve=xattr does not — ACLs travel with the mode in coreutils rather than with the extended attributes, so the option whose name says xattr is the one that will not carry them. File capabilities need the same suspicion: they live in an extended attribute too, they are dropped just as quietly, and auditing setuid binaries without also running getcap audits half the system. Permissions and Privilege, Properly is the page for what these things are; this one is about which of them survive being copied.
And rsync -aH is what Automated Backups prints. It is a good choice for a snapshot tree — --link-dest needs the hard links — and it is two letters short of carrying the attributes. One asymmetry for anybody who mixes tools: bsdtar restores security.capability from GNU tar’s own archive with no extra flags.
Verify: getcap -r /usr /opt /srv 2>/dev/null on the restored machine, and the same on the original while you still have it. Any binary on one list and not the other is a service that will fail at some future restart with a permission error on a file whose mode bits are visibly correct.
Stage 5: Links, holes, and the things that are not files
This is where a restore needs more disk than the machine it came from, and where the number in your backup report was never the number that mattered.
Sparseness. A file with a declared size of one gibibyte and four kilobytes of data in it occupies four kilobytes. Nothing about that is in the file; it is the filesystem’s arrangement of it, and every tool must be told to look. One directory that du reports as 21M:
tar -cf 1,094,717,440 bytes # about fifty times the disk it came from
tar -cSf 20,981,760 bytes # bsdtar needs no flag and matches this
restored, du of the tree:
tar -xf 1.1G rsync -a 1.1G rsync -aH 1.1G
tar -cSf 21M rsync -aH --sparse 21M cp -a 21M cp -r 81MTwo things fall out. rsync -aH — a snapshot backup’s flags — writes out every hole, so the backup of a virtual-machine image is the full declared size and so is the restore. And the number in your backup report is probably the apparent size: restic’s own output called a 40K directory processed 5 files, 128.026 MiB.
Hard links. Four names on one inode. GNU tar preserves them by default and always has; the tool that does not is the one this subject actually uses. Distinct inodes among the four names after each round trip: tar, bsdtar, cp -a and rsync -aH all give 1; rsync -a and cp -r give 4. The failure is not the disk space, though 8.1M becoming 529M is real — it is that the identity relationship is gone, and for a --link-dest snapshot tree that is the whole design.
The things that are not files. Fifos and device nodes come through tar, rsync -a (because -a includes -D) and cp -a perfectly, and are skipped by anything that drops -D or runs unprivileged, with skipping non-regular file. Sockets are the exception, and the interesting one:
tar ./app.sock: socket ignored
bsdtar ./app.sock: pax format cannot archive sockets
rsync -a, cp -a: the socket file is recreated
connecting to the restored socket:
after tar: [Errno 2] No such file or directory
after rsync -a: [Errno 111] Connection refusedA restored socket file is worse than a missing one. Those errors are the two halves of The Life of a Unix Socket‘s hinge, and that page measured the other end: a stale socket path gives EADDRINUSE to bind(), so the daemon that starts after your restore cannot take its own name back, and complains about an address already in use on a machine where nothing is listening.
Verify: du -sh and du -sh --apparent-size on the backup, before you need the space; if they differ by an order of magnitude your restore needs the larger number unless the command carries --sparse or -S. Then find / -xdev -type s on the restored machine, and delete what you find before starting the services.
Stage 6: What the walk could not see
Everything so far has been about a tool that walks a filesystem. This stage is about the two things such a tool cannot see at all: what is outside the tree, and what was moving outside the instant. One blindness is spatial and one temporal, and both produce restores that come up, run, and are wrong.
Outside the tree
The partition table lives in the first thirty-three sectors of the block device. Anything that walks a filesystem reads through the filesystem and never goes near them — and the filesystem UUID is the same problem with a sharper edge, because your machine names it in a file that is in the backup:
$ grep -a -c '730784A8-B61E-4AF5-A455-730370DEF1F5' contents.tar # the GPT label-id
0
old filesystem UUID : e5035ff4-1bfc-4d68-8d21-717e462be937
new filesystem UUID : a6627e08-90cf-4c79-934b-d51f58c3379a # after mkfs on the rebuild
$ grep UUID /etc/fstab # restored, verbatim
UUID=e5035ff4-1bfc-4d68-8d21-717e462be937 /srv/appdata ext4 defaults 0 2
the only backup of a partition table is one you took on purpose:
sfdisk -d /dev/sda > /backup/sda.table # restore: sfdisk /dev/sda < …The only copy of the old UUID anywhere in your backup is the line that refers to it. The thing it referred to was destroyed when the disk was rebuilt; the reference came back intact. That is From Power-On to a Login Prompt‘s worked diagnosis by another road. LVM’s volume-group metadata is the same category, with its own vgcfgbackup and vgcfgrestore. The LUKS header is the same category again, and the sharpest of the three.
Taking a header backup and keeping it somewhere other than the disk it came from is covered in Disk Encryption on a Laptop. What matters at restore time is what that file actually is:
format with passphrase A ; luksHeaderBackup -> header.img (16,777,216 bytes)
rotate to passphrase B ; A now refused, B accepted
months later, restore the header backup:
$ cryptsetup luksHeaderRestore … ; echo "exit=$?"
exit=0
passphrase A opens it? YES <- the one you rotated out is live again
passphrase B opens it? No key available with this passphrase.A header backup is a snapshot of the key slots as they were on the day you took it. Restoring it un-revokes every passphrase you have retired since and revokes the one you currently have, with exit 0 and no output. So the file is not merely as sensitive as the passphrase: it is a copy of one you have already decided to revoke. Still take it — the alternative is an unrecoverable disk — and re-take it after every rotation, destroying the old one.
Outside the instant
The standard warning about databases is that a copy taken while one is running may not restore. You will find it in most backup documentation, and in A Second Machine here. It warns about the outcome that announces itself, and says nothing about the one that does not. A complete hot copy of a running PostgreSQL data directory — rsync -aH straight over the top of it, mid-transaction — is crash-consistent:
LOG: database system was not properly shut down; automatic recovery in progress
LOG: redo done at 0/2A66658
LOG: database system is ready to accept connections
rows in the live database at the end : 20183
rows in the restored copy : 20105
# the same copy, with the sensible-looking exclusion --exclude='pg_wal/*'
LOG: invalid checkpoint record
PANIC: could not locate a valid checkpoint record
LOG: startup process was terminated by signal 6: AbortedThe complete copy is not corrupt. It is a valid database at a moment nobody recorded — and two copies of the same live database taken seconds apart recovered to two different log positions, both reporting success. The one that genuinely will not start is the incomplete one, made by an exclude list somebody wrote to avoid backing up logs.
The deterministic version is SQLite, which most self-hosted applications actually run and which backup guides rarely think to mention. In WAL mode the committed data lives in a sidecar file until a checkpoint moves it. Back up the main file, the way any file-level backup with an exclude list does:
live: 5,010 committed rows app.db 8,192 bytes app.db-wal 127,752 bytes
restored: PRAGMA integrity_check -> ok
SELECT count(*) -> 10Five thousand committed transactions gone, and the database says it is fine. One sentence for the whole subsection: dump databases rather than copy their files — not because the copy will fail, but because it will succeed at a moment nothing wrote down. The per-engine detail belongs elsewhere: Running PostgreSQL Properly, etcd, Honestly, and for SQLite sqlite3 app.db ".backup out.db" against the live file rather than a copy of it.
One last invisible thing. Timestamps mostly survive a restore; birth time never does. stat %w on a restored file is the moment of the restore, under every tool here. So a restored machine holds two classes of file — those carrying the original’s modification time and those carrying the restoring machine’s clock — and nothing distinguishes them. If that clock was wrong, every file the restore created is stamped wrong: How Time Works on a Linux Machine, including why timedatectl will insist it was fine.
Verify: sfdisk -d each disk and blkid the machine, and put both in the backup as text. Then compare /etc/fstab‘s UUIDs against blkid before the first reboot, not after it.
Stage 7: The other copy, and everyone who cannot tell them apart
Almost every account of this subject ends when the service comes back. That is where the mechanism ends; the reader’s problem ends somewhere else. The standard migration advice, including ours, is to leave the old server running for a week as your rollback — which is right, and which carries a consequence worth making explicit. A restore does not move a machine. It makes a second one — and while the first exists there are two things on the network claiming to be the same host, and every party who has to tell them apart is somewhere you cannot reach from either console.
The SSH host keys. Which way this breaks depends on which of our two pages you followed — Automated Backups includes /etc/ssh and Server Migration does not — and both directions are wrong differently:
restore without /etc/ssh: every client gets
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
Host key verification failed.
restore /etc wholesale: the old identity comes back, and so does every
client's trust in it — including the clients of the machine still runningThe first is loud, alarming and harmless: a warning that reads like a break-in is the correct response to a machine that genuinely is not the one you talked to yesterday. The second is silent, and is the one to think about. Two hosts now present the same key, and no client anywhere can distinguish them. Neither machine can see the collision, because the collision is not on either machine — it is in every known_hosts file in the organisation.
machine-id. The identifier almost everyone treats as cosmetic:
/etc/machine-id c1d60f590676409fadd11ab338e333b8
/var/lib/dbus/machine-id c1d60f590676409fadd11ab338e333b8
journal directory /var/log/journal/c1d60f590676409fadd11ab338e333b8
# an application-specific ID, derived from it by HMAC — deterministic
app A -> 5e4ec84094d240ba9c0b1dbc7b281e2d
# the same one, after another machine's /etc/machine-id is restored over it
app A -> 5352f34ec1f448fd8500b79d89522045Every derived identifier moved — which means the reverse: two machines carrying the same /etc/machine-id compute the same value for every one of them, with nothing to compare against. It also names the journal directory.
And the first-boot rules are a trap a restore walks into. machine-id(5) says an image for more than one machine should carry the file “either missing or an empty file”, and that a missing file means first boot while an empty one does not — so a restore that writes a populated machine-id onto a fresh install suppresses every ConditionFirstBoot=yes unit, and nothing reports it. Quoted from the manual page; no unit was watched being skipped here.
So the last step of a restore is not a check on this machine. It is a list of the parties now holding a stale belief about it, all of them elsewhere: every client’s known_hosts, the monitoring system that keys on hostname or machine ID, the log aggregator, anything that issued a licence or a token to this host, and whatever your infrastructure uses for a lease or a node identity. Not one of them is readable from the console you are sitting at.
Verify: you cannot, from here — that is the finding. What you can do is decide, deliberately, which identity this machine has. If it replaces the original: keep the host keys and the machine-id, and make sure the original is off rather than “left running for a week”. If it is a second machine, or a rebuild while the first is still alive: regenerate the host keys and truncate /etc/machine-id to empty — empty, not deleted and not a fresh value typed in by hand — and let systemd populate it at boot.
A worked diagnosis: the restore that worked, and everyone who could log in
A small application server loses a disk. It is rebuilt from the same distribution, the packages are reinstalled, and the nightly rsync -aH snapshot is restored over the top of it as root. The site serves, diff -r against the snapshot is clean, everybody logs in with their own password. It is called done. Over the following fortnight four things happen, and nobody connects them:
- The disk fills. A media directory that was thirty-six kilobytes is now two hundred and fifty-seven megabytes, and nothing has been uploaded.
- A nightly job that binds a low port stops working:
Permission deniedon a binary whose mode bits are visibly correct and which runs by hand undersudo. - The deploy user can no longer edit the files the web server writes. It could last month, and its group memberships have not changed.
- A partner’s CI job starts refusing to connect. Their engineer says the host key changed.
Every obvious hypothesis fails: nobody deployed, the package versions match, ls -l on the binary is right, and the four symptoms have nothing to do with each other. Run the hinge — the snapshot is still on the backup host, so both halves exist:
$ rsync -n -i -aHAX --checksum /mnt/backup/latest/srv/ /srv/
.f........x srv/bin/collector
.f.......ax srv/www/uploads/
.f........x srv/bin/tickd
hf srv/media/frame-0001.png => srv/media/frame-0000.pngFour lines, and the letters are the diagnosis. Three x: extended attributes present in the backup and absent on the machine — stage 4, and the missing one on collector is security.capability, which is why a binary with correct mode bits cannot bind port 80. One a: an ACL on the uploads directory, the named entry that let the deploy user write files the web server owns, gone with the same flags. One h: hard links that came back as copies, which is stage 5 and is the disk. The cause is a single command, and it is the one almost every backup script is built on: rsync -aH carries ownership, modes and hard links and does not carry ACLs, extended attributes or sparseness. The backup had all three; the restore declined them.
The fourth symptom is not in that output and never will be. The host key is stage 7, and stage 7 is not inside the tree — /etc/ssh was not in this backup’s source list, so there is nothing to compare and nothing to report. The hinge is silent about the one symptom a stranger reported, which is worth saying out loud: an empty branch means “nothing I can compare differs”, not “nothing differs”. The rebuilt machine generated fresh host keys at first boot, which is correct behaviour and which every client experiences as a security warning.
Three things generalise. The four symptoms arrived on four different days and belonged to three stages, which is why nobody connected them — a restore’s failures are not simultaneous, because each one waits for something to use the thing that was lost. Every command in the restore exited 0, and each was describing the copy rather than the machine. And the fix for three of the four was two more letters; the fix for the fourth was to talk to somebody outside the building.
Symptoms, and which stage they belong to
| Symptom | Stage | What is actually true |
|---|---|---|
The restore finished, diff says identical, something is still wrong | all | diff compares bytes. Six of the seven stages are not bytes |
ls -l shows numbers where names used to be | 2 | The number has no name here, and the next useradd -r will claim it |
| Everyone logs in, and nobody owns their own home directory | 2 | /etc/shadow is keyed by name and /etc/passwd by number; half a restore of those files is silently wrong |
Every restored file is owned by whoever ran the restore, or is 0600 | 3 | Only a privileged process can give a file away, and non-root extraction subtracts the umask |
| The backup job has been exiting 23 for months | 3 | Some files were never transferred. Usually private keys or device nodes |
A service fails with Permission denied on a binary whose mode is right | 4 | A file capability was dropped on extraction |
tar --acls --xattrs --selinux and the capability is still gone | 4 | GNU tar extracts only the user namespace by default. --xattrs-include='*', on the extract |
| An ACL that was set for years is not there any more | 4 | rsync needs -A; cp -p keeps ACLs and cp --preserve=xattr does not |
| The restore needs far more disk than the machine it came from | 5 | Sparse files written out in full, or hard links restored as copies |
| A daemon will not start: address already in use, nothing is listening | 5 | A socket file was recreated by the restore. Delete it |
| Emergency shell after the first reboot, or a database missing recent rows | 6 | An fstab UUID destroyed with the disk; or a hot copy that is valid at a moment nobody recorded |
| Two machines are indistinguishable to everything on the network | 7 | The host keys and machine-id were restored, and the original is still running |
Advice that has expired
| Still repeated | What to say now |
|---|---|
tar --acls --xattrs --selinux preserves everything | It preserves ACLs and user.*. Add --xattrs-include='*' on the extract or every file capability is dropped. Open at Red Hat since 2012 |
rsync -a means “everything” | rsync’s own help: archive mode is -rlptgoD (no -A,-X,-U,-N,-H) — and --sparse is a sixth omission not even in that list |
SHOW MASTER STATUS, RESET MASTER, PURGE MASTER LOGS | Removed in MySQL 8.4 and renamed around BINARY LOG. Every pre-2024 MySQL backup runbook has at least one of them. etcdctl snapshot restore went the same way in etcd 3.6.0 |
| “Test your restores”, as the whole of the advice | Every tool in this article exited 0. Testing a restore means comparing it against the backup with something that can see attributes, then checking the things that are in neither |
How to tell whether a page about restoring is worth reading
Every error on this page falls out of one model, and once you have it you can predict what any given article will be missing:
A restore is the backup command run backwards. Whatever the backup preserved, the restore returns — so the only thing that can go wrong is that the backup did not run.
Watch it generate. If a restore is the backup inverted, flags are symmetric — so almost nothing records that --xattrs means “store everything” on create and “restore user.* only” on extract. The account is irrelevant — so almost nothing says root gets -p for free and everybody else gets the umask. The target machine is not a party — so its /etc/passwd goes unmentioned, and the sentence everybody repeats about ownership has the names and the numbers the wrong way round. Verification means “are the bytes the same” — so diff and tar --compare are both offered as proof, and both are blind to the attribute classes the same documents spent paragraphs explaining. A hot database copy is either corrupt or it is not — so the warning is always that it “may not restore”, and never that the dangerous outcome is that it does. And one machine goes away and another appears — so the fact that both exist at once goes unsaid, even in the advice that tells you to keep the first one running for a week.
The one-question test. Does the page name a single thing the restore decides that the backup did not? If every decision on it belongs to backup time, the author has only ever restored onto the machine they backed up.
The absences worth noticing. Words that never appear on such a page: --numeric-ids, --xattrs-include, umask, machine-id, security.capability, sfdisk -d, exit 23, “which account ran the restore”, “the old machine is still running”. Present instead, in all of them: encryption, deduplication, retention, the 3-2-1 rule, off-site, and “test your restores”.
And the error has visibly been half-fixed. It is now widely understood that a filesystem backup is not a machine image — that it holds no partition table and no bootloader, and that bare-metal recovery needs something else alongside it. Nearly every tool’s documentation says so somewhere, usually under what it is not for. That is the spatial half, and it is the only half that has landed. The other two have not: that a filesystem backup is not a filesystem either, and that what it loses is decided at the far end rather than at the near one.
If you keep one sentence: a restore is not a return, it is a duplication — and the moment there are two, the only parties who can tell them apart are outside the machine.
Before you call the restore done
rsync -n -i -aHAX --checksum /backup/tree/ /restored/tree/ # 1-6: the hinge
getfattr -Rd -m 'security|system' /backup/tree/ | head # 1: did the backup have it
find /srv /home /var/www -xdev -printf '%u %g\n' | sort | uniq -c # 2: names or digits
getcap -r /usr /opt /srv 2>/dev/null # 4: against the old list
du -sh --apparent-size /backup/tree/ ; du -sh /backup/tree/ # 5: before the disk fills
find / -xdev -type s # 5: sockets the restore made
blkid ; grep UUID /etc/fstab # 6: before the first reboot
ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub ; cat /etc/machine-id # 7: who is thisTwo decisions are yours rather than a command’s. Run the restore as root — it is the recommendation with more measured evidence behind it than anything else here. And decide which machine this is before you restore /etc: a replacement keeps the host keys and the machine-id and requires the original to be off; anything else regenerates the keys and leaves /etc/machine-id empty.
Related reading
- Automated Backups — the page above this one. Its step 7 is the procedure this page finishes
- Permissions and Privilege, Properly — what ACLs, the mask and file capabilities are, and the order the kernel checks its six layers in
- Running PostgreSQL Properly — the eight stages between a
COMMITand data you could still restore a year later, including the restore itself - How Time Works on a Linux Machine — why a restored file’s timestamps are a receipt for the restoring machine’s clock, and why that clock may be wrong while the machine insists it is not