At some point the single server starts to feel fragile, and the instinct is to add another one. That instinct is often right and frequently premature, because a second machine solves one problem — the machine dying — while adding several that did not exist before.
This guide is half decision and half implementation, in that order, because the decision is the part people skip.
What this does not do. A second machine is not a backup — replication copies your mistakes faithfully and instantly, so a dropped table is dropped on both. It does not protect against a bad deploy, a compromised credential, or an expired domain, all of which take down two servers as easily as one. And it does not cover your provider having a bad day unless the second machine is somewhere else entirely. What it protects against is precisely one thing: this particular computer stops working.
Step 1: work out what your downtime actually costs
Two numbers, written down honestly.
How long can you be down? Not the aspirational answer. If the site is offline for four hours on a Tuesday afternoon, what actually happens — lost orders, an awkward email, or nothing anyone notices?
How much data can you lose? If you restore from a backup taken at 3am and it is now 4pm, is that an inconvenience or a catastrophe?
| Tolerable downtime | What you need |
|---|---|
| A day or more | Good backups. Nothing else. |
| A few hours | Backups plus a rehearsed rebuild — the rest of this guide is optional |
| Under an hour | A warm standby you can switch to by hand |
| Minutes, unattended | Automatic failover, and everything it brings with it |
Most self-hosted setups and a great many small businesses sit in the first two rows. If that is you, the honest advice is to stop reading and go and test a restore instead — it is cheaper, simpler, and protects against far more of what actually goes wrong.
Verify: time yourself rebuilding the server from scratch, from your backups, onto a fresh VPS. If that takes ninety minutes and you can tolerate two hours, you already have your answer and it costs nothing per month.
Step 2: make the rebuild fast before you make it redundant
Everything that makes a rebuild fast also makes a second machine easier, so this work is never wasted.
- The configuration is in git — compose files, Ansible playbooks, unit files. Nothing important should exist only on the server.
- The data is separate from the machine — a known set of volumes and database dumps, not files scattered through the filesystem.
- The rebuild is written down and someone other than you could follow it.
- DNS TTLs are already low, or you can lower them — see the DNS guide, because this is where a slow recovery usually comes from.
Verify: do it. Provision a new machine, restore, and point your own /etc/hosts at it to confirm the site works. Then destroy it. That exercise finds the undocumented step every time — usually a credential, a cron job, or a file somebody put in /opt two years ago.
Step 3: choose the shape of the second machine
| Shape | How it works | Recovery | Cost |
|---|---|---|---|
| Cold standby | Backups, plus a machine you build when needed | Hours | Almost nothing |
| Warm standby | Second machine built and current, data replicating, not serving | Minutes, by hand | A second server |
| Active-passive | As above, with automatic failover | Seconds to minutes | Plus the failover mechanism |
| Active-active | Both serving, load balanced | None visible | Plus shared state and session handling |
Warm standby is the sweet spot for almost everyone who has decided they need more than backups. It gives you most of the benefit, it is comprehensible, and — importantly — a human decides when to switch, which eliminates the entire category of failures where the automation fails over for a reason that was not a real outage.
Active-active is a different kind of project. Sessions, uploads, caches and scheduled jobs all have to be rethought, because “which machine has the file the user just uploaded” becomes a real question. Do not drift into it by accident.
Put the second machine somewhere genuinely separate — a different datacentre at minimum, and preferably a different provider. Two VMs on the same host protect you against remarkably little.
Step 4: replicate the data
This is the hard part, and it splits by data type.
Databases should use their own replication rather than file copying. PostgreSQL streaming replication gives you a standby that is seconds behind and can be promoted:
# on the standby, from a base backup of the primary
pg_basebackup -h primary.example.com -U replicator \
-D /var/lib/postgresql/16/main -R -P
# check the lag, on the primary
psql -c "SELECT client_addr, state, replay_lag FROM pg_stat_replication;"Never copy a running database’s data directory with rsync. You get a torn, inconsistent snapshot that may restore and may not, and you will find out which at the worst possible moment.
Files — uploads, media, anything the application writes — can be replicated more simply:
# scheduled one-way copy; --delete makes the standby match, so be certain
rsync -az --delete /var/lib/myapp/uploads/ standby:/var/lib/myapp/uploads/Or Syncthing for continuous sync, with the same warning as always: it propagates deletions, so it is replication and not a backup.
Verify: write a record on the primary, and read it on the standby within a minute. Then check replication lag has a monitor on it — silently-stopped replication is the classic failure here, and you discover it at the moment you promote a standby that is four weeks old.
Step 5: decide how traffic moves
Three mechanisms, in increasing order of complexity.
DNS. Change the A record to the standby’s address. Simple, works across providers, and bounded by your TTL — which is why the DNS guide insists on keeping it low if you might need this. Some resolvers and some applications cache beyond the TTL, so treat it as “most traffic within the TTL” rather than a clean switch.
A floating IP. Most providers offer an address you can reassign between their machines through an API, usually in seconds. This is much cleaner than DNS — no caching involved — and it only works within one provider, which is the trade.
# the shape of it, whatever your provider's CLI is called
provider floating-ip assign --ip 203.0.113.50 --server standbyA load balancer in front of both. The provider’s, or your own. This is the active-active answer and it moves the single point of failure to the balancer, which is usually a better place for it.
Whichever you choose, write the failover as a script rather than a list of steps. At the moment you need it you will be stressed and possibly on a phone.
#!/usr/bin/env bash
set -euo pipefail
# promote-standby.sh - run ON THE STANDBY, deliberately
sudo -u postgres pg_ctl promote -D /var/lib/postgresql/16/main
sudo systemctl start myapp
provider floating-ip assign --ip 203.0.113.50 --server "$(hostname)"
echo "Promoted. The old primary must NOT be started again without a rebuild."Verify: run the failover on a normal Tuesday. Not as a thought experiment — actually fail over, serve from the standby for a day, then fail back. Anything you have not tested does not work, and this is the least forgiving thing on the site to discover that about.
Step 6: understand split brain before you automate anything
Automatic failover introduces a failure that cannot happen with one machine. The standby decides the primary is dead and promotes itself. But the primary was not dead — the network between them was. Now two machines both believe they are authoritative, both accept writes, and the two datasets diverge.
Reconciling that afterwards ranges from tedious to impossible. It is genuinely worse than the outage would have been.
Real cluster software solves this with quorum — an odd number of voters, and a node that cannot see a majority refuses to act — and with fencing, where the winner forcibly powers off the loser. Both are why proper high availability wants three machines rather than two. Two nodes cannot form a majority; each simply sees the other missing.
The practical conclusion for a small setup: keep a human in the loop. Manual promotion, from a script, after a person has confirmed the primary is genuinely gone. You lose a few minutes of recovery time and you remove the entire failure mode.
And after promoting, treat the old primary as contaminated. Do not start it up to see if it comes back. Rebuild it as the new standby.
Before you call it done
- You have written down your tolerable downtime and data loss, and the setup matches them
- Backups still exist and are still tested — the standby did not replace them
- The second machine is in a different datacentre, ideally a different provider
- Database replication uses the database’s own mechanism, not file copying
- Replication lag is monitored, with an alert when it stops
- Failover is a script, stored somewhere you can reach without the primary
- You have failed over for real, and failed back, at least once
- DNS TTL low enough for your recovery target, if DNS is the mechanism
- Promotion requires a human decision, unless you have three machines and real quorum
- Both machines are patched, monitored and hardened — a neglected standby is not a standby
Related reading
- Automated backups — what most people need instead of this
- Getting a domain and its DNS right — TTL, and why it decides your recovery time
- Putting a database somewhere sensible — the thing hardest to replicate
- Server migration — the same moves, planned rather than urgent
- Monitoring without a full stack — how you find out the primary is gone