An etcd restore reports success. Every key is back. Hours later a controller is reconciling objects that do not exist, a Job is being recreated, an operator’s cache disagrees with kubectl get, and nothing correlates with the restore, because the restore worked. The mechanism is a number nobody looks at, and the restore command in the Kubernetes documentation does not set the two flags that prevent it. Almost nobody chooses etcd. It arrives underneath Kubernetes, works for a year or two without being thought about, and then one day it is all that stands between you and a cluster you cannot restore. This page is for that day, and for the fortnight before it.
Kubernetes, Honestly left etcd out and ended its second stage one clause early: the API server is the only component that talks to etcd, which is why restoring etcd is the only restore that exists. True, and what it does not say is that an etcd restore that reports success can quietly break every controller in your cluster. So this is not a tour of etcd. It follows one write through six stages and ends with the restore that gives you your data back and your revision numbers back, but not the same ones. Where etcd sits in a control plane is that page’s; why quorum needs an odd number is A Second Machine‘s; what an fsync is and why a client waits on exactly one of them is Running PostgreSQL Properly‘s and The Life of a Write‘s. If your control plane is managed, EKS, GKE or AKS, there is no etcd to ask and most of this page is not relevant to you.
The six stages of a write. Every section below is named after one.
- Which member answered — and whether it was allowed to.
- The leader, the log, and the one
fsync— the only thing the client actually waits for. - Apply — the backend, and the number that gets issued.
- Read and watch — and what it costs to be certain.
- Compact, defrag and the quota — the deliberate destruction of history.
- The restore, and the revision it reuses.
The diagnostic hinge: a revision is not a name. It is a position in one particular cluster’s history — and “one particular cluster” is a thing that can be replaced underneath you without anybody being told.
Every answer etcd gives you is signed. Read the signature before you believe the answer:
etcdctl endpoint status -w json | jq '.[0].Status.header'
{
"cluster_id": 3867748490576809579,
"member_id": 6526328719644797682,
"revision": 2002,
"raft_term": 2
}Three of those are the ones to read, and the first is the question.
cluster_idis not the one your clients were talking to yesterday → you are downstream of a restore, whether or not anybody told you. Every client holding a revision is holding a number that now means something else, and it will not be told. Stage 6, and nothing else on this page will help until that is settled.- It matches, and reads at old revisions are refused (
mvcc: required revision has been compacted) → your history has been truncated on purpose, by something. Stage 5. The keys are fine; the history is not. - It matches, and
raft_termclimbs between checks → elections. Stage 2 — and read it with the worked diagnosis in hand, because etcd will tell you it is the disk and it usually is not. - Everything agrees and old revisions still read → your history is intact, and your problem is measured in milliseconds (stage 2) or in bytes (stage 5) rather than in revisions.
The signature is not a health check. It is an identity check. etcd will answer you confidently after it has become a different cluster, and that is the failure this page exists for. It is also worth knowing that endpoint status is a local call: it answers with no leader and no quorum, so the hinge still evaluates during the outage, when almost nothing else does.
Stage 1: Which member answered
You gave a client a list of endpoints. It picked one. Everything that follows depends on which one, and on what that member is allowed to do — and the client will not tell you any of it.
Send a write to a follower and it simply works. The etcd server forwards the proposal to the leader over the peer channel; the client library is not involved, and nothing in the response records that it happened:
etcdctl --endpoints=<a follower> put /k via-follower
OKThere is one member that behaves differently, and it is the one people put in endpoint lists by mistake. A learner is not a read replica. It refuses writes, and it refuses linearizable reads, with an error that at least names itself:
Error: etcdserver: rpc not supported for learnerAnd it serves serializable reads perfectly happily, from whatever it has caught up to. So a learner in a client’s endpoint list does not fail loudly; it makes some fraction of that client’s reads quietly stale, with no error and no metric. Learners arrived in etcd 3.4.0 and have behaved this way throughout.
The membership arithmetic, and the member that was never started
Quorum counts voting members in the membership list, not running processes. A member added with member add and never started counts immediately. A learner does not count at all.
That produces the failure that actually kills clusters, and it is not the one everybody warns about. The old advice — never add the replacement before removing the dead member — describes a guard rail that etcd has had since 3.3, in 2018: --strict-reconfig-check defaults to true and refuses the request outright:
# two of three members up
etcdctl member add spare --peer-urls=http://10.0.0.9:2380
Error: etcdserver: unhealthy clusterWhat is not guarded is adding a member to a healthy cluster and then failing to start it — a wrong peer URL, an image that will not pull, a firewall rule. The check passes, because at the moment of the check the cluster was fine. Then the arithmetic changes underneath you:
four healthy members, two more added and never started
-> 6 voting members, quorum 4, 4 alive
one member then dies
-> etcdctl member list Error: context deadline exceeded
-> etcdctl member remove Error: context deadline exceededRead the last two lines carefully, because they are the reason “just remove the dead members” is not a plan. Membership changes are Raft proposals. They need the quorum you have already lost. member remove is executable only while you do not need it.
Two smaller facts belong here. --max-learners defaults to 1, so you cannot stage two replacements at once (etcdserver: too many learner members in cluster). And when you do lose quorum, before you reach for --force-new-cluster on a live data directory — the one move with no undo — take the free snapshot. It works. That is stage 6’s material and it is the first thing to do, not the last.
Stage 2: The leader, the log, and the one fsync
The leader appends your write to its Raft log, replicates it, waits for a majority of members to have persisted it, and then applies it locally before answering. The shape is the same as any write-ahead log, and one correction to the usual summary matters here: the leader does not answer at commit, it answers after applying. Followers apply asynchronously, which is why a linearizable read served by a follower costs a round trip to the leader, and why, when quorum is gone, the survivor logs its failure by that name: waiting for ReadIndex response took too long, retrying. The second thing this stage owns is the reason it is in the article at all.
etcd_disk_wal_fsync_duration_seconds is not a disk metric
etcd’s own guidance is a p99 wal_fsync under 10 ms, and every etcd page on the internet repeats it. The number is fine. The instrument is not what it sounds like: it is a stopwatch started and stopped by Go code inside the etcd process, and a process that is not being scheduled cannot stop a stopwatch.
Here is one etcd leader with a CPU quota applied and nothing else changed — a Kubernetes limits.cpu: 50m — measured against fio writing to the same filesystem in the same minute:
mean wal_fsync | mean backend_commit | fio --fdatasync=1, same directory | |
|---|---|---|---|
| before | 0.533 ms | — | avg 206 µs, p1 101 µs |
| during | 7.84 ms | 9.28 ms | avg 250 µs, p1 102 µs |
A fifteenfold degradation reported by etcd, and no change at all in the disk. Both of the metrics everybody recommends agreed with each other and both were describing the scheduler. This is the worked diagnosis at the end of the page. Elections belong here too, and they are worth being able to recognise in a journal at three in the morning:
is starting a new election at term 2
became pre-candidate at term 2
became candidate at term 3
became leader at term 3Four lines, in that order, is an election. If raft_term in the hinge is climbing, this is what is in the log.
Stage 3: Apply, and the number that gets issued
Applying means writing into the backend — a bbolt file, one per member — and issuing a revision. This is the centre of the article, and the single most useful thing to hold about etcd is that the revision is the object and the key is an index into it.
There are four numbers, not one, and a single response contains all of them:
{"header":{"cluster_id":3867748490576809579,"member_id":6526328719644797682,
"revision":4,"raft_term":2},
"kvs":[{"key":"L2E=","create_revision":2,"mod_revision":4,"version":2,
"value":"dGhyZWU="}],"count":1}| Number | Scope | What it means |
|---|---|---|
header.revision | The whole cluster | The store’s current position. Every write of any kind advances it, including a delete. |
create_revision | One key | The revision at which this key was created. |
mod_revision | One key | The revision at which it was last modified. This is what a compare-and-swap compares. |
version | One key | A per-key counter, starting at 1. Nothing to do with the other three. |
The revision is not the Raft index. They are printed side by side and they diverge immediately: a cluster showing RAFT INDEX 25 had revision 7. The Raft index counts everything the log carries, including membership changes and internal traffic; the revision counts only changes to the key space. Anything comparing the two is confused.
One more correction to the usual summary of stage 2, because it lands here: an error return is not proof the write did not happen. Under a quota alarm, a Put can return ErrGRPCNoSpace and still be applied — the quota is checked at the API layer and again at the apply layer, and the apply layer raises the alarm without blocking the transaction. If you are writing retry logic against etcd, that is the case it has to survive.
Stage 4: Read and watch
Reads come in two kinds and the default is the expensive one. A linearizable read — the default — guarantees you see everything committed before the read began, and pays for it with a ReadIndex round trip to the leader. A serializable read is served straight from whichever member you asked, with no round trip and no guarantee about freshness:
etcdctl get /k # linearizable: costs a trip to the leader
etcdctl get /k --consistency=s # serializable: local, fast, possibly staleThe failure mode is the one from stage 1 in a different costume. On a cluster that has lost quorum, writes and linearizable reads both return context deadline exceeded — and a serializable read returns the last value it saw, with no warning that the cluster is dead. That is genuinely useful during an incident and genuinely dangerous in an application.
A watch is a stream, and it can be resumed from a revision. On a live cluster, resuming from a revision that has been destroyed fails exactly as it should:
$ etcdctl watch /a --rev=2
watch was canceled (etcdserver: mvcc: required revision has been compacted)
Error: watch is canceled by the serverLoud, specific, and the correct behaviour. Remember what it looks like, because stage 6 is about the case where the same request produces no error at all.
Two health endpoints, and they disagree with each other on purpose. On a member with no leader, serving neither reads nor writes:
curl -s -o /dev/null -w '%{http_code}\n' localhost:2379/livez # 200
curl -s -o /dev/null -w '%{http_code}\n' localhost:2379/readyz # 503
curl -s localhost:2379/health
{"health":"false","reason":"RAFT NO LEADER"}
# /readyz names the failing check:
[-]linearizable_read failed: context canceled
[+]non_learner ok [+]data_corruption ok [+]serializable_read ok/livez returns 200 through a total outage, and that is correct: it is a liveness probe, and restarting a member that has merely lost contact with its peers would make things worse. But any dashboard or uptime check pointed at /livez stays green while the cluster serves nothing. Point monitoring at /readyz, and read the body rather than the code.
Stage 5: Compact, defrag and the quota
Every write leaves the previous version of the key behind. Nothing removes those versions until something compacts, and compaction is not housekeeping — it is the deliberate destruction of the only reference your clients have. Stage 4’s error is what a client sees afterwards.
Compaction and defragmentation are two separate operations and the difference is the most misunderstood thing in the subject. Forty thousand revisions of one 1 KiB key, on a single member:
| DB SIZE | IN USE | NOT IN USE | the file on disk | |
|---|---|---|---|---|
| after the churn | 74 MB | 74 MB | 0% | 84,004,864 |
after compact | 74 MB | 234 kB | 100% | 84,004,864 |
after defrag | 25 kB | 16 kB | 34% | 32,768 |
Compaction frees space inside the file. Defragmentation gives it back to the filesystem. Note also that the file on disk and the number etcd reports are not the same — 84,004,864 bytes against a reported 73,691,136 — and it is the reported number the quota is enforced against.
Two things compaction does not do. It does not remove keys: a key last written long before the compaction point survives with its original value, readable, while every revision below the compaction point is gone. And compact(N) leaves revision N itself readable; it is N−1 that disappears.
Two numbers, and only one of them is your fault.
etcdctl endpoint status --cluster -w table
… │ DB SIZE │ IN USE │ PERCENTAGE NOT IN USE │ QUOTA │ …IN USEhigh → you have too much live data, or too much retained history. Compact. Defragmenting now buys nothing.IN USElow,DB SIZEhigh (NOT IN USEnear 100%) → you have already compacted and the file has not caught up. Defrag, one member at a time, and budget roughly thirteen seconds per 500 MB of member.- Both high and nearly equal → nothing is compacting at all. This is not a maintenance problem, it is an unconfigured one, and it is the branch missing from every other page on the subject.
That third branch is common because of a default nobody expects. --auto-compaction-retention defaults to 0s, and 0 means never. Under Kubernetes you never find out, because the API server runs its own compactor every five minutes against a top-level key of its own. Run etcd for anything else, follow etcd’s own documentation, and you have a database that grows until it stops.
Defragmentation is an outage, and it lies about failing
A member being defragmented serves nothing for the duration. On a 466 MB member with a reader issuing a linearizable get every 50 ms:
Finished defragmenting etcd member[127.0.0.1:2379]. took 13.347056448s
read latencies, ms: 5039 5038 3186 63 29 ... median 15The two 5,039 ms readings are not slow reads. They are etcdctl‘s own five-second command timeout: those reads failed. Which is why etcdctl defrag --cluster is a self-inflicted control-plane outage rather than a maintenance command: it iterates every member in sequence, each unavailable in turn.
And then the part that has quietly convinced a great many people their maintenance is broken:
$ etcdctl defrag
Failed to defragment etcd member[127.0.0.1:2379]. took 5.004255622s. (context deadline exceeded)
# and 6.88 s later, in the server's own log:
"msg":"finished defragmenting directory","current-db-size-bytes-diff":-227414016,
"current-db-size":"426 MB","took":"6.882079661s"etcdctl‘s default --command-timeout is five seconds, so on any database big enough to need defragmenting the client reports failure while the server succeeds and reclaims the space. A nightly defrag job that has “failed” every night for a year has been working every night for a year. Raise the timeout, or use etcd-defrag, the tool the Kubernetes documentation points at, which defaults it to thirty seconds and is also where the “defragment the leader last” advice comes from.
The quota alarm, and why the permitted operation is the wrong one
The backend quota defaults to 2 GiB. Cross it and etcd raises a NOSPACE alarm against the member that went over, enforced cluster-wide, and starts refusing writes with a gRPC ResourceExhausted. The alarm names a member ID, which is why endpoint status without --cluster will not tell you which member is the problem.
etcd’s documentation says the cluster then “only accepts key reads and deletes”, which reads like permission. It is not advice:
Under a NOSPACE alarm | What actually happens | |
|---|---|---|
| Read a key | ✓ | Works normally. |
| Delete keys | ✓ | Permitted, and it makes things worse. Deletes are writes; they add tombstone revisions. 6,000 deletes took the database from 16,842,752 to 17,195,008 bytes. |
| Put a key | ✗ | etcdserver: mvcc: database space exceeded |
| Grant a lease | ✗ | Also refused. Anything holding a lease is now on a clock it cannot renew. |
| Compact | ✓ | Works, and is the actual fix. Space comes back at compaction, never at deletion. |
| Defragment | ✓ | Works, and returns the file to the filesystem. |
| Clear the alarm | ✓ | Only etcdctl alarm disarm does this. Compacting and defragmenting back down to 24 kB does not clear it; puts keep failing until you disarm. |
The sequence, then, is compact → defrag → alarm disarm, in that order, and nothing before the last step will let a write through. One more thing worth knowing before you set a larger quota: there is a documented ceiling of about 8.6 GB, and exceeding it is a warning rather than a refusal — etcd logs quota exceeds the maximum value and starts anyway, which means a typo in that flag is a thing you find out about much later.
Stage 6: The restore, and the revision it reuses
Two things before the mechanism, because both are useful in the first five minutes of an incident. etcdctl snapshot save does not need quorum. On a cluster that could serve neither a read nor a write, where member list and member remove both timed out, snapshot save returned Snapshot saved at … and etcdutl snapshot status read it back cleanly. Your last-resort backup is available on a dead cluster, and taking it is free. Do that before --force-new-cluster, which has no undo.
And the command changed under everyone. etcdctl snapshot restore, etcdctl snapshot status and etcdctl defrag --data-dir were all removed in etcd 3.6.0 on 15 May 2025. They are etcdutl now. Kubernetes 1.34 was the first kubeadm to ship etcd 3.6, so this is the upgrade on which every saved runbook became a command that does not exist.
What a restore actually does
It builds a new cluster from the snapshot. Watch the cluster ID and the revision:
state at snapshot time:<br> clusterID=5920653729087557766 rev=2002 raftIndex=2009
(500 more writes happen; revision reaches 2503)
after etcdutl snapshot restore:<br> clusterID=13133426046165070218 rev=2002 raftIndex=6
keys written after the snapshot: Count 0New cluster ID. Raft index reset. Post-snapshot writes gone, which is expected and is why you took the snapshot. And the revision preserved at 2002 — which sounds like the considerate option and is the whole problem.
A client that was watching, or holding a resource version, resumed from revision 2600. Immediately after the restore it gets an error nobody has an alert for:
Error: etcdserver: mvcc: required revision is a future revisionThen the cluster does 700 ordinary writes, passes 2600 on its way back up, and the same request returns this:
$ etcdctl get --rev=2600 --prefix /
/new-00000000
/new-00000001
/new-00000002
$ etcdctl watch --rev=2600 --prefix /
PUT
/new-00000597
…
PUT
/new-00000599
(exit 0)No error. A clean stream of a different history, and exit zero. Compare the compacted watch in stage 4, which failed loudly and correctly. The revision number 2600 has been reissued, attached to content that has nothing to do with what the client saw at 2600 yesterday, and there is no field in the response that says so.
What that looks like operationally is a controller reconciling objects that do not exist, a Job being recreated, an operator’s cache disagreeing with kubectl get — hours after the restore, correlating with nothing, and not looking remotely like a restore problem.
The two flags
The same snapshot, restored with two more flags:
etcdutl snapshot restore snap.db \
--data-dir /var/lib/etcd \
--bump-revision 1000000 --mark-compacted
revision after restore: 1002002
data still there: /canary before-snapshot
read at revision 2600: Error: etcdserver: mvcc: required revision has been compacted
watch at revision 2600: watch was canceled (…required revision has been compacted)All the data, and every stale resume refused loudly instead of answered wrongly. --bump-revision takes a number of revisions rather than a revision, and etcd’s own recovery page suggests a billion to cover a week-old snapshot; --mark-compacted is what turns “future revision” into “has been compacted”, which is the error clients already know how to handle.
These landed in etcd 3.5.10, on 27 October 2023. etcd’s own recovery documentation calls bumping the revision “highly recommended … when using Kubernetes in general”. The Kubernetes documentation’s restore recipe does not mention either flag. k3s does not pass them either: its --cluster-reset-restore-path builds a restore config containing the snapshot path, the name, the output directories, the peer URLs and the initial cluster, and nothing else. A k3s restore reuses revisions. Restoring etcd is the only restore that exists. It is also the only restore that can succeed completely and take your controllers with it, and the difference is two flags that the instructions you will follow do not contain.
A worked diagnosis: the heartbeat that blamed the disk
Intermittent API server latency, and occasionally etcdserver: request timed out in its log. In etcd’s own log, steadily:
{"level":"warn","msg":"leader failed to send out heartbeat on time; took too long,
leader is overloaded likely from slow disk","heartbeat-interval":"100ms",
"expected-duration":"200ms","exceeded-duration":"194.834µs"}<br>{"level":"warn","msg":"apply request took too long","took":"100.94362ms",
"expected-duration":"100ms","request":"put:<key:\"/x9-00000266\" value_size:1024 >"}Every obvious observable says nothing is wrong. No elections — etcd_server_leader_changes_seen_total stayed at 1 across the whole incident, so the symptom everybody looks for is absent. No alarms, no NOSPACE, the database is 1.9 MB, all three members healthy, endpoint status clean. And the metric everyone reaches for agrees with the log: mean wal_fsync went from 0.533 ms to 7.84 ms, with a tail into the quarter-second buckets.
Except that fio, writing 2,300-byte blocks with --fdatasync=1 into the same directory during the same window, measured no change at all: p1 of 101 µs before, 102 µs during. The disk was fine the whole time.
The answer was one file:
$ cat /sys/fs/cgroup/cpu/etcdslow/cpu.stat
nr_periods 1787<br>nr_throttled 1785
throttled_time 2043991186061,785 of 1,787 scheduling periods throttled. The member was running under a CPU quota of 5 ms per 100 ms period — a Kubernetes limits.cpu: 50m. And the smoking gun is in the warning itself: etcd is complaining about a heartbeat that was late by 194 microseconds, and elsewhere in the same run by 16.656 microseconds, and naming the disk while it does it.
Where it comes from. kubeadm’s etcd static pod ships CPU and memory requests and no limits at all, which is correct. So this failure does not arrive with etcd. It arrives on the day somebody adds a LimitRange, or a vertical autoscaler, or a well-meant limits: block, to the namespace the control plane lives in.
And note where the hinge points. cluster_id matches, raft_term is not climbing, old revisions still read, so the hinge says “your history is intact, your problem is milliseconds” and sends you to stage 2. That is the right stage, and the cause is one layer below anything stage 2 can see. The hinge is a localiser, not an oracle; it puts you in the right room and does not open the cupboard.
The moral, and it generalises well past etcd: a latency you measure from inside a process includes the time the process spent not running. Everything etcd knows about its disk it learned by holding a stopwatch, and a stopwatch held by a descheduled process is measuring the scheduler.
Symptoms and which stage owns them
| What you see | Stage | What to run |
|---|---|---|
| Controllers acting on objects that do not exist, hours after a restore | 6 | compare cluster_id with what your clients saw yesterday |
| Watches resume with no error and the wrong content | 6 | the restore did not use --mark-compacted |
required revision is a future revision | 6 | same cause, caught early; act now |
required revision has been compacted | 5 | expected — something compacted; find out what |
| Database keeps growing and the nightly defrag “fails” | 5 | etcdctl --command-timeout=30s defrag; then check IN USE |
| Compaction ran and the file is the same size | 5 | that is correct; defrag returns it to the filesystem |
Writes refused with database space exceeded after a clean-up | 5 | etcdctl alarm disarm — nothing else clears it |
| Deleting keys made the database larger | 5 | deletes are writes; compact instead |
DB SIZE and IN USE both high and nearly equal | 5 | nothing is compacting; check --auto-compaction-retention |
| etcd says the disk is slow | 2 | fio --fdatasync=1 on the data directory, and cpu.stat |
raft_term climbing; elections in the log | 2 | same two checks, in that order |
| Reads are fast, stale and never wrong-looking | 4 | something is doing serializable reads, or talking to a learner |
| A monitor is green through an outage | 4 | it is watching /livez; move it to /readyz |
context deadline exceeded on member list | 1 | quorum is gone; take a snapshot before anything else |
| Cluster died after a member was replaced | 1 | count the membership list, not the running processes |
| A write returned an error and happened anyway | 3 | a quota alarm; the apply layer does not block |
Before you call it done
Every value below is chosen by whoever installed your cluster: k3s runs etcd with a heartbeat interval and an election timeout five times etcd’s own defaults, kubeadm pins a snapshot count and sets no quota, and a managed control plane will not tell you anything.
- Know what you are actually running. One log line contains every default and every override.
kubectl -n kube-system logs etcd-$(hostname) | grep -m1 'starting an etcd server'— or, on k3s,journalctl -u k3s | grep -m1 'starting an etcd server'. - Find out who is compacting. If
--auto-compaction-retentionis0sand there is no kube-apiserver in front of this etcd, nothing is. - Read both size numbers, across the cluster.
etcdctl endpoint status --cluster -w table, and compareDB SIZEagainstIN USEusing the three branches above. Alsols -lthe backend file, because it is not the number etcd reports. - Measure the disk yourself, once, so you never argue about it again.
fio --rw=write --ioengine=sync --fdatasync=1 --directory=/var/lib/etcd --size=22m --bs=2300 --name=etcdtest— read thefsync/fdatasyncpercentiles, not the bandwidth. If that is healthy and etcd says otherwise, go and look atcpu.stat. - Point your monitoring at
/readyz, not/livez, and alert on the body rather than the status code. - Take a snapshot and restore it somewhere else — with
etcdutl, and with--bump-revisionand--mark-compacted. Then check the revision on the restored cluster is far above the original. If your restore path is k3s’s--cluster-reset-restore-path, it does not pass those flags, and you should know that before the day you need it. - Write down the current
cluster_idsomewhere your future self will find it during an incident. It is the first half of the hinge, and it is useless if you have nothing to compare against.
Related reading
- Kubernetes, Honestly — where etcd sits in a control plane, and what the layer above it assumes
- Running PostgreSQL Properly — the same shape on a different system, and the definitive treatment of the one
fsync - A Second Machine — quorum, split brain, and whether you need redundancy at all
- The Life of a Write — what an
fsyncreaches, and what it does not - Automated Backups — the discipline this page’s stage 6 assumes you already have
- Restoring a Linux Server — the general case of stage 6, for a whole machine rather than one database: what the archive kept, what the restore threw away, and the seven stages in between
- Monitoring Without a Full Stack — where
/readyzshould be wired