Almost nobody chooses etcd. It arrives underneath Kubernetes, it works for a year or two without being thought about, and then one day it is all that stands between you and a cluster you cannot restore. This page is for that day, and for the fortnight before it.
Kubernetes, Honestly deliberately left etcd out, and it ended its second stage with a sentence that is exactly right and stops one clause too early: the API server is the only component that talks to etcd, which is why restoring etcd is the only restore that exists. True. What it does not say — because it is not a page about etcd — is that an etcd restore that reports success can quietly break every controller in your cluster, that the mechanism is a number nobody looks at, and that the restore command in the Kubernetes documentation does not set the two flags that prevent it.
So this is not a tour of etcd. It follows one write through six stages, and it ends with the restore that gives you your data back and your revision numbers back — but not the same ones.
What this page does not cover, and where it went instead.
- Where etcd sits in Kubernetes — that the write happens inside the API server’s create path, that no other component touches etcd, and that controllers coordinate by watching rather than by talking. Kubernetes, Honestly.
- Why quorum needs an odd number, what split brain is, and why two machines are worse than one. A Second Machine makes that argument properly. This page owns only the arithmetic that surprises people, which is what n counts.
- What an
fsyncis, and why a client waits on exactly one of them. Running PostgreSQL Properly and The Life of a Write between them own the whole of it. This page owns one narrow and unpleasant fact: the metric etcd gives you for thatfsyncis not a disk metric.
And one honest limitation, stated here rather than at the end: if your control plane is managed — EKS, GKE, AKS — most of this page is not relevent to you. There is no etcd to ask. That is not a failure of the page; it is worth knowing which half of the operational surface you gave away.
The six stages of a write. Every section below is named after one.
- Which member answered — and whether it was allowed to.
- The leader, the log, and the one
fsync— the only thing the client actually waits for. - Apply — the backend, and the number that gets issued.
- Read and watch — and what it costs to be certain.
- Compact, defrag and the quota — the deliberate destruction of history.
- The restore, and the revision it reuses.
The diagnostic hinge: a revision is not a name. It is a position in one particular cluster’s history — and “one particular cluster” is a thing that can be replaced underneath you without anybody being told.
Every answer etcd gives you is signed. Read the signature before you believe the answer:
etcdctl endpoint status -w json | jq '.[0].Status.header'
{
"cluster_id": 3867748490576809579,
"member_id": 6526328719644797682,
"revision": 2002,
"raft_term": 2
}Three of those are the ones to read, and the first is the question.
cluster_idis not the one your clients were talking to yesterday → you are downstream of a restore, whether or not anybody told you. Every client holding a revision is holding a number that now means something else, and it will not be told. Stage 6, and nothing else on this page will help until that is settled.- It matches, and reads at old revisions are refused (
mvcc: required revision has been compacted) → your history has been truncated on purpose, by something. Stage 5. The keys are fine; the history is not. - It matches, and
raft_termclimbs between checks → elections. Stage 2 — and read it with the worked diagnosis in hand, because etcd will tell you it is the disk and it usually is not. - Everything agrees and old revisions still read → your history is intact, and your problem is measured in milliseconds (stage 2) or in bytes (stage 5) rather than in revisions.
The signature is not a health check. It is an identity check. etcd will answer you confidently after it has become a different cluster, and that is the failure this page exists for. It is also worth knowing that endpoint status is a local call: it answers with no leader and no quorum, so the hinge still evaluates during the outage, when almost nothing else does.
Stage 1: Which member answered
You gave a client a list of endpoints. It picked one. Everything that follows depends on which one, and on what that member is allowed to do — and the client will not tell you any of it.
Send a write to a follower and it simply works. The etcd server forwards the proposal to the leader over the peer channel; the client library is not involved, and nothing in the response records that it happened:
etcdctl --endpoints=<a follower> put /k via-follower
OKThere is one member that behaves differently, and it is the one people put in endpoint lists by mistake. A learner is not a read replica. It refuses writes, and it refuses linearizable reads, with an error that at least names itself:
Error: etcdserver: rpc not supported for learnerAnd it serves serializable reads perfectly happily, from whatever it has caught up to. So a learner in a client’s endpoint list does not fail loudly; it makes some fraction of that client’s reads quietly stale, with no error and no metric. Learners arrived in etcd 3.4.0 and have behaved this way throughout.
The membership arithmetic, and the member that was never started
A Second Machine covers why a majority has to be a majority. What that argument leaves out is what is being counted, and it is not what you would guess. Quorum counts voting members in the membership list — not running processes. A member added with member add and never started counts immediately. A learner does not count at all.
That produces the failure that actually kills clusters, and it is not the one everybody warns about. The old advice — never add the replacement before removing the dead member — describes a guard rail that etcd has had since 3.3, in 2018: --strict-reconfig-check defaults to true and refuses the request outright:
# two of three members up
etcdctl member add spare --peer-urls=http://10.0.0.9:2380
Error: etcdserver: unhealthy clusterWhat is not guarded is adding a member to a healthy cluster and then failing to start it — a wrong peer URL, an image that will not pull, a firewall rule. The check passes, because at the moment of the check the cluster was fine. Then the arithmetic changes underneath you:
four healthy members, two more added and never started
-> 6 voting members, quorum 4, 4 alive
one member then dies
-> etcdctl member list Error: context deadline exceeded
-> etcdctl member remove Error: context deadline exceededRead the last two lines carefully, because they are the reason “just remove the dead members” is not a plan. Membership changes are Raft proposals. They need the quorum you have already lost. member remove is executable only while you do not need it.
Two smaller facts belong here. --max-learners defaults to 1, so you cannot stage two replacements at once (etcdserver: too many learner members in cluster). And when you do lose quorum, before you reach for --force-new-cluster on a live data directory — the one move with no undo — take the free snapshot. It works. That is stage 6’s material and it is the first thing to do, not the last.
Stage 2: The leader, the log, and the one fsync
The leader appends your write to its Raft log, replicates it, waits for a majority of members to have persisted it, and then applies it locally before answering. Running PostgreSQL Properly covers what a write-ahead log is and why exactly one fsync is the thing a client waits on; the shape is the same here and there is no point restating it.
Two things this stage does own. The first is a correction to the usual summary: the leader does not answer at commit, it answers after applying. Followers apply asynchronously, which is why a linearizable read served by a follower costs a round trip to the leader — and why, when quorum is gone, the survivor logs its failure by that name: waiting for ReadIndex response took too long, retrying.
The second is the reason this stage is in the article at all.
etcd_disk_wal_fsync_duration_seconds is not a disk metric
etcd’s own guidance is a p99 wal_fsync under 10 ms, and every etcd page on the internet repeats it. The number is fine. The instrument is not what it sounds like: it is a stopwatch started and stopped by Go code inside the etcd process, and a process that is not being scheduled cannot stop a stopwatch.
Here is one etcd leader with a CPU quota applied and nothing else changed — a Kubernetes limits.cpu: 50m — measured against fio writing to the same filesystem in the same minute:
mean wal_fsync | mean backend_commit | fio --fdatasync=1, same directory | |
|---|---|---|---|
| before | 0.533 ms | — | avg 206 µs, p1 101 µs |
| during | 7.84 ms | 9.28 ms | avg 250 µs, p1 102 µs |
A fifteenfold degradation reported by etcd, and no change at all in the disk. Both of the metrics everybody recommends agreed with each other and both were describing the scheduler. This is the worked diagnosis at the end of the page, and it is the reason this stage exists rather than delegating entirely.
Elections belong here too, and they are worth being able to recognise in a journal at three in the morning rather than as a state machine:
is starting a new election at term 2
became pre-candidate at term 2
became candidate at term 3
became leader at term 3Four lines, in that order, is an election. If raft_term in the hinge is climbing, this is what is in the log.
Stage 3: Apply, and the number that gets issued
Applying means writing into the backend — a bbolt file, one per member — and issuing a revision. This is the centre of the article, and the single most useful thing to hold about etcd is that the revision is the object and the key is an index into it.
There are four numbers, not one, and a single response contains all of them:
{"header":{"cluster_id":3867748490576809579,"member_id":6526328719644797682,
"revision":4,"raft_term":2},
"kvs":[{"key":"L2E=","create_revision":2,"mod_revision":4,"version":2,
"value":"dGhyZWU="}],"count":1}| Number | Scope | What it means |
|---|---|---|
header.revision | The whole cluster | The store’s current position. Every write of any kind advances it, including a delete. |
create_revision | One key | The revision at which this key was created. |
mod_revision | One key | The revision at which it was last modified. This is what a compare-and-swap compares. |
version | One key | A per-key counter, starting at 1. Nothing to do with the other three. |
That is a table because those four are field definitions out of etcd’s protobuf — identical on every cluster in the world — and because you have to hold all four at once to read a single response. Almost nothing else on this page is a table, because almost nothing else about etcd is the same on two machines.
The revision is not the Raft index. They are printed side by side and they diverge immediately: a cluster showing RAFT INDEX 25 had revision 7. The Raft index counts everything the log carries, including membership changes and internal traffic; the revision counts only changes to the key space. Anything comparing the two is confused.
One more correction to the usual summary of stage 2, because it lands here: an error return is not proof the write did not happen. Under a quota alarm, a Put can return ErrGRPCNoSpace and still be applied — the quota is checked at the API layer and again at the apply layer, and the apply layer raises the alarm without blocking the transaction. If you are writing retry logic against etcd, that is the case it has to survive.
Stage 4: Read and watch
Reads come in two kinds and the default is the expensive one. A linearizable read — the default — guarantees you see everything committed before the read began, and pays for it with a ReadIndex round trip to the leader. A serializable read is served straight from whichever member you asked, with no round trip and no guarantee about freshness:
etcdctl get /k # linearizable: costs a trip to the leader
etcdctl get /k --consistency=s # serializable: local, fast, possibly staleThe failure mode is the one from stage 1 in a different costume. On a cluster that has lost quorum, writes and linearizable reads both return context deadline exceeded — and a serializable read returns the last value it saw, with no warning that the cluster is dead. That is genuinely useful during an incident and genuinely dangerous in an application.
A watch is a stream, and it can be resumed from a revision. On a live cluster, resuming from a revision that has been destroyed fails exactly as it should:
$ etcdctl watch /a --rev=2
watch was canceled (etcdserver: mvcc: required revision has been compacted)
Error: watch is canceled by the serverLoud, specific, and the correct behaviour. Remember what it looks like, because stage 6 is about the case where the same request produces no error at all.
Two health endpoints, and they disagree with each other on purpose. On a member with no leader, serving neither reads nor writes:
curl -s -o /dev/null -w '%{http_code}\n' localhost:2379/livez # 200
curl -s -o /dev/null -w '%{http_code}\n' localhost:2379/readyz # 503
curl -s localhost:2379/health
{"health":"false","reason":"RAFT NO LEADER"}
# /readyz names the failing check:
[-]linearizable_read failed: context canceled
[+]non_learner ok [+]data_corruption ok [+]serializable_read ok/livez returns 200 through a total outage, and that is correct — it is a liveness probe, and restarting a member that has merely lost contact with its peers would make things worse. kubeadm wires it up exactly that way. But any dashboard or uptime check pointed at /livez stays green while the cluster serves nothing. Point monitoring at /readyz, and read the body rather than the code.
Stage 5: Compact, defrag and the quota
Every write leaves the previous version of the key behind. Nothing removes those versions until something compacts, and compaction is not housekeeping — it is the deliberate destruction of the only reference your clients have. Stage 4’s error is what a client sees afterwards.
Compaction and defragmentation are two separate operations and the difference is the most misunderstood thing in the subject. Forty thousand revisions of one 1 KiB key, on a single member:
| DB SIZE | IN USE | NOT IN USE | the file on disk | |
|---|---|---|---|---|
| after the churn | 74 MB | 74 MB | 0% | 84,004,864 |
after compact | 74 MB | 234 kB | 100% | 84,004,864 |
after defrag | 25 kB | 16 kB | 34% | 32,768 |
Compaction frees space inside the file. Defragmentation gives it back to the filesystem. Note also that the file on disk and the number etcd reports are not the same — 84,004,864 bytes against a reported 73,691,136 — and it is the reported number the quota is enforced against.
Two things compaction does not do. It does not remove keys: a key last written long before the compaction point survives with its original value, readable, while every revision below the compaction point is gone. And compact(N) leaves revision N itself readable; it is N−1 that disappears.
Two numbers, and only one of them is your fault.
etcdctl endpoint status --cluster -w table
… │ DB SIZE │ IN USE │ PERCENTAGE NOT IN USE │ QUOTA │ …IN USEhigh → you have too much live data, or too much retained history. Compact. Defragmenting now buys nothing.IN USElow,DB SIZEhigh (NOT IN USEnear 100%) → you have already compacted and the file has not caught up. Defrag, one member at a time, and budget roughly thirteen seconds per 500 MB of member.- Both high and nearly equal → nothing is compacting at all. This is not a maintenance problem, it is an unconfigured one, and it is the branch missing from every other page on the subject.
That third branch is common because of a default nobody expects. --auto-compaction-retention defaults to 0s, and 0 means never. Under Kubernetes you never find out, because the API server runs its own compactor every five minutes against a top-level key of its own. Run etcd for anything else, follow etcd’s own documentation, and you have a database that grows until it stops.
Defragmentation is an outage, and it lies about failing
A member being defragmented serves nothing for the duration. On a 466 MB member with a reader issuing a linearizable get every 50 ms:
Finished defragmenting etcd member[127.0.0.1:2379]. took 13.347056448s
read latencies, ms: 5039 5038 3186 63 29 ... median 15The two 5,039 ms readings are not slow reads. They are etcdctl‘s own five-second command timeout: those reads failed. Which is why etcdctl defrag --cluster — one keystroke away — is a self-inflicted control-plane outage rather than a maintenance command. It iterates every member in sequence, each unavailable in turn.
And then the part that has quietly convinced a great many people their maintenance is broken:
$ etcdctl defrag
Failed to defragment etcd member[127.0.0.1:2379]. took 5.004255622s. (context deadline exceeded)
# and 6.88 s later, in the server's own log:
"msg":"finished defragmenting directory","current-db-size-bytes-diff":-227414016,
"current-db-size":"426 MB","took":"6.882079661s"etcdctl‘s default --command-timeout is five seconds, so on any database big enough to need defragmenting the client reports failure while the server succeeds and reclaims the space. A nightly defrag job that has “failed” every night for a year has been working every night for a year. Raise the timeout, or use etcd-defrag, the tool the Kubernetes documentation points at, which defaults it to thirty seconds and is also where the “defragment the leader last” advice actually comes from — it is that tool’s guidance, not etcd’s.
The quota alarm, and why the permitted operation is the wrong one
The backend quota defaults to 2 GiB. Cross it and etcd raises a NOSPACE alarm against the member that went over, enforced cluster-wide, and starts refusing writes with a gRPC ResourceExhausted. The alarm names a member ID, which is why endpoint status without --cluster will not tell you which member is the problem.
etcd’s documentation says the cluster then “only accepts key reads and deletes”, which reads like permission. It is not advice:
Under a NOSPACE alarm | What actually happens | |
|---|---|---|
| Read a key | ✓ | Works normally. |
| Delete keys | ✓ | Permitted, and it makes things worse. Deletes are writes; they add tombstone revisions. 6,000 deletes took the database from 16,842,752 to 17,195,008 bytes. |
| Put a key | ✗ | etcdserver: mvcc: database space exceeded |
| Grant a lease | ✗ | Also refused. Anything holding a lease is now on a clock it cannot renew. |
| Compact | ✓ | Works, and is the actual fix. Space comes back at compaction, never at deletion. |
| Defragment | ✓ | Works, and returns the file to the filesystem. |
| Clear the alarm | ✓ | Only etcdctl alarm disarm does this. Compacting and defragmenting back down to 24 kB does not clear it; puts keep failing until you disarm. |
That is a table because it is a behaviour matrix defined by etcd’s applier, identical on every machine, and because during an incident you need to scan it rather than read a paragraph.
The sequence, then, is compact → defrag → alarm disarm, in that order, and nothing before the last step will let a write through. One more thing worth knowing before you set a larger quota: there is a documented ceiling of about 8.6 GB, and exceeding it is a warning rather than a refusal — etcd logs quota exceeds the maximum value and starts anyway, which means a typo in that flag is a thing you find out about much later.
Stage 6: The restore, and the revision it reuses
Two things before the mechanism, because both are useful in the first five minutes of an incident.
etcdctl snapshot save does not need quorum. On a cluster that could serve neither a read nor a write — where member list and member remove both timed out — snapshot save returned Snapshot saved at … and etcdutl snapshot status read it back cleanly. Your last-resort backup is available on a dead cluster, and taking it is free. Do that before --force-new-cluster, which has no undo.
And the command changed under everyone. etcdctl snapshot restore, etcdctl snapshot status and etcdctl defrag --data-dir were all removed in etcd 3.6.0 on 15 May 2025. They are etcdutl now. Kubernetes 1.34 was the first kubeadm to ship etcd 3.6, so this is the upgrade on which every saved runbook became a command that does not exist.
What a restore actually does
It builds a new cluster from the snapshot. Watch the cluster ID and the revision:
state at snapshot time:<br> clusterID=5920653729087557766 rev=2002 raftIndex=2009
(500 more writes happen; revision reaches 2503)
after etcdutl snapshot restore:<br> clusterID=13133426046165070218 rev=2002 raftIndex=6
keys written after the snapshot: Count 0New cluster ID. Raft index reset. Post-snapshot writes gone, which is expected and is why you took the snapshot. And the revision preserved at 2002 — which sounds like the considerate option and is the whole problem.
A client that was watching, or holding a resource version, resumed from revision 2600. Immediately after the restore it gets an error nobody has an alert for:
Error: etcdserver: mvcc: required revision is a future revisionThen the cluster does 700 ordinary writes, passes 2600 on its way back up, and the same request returns this:
$ etcdctl get --rev=2600 --prefix /
/new-00000000
/new-00000001
/new-00000002
$ etcdctl watch --rev=2600 --prefix /
PUT
/new-00000597
…
PUT
/new-00000599
(exit 0)No error. A clean stream of a different history, and exit zero. Compare that with the compacted watch in stage 4, which failed loudly and correctly. The difference is that the revision number 2600 has been reissued, attached to content that has nothing to do with what the client saw at 2600 yesterday, and there is no field in the response that says so.
What that looks like operationally is a controller reconciling objects that do not exist, a Job being recreated, an operator’s cache disagreeing with kubectl get — hours after the restore, correlating with nothing, and not looking remotely like a restore problem.
The two flags
The same snapshot, restored with two more flags:
etcdutl snapshot restore snap.db \
--data-dir /var/lib/etcd \
--bump-revision 1000000 --mark-compacted
revision after restore: 1002002
data still there: /canary before-snapshot
read at revision 2600: Error: etcdserver: mvcc: required revision has been compacted
watch at revision 2600: watch was canceled (…required revision has been compacted)All the data, and every stale resume refused loudly instead of answered wrongly. --bump-revision takes a number of revisions rather than a revision, and etcd’s own recovery page suggests a billion to cover a week-old snapshot; --mark-compacted is what turns “future revision” into “has been compacted”, which is the error clients already know how to handle.
These landed in etcd 3.5.10, on 27 October 2023. etcd’s own recovery documentation calls bumping the revision “highly recommended … when using Kubernetes in general”. The Kubernetes documentation’s restore recipe does not mention either flag, and the word “revision” appears on that page twice, both times inside a table header. k3s does not pass them either: its --cluster-reset-restore-path builds a restore config containing the snapshot path, the name, the output directories, the peer URLs and the initial cluster, and nothing else. A k3s restore reuses revisions.
Which is the correction this page owes Kubernetes, Honestly. Restoring etcd is the only restore that exists. It is also the only restore that can succeed completely and take your controllers with it, and the difference is two flags that the instructions you will follow do not contain.
A worked diagnosis: the heartbeat that blamed the disk
Intermittent API server latency, and occasionally etcdserver: request timed out in its log. In etcd’s own log, steadily:
{"level":"warn","msg":"leader failed to send out heartbeat on time; took too long,
leader is overloaded likely from slow disk","heartbeat-interval":"100ms",
"expected-duration":"200ms","exceeded-duration":"194.834µs"}<br>{"level":"warn","msg":"apply request took too long","took":"100.94362ms",
"expected-duration":"100ms","request":"put:<key:\"/x9-00000266\" value_size:1024 >"}Every obvious observable says nothing is wrong. No elections — etcd_server_leader_changes_seen_total stayed at 1 across the whole incident, so the symptom everybody looks for is absent. No alarms, no NOSPACE, the database is 1.9 MB, all three members healthy, endpoint status clean. And the metric everyone reaches for agrees with the log: mean wal_fsync went from 0.533 ms to 7.84 ms, with a tail into the quarter-second buckets.
Except that fio, writing 2,300-byte blocks with --fdatasync=1 into the same directory during the same window, measured no change at all: p1 of 101 µs before, 102 µs during. The disk was fine the whole time.
The answer was one file:
$ cat /sys/fs/cgroup/cpu/etcdslow/cpu.stat
nr_periods 1787<br>nr_throttled 1785
throttled_time 2043991186061,785 of 1,787 scheduling periods throttled. The member was running under a CPU quota of 5 ms per 100 ms period — a Kubernetes limits.cpu: 50m. And the smoking gun is in the warning itself: etcd is complaining about a heartbeat that was late by 194 microseconds, and elsewhere in the same run by 16.656 microseconds, and naming the disk while it does it.
Where it comes from. kubeadm’s etcd static pod ships CPU and memory requests and no limits at all, which is correct. So this failure does not arrive with etcd. It arrives on the day somebody adds a LimitRange, or a vertical autoscaler, or a well-meant limits: block, to the namespace the control plane lives in.
And note where the hinge points. cluster_id matches, raft_term is not climbing, old revisions still read — so the hinge says “your history is intact, your problem is milliseconds” and sends you to stage 2. That is the right stage, and the cause is one layer below anything stage 2 can see, in the scheduler. The hinge is a localiser, not an oracle; it puts you in the right room and does not open the cupboard.
The moral, and it generalises well past etcd: a latency you measure from inside a process includes the time the process spent not running. Everything etcd knows about its disk it learned by holding a stopwatch, and a stopwatch held by a descheduled process is measuring the scheduler.
Symptoms and which stage owns them
| What you see | Stage | What to run |
|---|---|---|
| Controllers acting on objects that do not exist, hours after a restore | 6 | compare cluster_id with what your clients saw yesterday |
| Watches resume with no error and the wrong content | 6 | the restore did not use --mark-compacted |
required revision is a future revision | 6 | same cause, caught early; act now |
required revision has been compacted | 5 | expected — something compacted; find out what |
| Database keeps growing and the nightly defrag “fails” | 5 | etcdctl --command-timeout=30s defrag; then check IN USE |
| Compaction ran and the file is the same size | 5 | that is correct; defrag returns it to the filesystem |
Writes refused with database space exceeded after a clean-up | 5 | etcdctl alarm disarm — nothing else clears it |
| Deleting keys made the database larger | 5 | deletes are writes; compact instead |
DB SIZE and IN USE both high and nearly equal | 5 | nothing is compacting; check --auto-compaction-retention |
| etcd says the disk is slow | 2 | fio --fdatasync=1 on the data directory, and cpu.stat |
raft_term climbing; elections in the log | 2 | same two checks, in that order |
| Reads are fast, stale and never wrong-looking | 4 | something is doing serializable reads, or talking to a learner |
| A monitor is green through an outage | 4 | it is watching /livez; move it to /readyz |
context deadline exceeded on member list | 1 | quorum is gone; take a snapshot before anything else |
| Cluster died after a member was replaced | 1 | count the membership list, not the running processes |
| A write returned an error and happened anyway | 3 | a quota alarm; the apply layer does not block |
Advice that has expired
| Commonly said | What is actually true |
|---|---|
| “A restore that reports success has restored your cluster” | It has restored your data, on a new cluster ID, and will shortly reissue revision numbers your clients already hold. --bump-revision and --mark-compacted since etcd 3.5.10, October 2023. |
“etcdctl snapshot restore“ | Removed in etcd 3.6.0, 15 May 2025. So were snapshot status and defrag --data-dir. It is etcdutl. |
“export ETCDCTL_API=3“ | Unnecessary since 3.4.0 in 2019; since 3.6.0 it logs unrecognized environment variable. On 3.5.x, setting it to 2 still breaks the client. |
| “Adding the replacement before removing the dead member kills the cluster” | --strict-reconfig-check has refused that since etcd 3.3 in 2018. What kills clusters is adding a member to a healthy cluster and never starting it. |
| “You cannot back up a cluster that has lost quorum” | snapshot save works with no quorum, and it is the first thing to do. |
“Under a NOSPACE alarm, delete keys to free space” | Deletes are writes. 6,000 of them made the database larger. |
| “Compaction will reclaim the disk space” | Inside the file only. IN USE fell from 74 MB to 234 kB while the file stayed at 84 MB. |
| “The heartbeat warning means the disk is slow” | etcd says so in the message text. The reproduction above says it about a 16 µs delay on an idle disk. |
| “A learner is a read replica” | It refuses linearizable reads and serves serializable ones, silently. |
“/livez tells you etcd is working” | 200 with no leader, serving nothing. |
“--snapshot-count defaults to 100,000″ | 10,000 since 3.6.0. etcd’s own v3.6 maintenance page still says 100,000. |
| “etcd 3.5 is the current release” | 3.7.0 shipped 8 July 2026; 3.6.0 was May 2025. Supported: 3.7 and 3.6. |
How to tell whether a page about etcd is worth reading
All of the above comes from one model, and naming it is more useful than the list: the corpus treats etcd as PostgreSQL with three copies — a store of keys that happens to be replicated — when it is a replicated log that happens to expose a key-value view of its own tail. So it treats the key as the object and the revision as a timestamp printed on it, and everything else follows. If the key is the object, a backup is a set of keys, so a restore that returns the right keys has succeeded, so nobody mentions the revision. If the key is the object, deleting keys frees space. If compaction is VACUUM, it should shrink the file. If it is a database, a slow database is a disk problem.
Invert it and the whole page falls out: the revision is the object and the key is an index into it.
The one-sentence test is unusually reliable here. Find the page’s defragmentation section, and see whether it mentions that etcdctl defrag prints an error on any database large enough to need defragmenting. Everybody who has run one has seen context deadline exceeded; nobody who has only read about it mentions it. Failing that: does the page give a single number for how long anything takes? “Defragmentation is an expensive operation” is what you write after reading. “Thirteen seconds per 500 MB, during which the member serves nothing” is what you write after watching one.
The silences are sharper than the claims:
| The page never mentions… | …which tells you |
|---|---|
--bump-revision or --mark-compacted, on a page about restoring | The author has never restored a cluster that had live watchers, and so has never restored a real Kubernetes cluster and watched what the controllers did afterwards. This one disqualifies the Kubernetes documentation itself. |
| That auto-compaction is off by default | They have only ever seen etcd under Kubernetes, where the API server hides it. |
--command-timeout, anywhere near a maintenance command | Every command they ran finished in under five seconds. Their cluster was a laptop. |
| A member ID in any example output | They never had two members. Alarms, the ERRORS column and every peer log line are keyed by member ID. |
IN USE as distinct from DB SIZE | They have never watched compaction fail to shrink a file, and will recommend compaction alone as the cure for a full disk. |
| CPU, on a page that discusses leader churn | They copied etcd’s warning string and did not test it. |
/readyz as distinct from /livez | They have never lost quorum. Anyone who has noticed liveness stayed green. |
One page has visibly fixed itself, and which half it fixed is the interesting part. Kubernetes’ own “Operating etcd clusters” page has absorbed the tooling split — it now has a section explaining etcdutl, and presents restore as tabs with the etcdctl one labelled deprecated. Somebody did that work. What it has not absorbed is anything about time: the deprecation note is still in the future tense sixteen months after the removal happened, the deprecated tab still opens with export ETCDCTL_API=3, and the prerequisites still recommend etcd versions that reached end of life. And the restore recipe it gives has no --bump-revision and no --mark-compacted.
That is the shape of the whole corpus in one page. It fixed which binary you type, and left on the floor the flag that decides whether your controllers work afterwards.
Before you call it done
Seven checks. None of them can be answered from a table, because every value is chosen by whoever installed your cluster — k3s runs etcd with a heartbeat interval and an election timeout five times etcd’s own defaults, kubeadm pins a snapshot count and sets no quota, and a managed control plane will not tell you anything.
- Know what you are actually running. One log line contains every default and every override.
kubectl -n kube-system logs etcd-$(hostname) | grep -m1 'starting an etcd server'— or, on k3s,journalctl -u k3s | grep -m1 'starting an etcd server'. - Find out who is compacting. If
--auto-compaction-retentionis0sand there is no kube-apiserver in front of this etcd, nothing is. - Read both size numbers, across the cluster.
etcdctl endpoint status --cluster -w table, and compareDB SIZEagainstIN USEusing the three branches above. Alsols -lthe backend file, because it is not the number etcd reports. - Measure the disk yourself, once, so you never argue about it again.
fio --rw=write --ioengine=sync --fdatasync=1 --directory=/var/lib/etcd --size=22m --bs=2300 --name=etcdtest— read thefsync/fdatasyncpercentiles, not the bandwidth. If that is healthy and etcd says otherwise, go and look atcpu.stat. - Point your monitoring at
/readyz, not/livez, and alert on the body rather than the status code. - Take a snapshot and restore it somewhere else — with
etcdutl, and with--bump-revisionand--mark-compacted. Then check the revision on the restored cluster is far above the original. If your restore path is k3s’s--cluster-reset-restore-path, it does not pass those flags, and you should know that before the day you need it. - Write down the current
cluster_idsomewhere your future self will find it during an incident. It is the first half of the hinge, and it is useless if you have nothing to compare against.
And the honest limitation, now that the spine has been walked: none of this makes etcd reliable. It makes the failures legible. A three-member cluster on one host is still one host; a snapshot you have never restored is still a hypothesis, as Running PostgreSQL Properly puts it; and the restore you rehearse today is the one that will be run at four in the morning by somebody who has not read this page. Leave the two flags in the runbook.
Related reading
- Kubernetes, Honestly — where etcd sits in a control plane, and what the layer above it assumes
- Running PostgreSQL Properly — the same shape on a different system, and the definitive treatment of the one
fsync - A Second Machine — quorum, split brain, and whether you need redundancy at all
- The Life of a Write — what an
fsyncreaches, and what it does not - Automated Backups — the discipline this page’s stage 6 assumes you already have
- Restoring a Linux Server — the general case of stage 6, for a whole machine rather than one database: what the archive kept, what the restore threw away, and the seven stages in between
- Monitoring Without a Full Stack — where
/readyzshould be wired