etcd Internals: Raft Consensus, MVCC Key-Value Storage, and Watch Streams

A network partition cuts through a multi-region cloud deployment, isolating Control Plane Node 1 from Control Plane Nodes 2 and 3 in a 3-node etcd cluster.

A developer attempts to execute a deployment update against the isolated Node 1:

kubectl apply -f deployment.yaml --server=https://node1.example.com:6443

Node 1 rejects the write request with:

Error from server: etcdserver: leader changed / no leader

Simultaneously, an update executed against Node 2 or Node 3 completes successfully in 4 milliseconds.

Why did Node 1 reject writes while Nodes 2 and 3 continued processing updates without data corruption or split-brain conflicts?

The answer lies in Raft Consensus Quorum Math and the internal storage engine powering etcd.

1. The Role of etcd in Kubernetes

etcd is a strongly consistent, distributed key-value store written in Go.

In Kubernetes, etcd serves as the single source of truth for the entire cluster. Every PodSpec, Service network endpoint, secret payload, and node status metric is stored as a key path under /registry/:

/registry/pods/default/nginx-pod-7d4f9b
/registry/deployments/production/payment-api
/registry/services/specs/default/web-service

If etcd loses consistency or suffers data corruption, the Kubernetes control plane collapses.

2. Raft Consensus Protocol & Quorum Arithmetic

To guarantee consistency across network partitions and host crashes, etcd implements the Raft Consensus Algorithm.

[ Leader Node (Node 2) ] --- Replicates Log Entries ---> [ Follower Node 3 ]
         |
         | (Network Partition - Node 1 Isolated!)
         x
[ Follower Node 1 ] (Isolated - Cannot reach Quorum of 2!)

Raft Quorum Mathematics

A Raft cluster requires a Quorum (a strict majority) of nodes to acknowledge a write before committing it to disk storage:

$\text{Quorum Size} = \left\lfloor \frac{N}{2} \right\rfloor + 1$

For a 3-Node Cluster ( $N=3$ ): $\text{Quorum} = \lfloor 3/2 \rfloor + 1 = \mathbf{2 \text{ Nodes}}$ .
For a 5-Node Cluster ( $N=5$ ): $\text{Quorum} = \lfloor 5/2 \rfloor + 1 = \mathbf{3 \text{ Nodes}}$ .

How the Partition Was Handled:

When Node 1 was isolated, it could reach only 1 node (itself). Since $1 < \text{Quorum (2)}$ , Node 1 refused to accept writes, preventing split-brain data corruption.
Nodes 2 and 3 could communicate with each other ( $2 \ge \text{Quorum (2)}$ ). They maintained quorum, elected Node 2 as Leader, and processed cluster updates smoothly.

Rule of Thumb: Always deploy etcd clusters using an odd number of nodes (3, 5, or 7) to maximize fault tolerance without increasing quorum requirements unnecessarily.

3. Storage Engine Architecture: MVCC & bbolt

Unlike traditional key-value stores that overwrite values in-place, etcd uses a Multi-Version Concurrency Control (MVCC) architecture backed by an embedded B+ Tree storage engine (bbolt).

etcd never overwrites historical key entries on disk. Every state modification increments a global 64-bit Revision Counter.

[ In-Memory Index (B-Tree) ]           [ Disk Storage: bbolt (B+ Tree) ]

Key: "/registry/pods/nginx"            Revision (104, 0) -> Data: {Replicas: 1}
      |                                Revision (108, 0) -> Data: {Replicas: 3}
      +---> Points to Revisions        Revision (115, 0) -> Data: {Replicas: 5}

Dual-Index Memory & Disk Mapping

In-Memory Index (B-Tree): Maps logical Kubernetes keys (e.g., /registry/pods/nginx) to a list of historical Revision Numbers.
On-Disk B+ Tree (bbolt): Stores keys formatted as (revision_number, sub_id) mapped directly to raw JSON payload bytes.

Why Revision-Based MVCC Matters:

Non-Blocking Historical Reads: Components can read historic cluster snapshots at revision $R_{104}$ while write operations commit new versions at $R_{115}$ without lock contention.
Deterministic Event History: Every modification in the cluster gets an incremental revision number, allowing subscribers to replay exact change events in historical order.

4. Efficient State Sync: gRPC Watch Streams

How do controllers inside kube-controller-manager or kubelet learn about cluster changes?

If 500 controllers constantly polled the API Server (GET /api/v1/pods every 1 second), database CPU usage would hit 100% just processing repetitive read queries.

etcd solves this using HTTP/2 gRPC Watch Streams:

Client (kube-apiserver)                             etcd Engine
  |                                                     |
  |--- 1. Watch(key="/registry/pods", rev=104) ------->|
  |                                                     | (Establishes persistent gRPC Stream)
  |                                                     |
  |                        [ State Event Occurs! ]      |
  |                        Key modified at Rev 105      |
  |                                                     |
  |<-- 2. Stream Event: PUT (Rev 105, Data...) ---------| (Instant Push Notification!)

How Watch Streams Work:

A component opens a long-lived gRPC stream requesting to watch a key prefix starting from a specific revision (e.g., Watch(prefix="/registry/pods", start_rev=104)).
When any client modifies a matching key, etcd pushes a WatchResponse event payload down the open HTTP/2 stream instantly.
If a network disconnect occurs, the client reconnects specifying its last seen revision (start_rev=108). etcd reads its MVCC storage and streams back all missed events between revision 108 and current revision 115!

5. History Compaction (`etcdctl compact`)

Because etcd preserves historical revisions for MVCC watch streams, disk usage grows continuously over time.

If left unmanaged, the database triggers a safety quota error:

RPC error: code = ResourceExhausted desc = etcdserver: mvcc: database space exceeded

To prevent database space exhaustion, Kubernetes periodically executes History Compaction:

kube-apiserver issues a compaction request (e.g., Compact(revision=50000)).
etcd discards historical revision entries older than revision 50,000, freeing disk pages for reuse.

Summary & Next Steps

etcd provides the distributed memory foundation for Kubernetes:

Raft Consensus enforces strict majority quorums ( $\lfloor N/2 \rfloor + 1$ ) to eliminate split-brain data corruption across network partitions.
MVCC & bbolt store key modifications using incremental 64-bit revision numbers without overwriting historical data.
gRPC Watch Streams push state changes instantly over persistent HTTP/2 connections, eliminating polling overhead.
Compaction recycles old revision records to maintain database storage health.

In the next article, we examine The Kubelet and Container Runtime Interface (CRI): How Nodes Execute Pods.

References & Further Reading

Cloud Native Computing Foundation. Kubelet Architecture & Kube-Proxy Modes (iptables vs IPVS). CNCF Docs.
Netfilter Project. Linux iptables(8) & ipvsadm(8) Manuals. Linux Netfilter Docs.
Isovalent / CNCF. Replacing kube-proxy with eBPF in Cilium. Cilium Docs.