Kafka Partition Replication: High Watermark, LEO & In-Sync Replicas (ISR)

Suppose a hardware failure strikes Broker 2 inside your 5-node Kafka cluster at 3:00 AM. Power supplies cut out instantly.

Broker 2 happens to be the Partition Leader for orders-0.

When Broker 2 dies, incoming producer writes and consumer reads destined for orders-0 freeze.

How does Kafka detect Broker 2’s death, elect a replacement partition leader from remaining nodes within milliseconds, and guarantee that no committed messages disappear?

Kafka delivers high availability through Fetch-Based Replication, In-Sync Replica (ISR) Tracking, and High Watermark Offsets.

Leader vs Follower Replicas

When you create a topic with a replication-factor = 3, Kafka creates 3 copies of every partition log across separate broker nodes:

1 Leader Replica: Handles ALL producer write requests and ALL consumer read requests (by default).
2 Follower Replicas: Do not handle client requests. They act as internal consumers, constantly pulling log records from the leader to stay updated.

PRODUCER / CONSUMER CLIENTS
     |
     v (All Client Reads & Writes)
+-------------------------------------------------------+
| BROKER 1 (Partition 0 LEADER)                         |
| Log End Offset (LEO): 100                             |
+-------------------------------------------------------+
   |                                   |
   | (Fetch Request)                   | (Fetch Request)
   v                                   v
+-----------------------+   +-----------------------+
| BROKER 2 (Follower)   |   | BROKER 3 (Follower)   |
| LEO: 100 (In-Sync!)   |   | LEO: 98 (Lagging!)    |
+-----------------------+   +-----------------------+

Log End Offset (LEO) vs High Watermark (HW)

Replication tracking relies on two offset markers maintained for every partition log:

1. Log End Offset (LEO)

The offset of the next record to be written to a local partition log file. Each replica (leader and followers) maintains its own local LEO.

2. High Watermark (HW)

The highest offset that has been successfully replicated to all active In-Sync Replicas (ISR).

LEADER LOG STATE
Offset:    0      1      2      3      4      5
        +------+------+------+------+------+------+
        | Msg  | Msg  | Msg  | Msg  | Msg  | Msg  |
        +------+------+------+------+------+------+
                                ^                 ^
                                |                 |
                       High Watermark (HW): 3  Log End Offset (LEO): 5

The High Watermark Rule: Consumers are ONLY allowed to read messages up to the High Watermark offset.

Even if the leader has written records up to LEO 5, consumers cannot see records at offset 4 or 5 until follower replicas catch up and advance the High Watermark to 5. This prevents consumers from reading data that could vanish if the leader crashes before replication completes.

The In-Sync Replica (ISR) Pool

Not all follower replicas are equal. A follower that falls behind due to network congestion or GC pauses is a liability during leader failover.

Kafka maintains a dynamic subset of replicas called the In-Sync Replica (ISR) pool.

A follower replica remains in the ISR pool if and only if:

It maintains an active session connection with the cluster controller broker (replica.lag.time.max.ms = 30000).
It continually fetches records from the leader, staying within 30 seconds of the leader’s LEO.

If Follower 3 stops sending fetch requests for 30 seconds, the leader drops Follower 3 from the ISR pool:

$\text{Replication Factor: 3} \longrightarrow \text{Active ISR Pool: [Broker 1, Broker 2]}$

Leader Failover Step-by-Step

Suppose Broker 1 (the leader) experiences a physical motherboard failure:

STEP 1: DETECT FAILURE
Broker 1 dies. The Cluster Controller detects heartbeat loss on Broker 1.

STEP 2: ELECT NEW LEADER FROM ISR
The Controller inspects the active ISR pool: [Broker 1 (Dead), Broker 2 (Alive)].
Broker 2 is selected as the new Partition Leader for orders-0.

STEP 3: BROADCAST METADATA UPDATE
The Controller broadcasts a LeaderAndIsr request to all brokers and client SDKs.
Producers and consumers redirect TCP connections to Broker 2.

STEP 4: LOG TRUNCATION & RECOVERY
Broker 2 becomes Leader at HW 100.
When Broker 1 eventually recovers, it truncates its local log back to HW 100 
to discard any un-replicated records, then resumes fetching from Broker 2.

Because leader election selects exclusively from live members of the ISR pool, the new leader is guaranteed to contain all messages up to the High Watermark.

Leader Epochs: Preventing Log Divergence Bugs

In older Kafka versions, log truncation relied solely on High Watermark offsets. In edge scenarios where a leader crashed and restarted quickly, replicas experienced silent data loss or log divergence bugs.

Modern Kafka uses Leader Epochs.

A Leader Epoch is a 32-bit integer incremented every time a new leader takes over a partition:

LEADER EPOCH SEQUENCE:
Epoch 0: Broker 1 leads offsets 0 to 450
Epoch 1: Broker 2 leads offsets 451 to 890
Epoch 2: Broker 3 leads offsets 891 onward

Every record batch stores its Leader Epoch in its header. During failover, followers query the new leader’s OffsetForLeaderEpoch API instead of relying on local High Watermarks, guaranteeing exact log alignment across replicas.

Quick Summary

The partition Leader handles all client reads and writes; Followers fetch logs asynchronously.
Log End Offset (LEO) tracks local write tail; High Watermark (HW) tracks the highest offset replicated to all ISR members.
Consumers can only read messages up to the High Watermark offset.
Followers that lag behind by more than replica.lag.time.max.ms are evicted from the In-Sync Replica (ISR) pool.
Leader Epochs prevent log truncation divergence bugs during node crashes.

References & Further Reading

Apache Kafka Wiki. KIP-98: Transactional Messaging Specification. Kafka Improvement Proposals.
Junqueira, F., & Reed, B. (2013). ZooKeeper: Distributed Process Coordination. O’Reilly Media.
Gray, J. (1978). Notes on Data Base Operating Systems (2PC Protocol). Springer.