Kafka KRaft Consensus Mode: Replacing ZooKeeper for Million-Partition Scale

Suppose you operate a large-scale Kafka deployment at a major enterprise running 500 broker nodes and 1,000,000 topic partitions.

At 4:00 AM, the broker acting as the Kafka Controller node crashes due to a hardware memory fault.

Under older Kafka architectures (Kafka 2.x and earlier), the cluster relied on an external Apache ZooKeeper ensemble to manage cluster metadata.

When the Controller died, ZooKeeper had to elect a new Controller broker. The new Controller then had to read metadata for all 1,000,000 partitions from ZooKeeper nodes and write metadata updates to all 500 brokers.

The result? The controller failover process took over 15 minutes. For 15 minutes, topic creation, partition rebalancing, and leader failovers were completely frozen across the company.

This metadata bottleneck led to KIP-500: replacing ZooKeeper with KRaft (Kafka Raft Metadata Mode).

Why ZooKeeper Reached its Architectural Limits

For the first decade of Kafka’s existence, ZooKeeper managed topic configurations, partition leader assignments, and broker membership.

While ZooKeeper worked well for small clusters, it created three fundamental scaling limitations:

1. Dual-System State Inconsistency

Kafka brokers maintained cluster state in JVM memory, while ZooKeeper maintained cluster state in its zNode tree. Discrepancies between ZooKeeper state and broker state caused subtle split-brain bugs.

2. High Partition Ceiling ( $O(N)$ Metadata Synchronization)

When a metadata change occurred (such as a broker crash), the Kafka Controller serialized and sent RPC requests to every broker individually. As partition count grew past 200,000, metadata synchronization times exploded.

3. Operational Complexity

System administrators had to deploy, secure, monitor, and tune two distinct distributed systems (Kafka JVMs and ZooKeeper JVMs) with separate security ACLs and configuration parameters.

The KRaft Solution: Self-Managed Consensus

Introduced in Kafka 2.8 and fully production-ready in Kafka 3.3+, KRaft eliminates ZooKeeper entirely.

Kafka brokers manage their own cluster metadata consensus using a specialized variant of the Raft consensus algorithm.

LEGACY ZOOKEEPER ARCHITECTURE:
Kafka Brokers <==============> External ZooKeeper Cluster (Dual State!)

MODERN KRAFT ARCHITECTURE:
+-----------------------------------------------------------------+
| KAFKA CLUSTER                                                   |
|                                                                 |
| [ Broker 1 ]   [ Broker 2 ]   [ Broker 3 ]                      |
|                                                                 |
| KRaft Controller Quorum (Active Leader Controller + Followers)  |
| Event-driven @metadata log replicated directly to all brokers!  |
+-----------------------------------------------------------------+

How KRaft Works: The `@metadata` Event Log

In KRaft mode, cluster metadata is stored as an event stream inside a special single-partition internal topic named @metadata.

Topic creations, partition leader assignments, and ACL updates are appended to the @metadata log as immutable records:

@metadata LOG RECORD STREAM:
Offset 0: RegisterBrokerRecord (Broker 1)
Offset 1: CreateTopicRecord (Topic: "orders", Partitions: 3)
Offset 2: PartitionChangeRecord (Topic: "orders", Partition 0, Leader: Broker 2)

The Active Controller Quorum

A small subset of brokers (typically 3 or 5 nodes) are designated as KRaft Controller Nodes.

These nodes elect a single Active Controller using Raft consensus rules. The Active Controller appends new metadata records to the @metadata log and replicates them to follower controller nodes.

Why KRaft Delivers Microsecond Failovers

KRaft transforms metadata propagation from a slow RPC push model into an asynchronous event-driven log stream.

All brokers in the cluster continuously fetch records from the @metadata log and update their local in-memory metadata caches in real time.

ACTIVE CONTROLLER                  BROKER 1 (Local Cache)       BROKER 2 (Local Cache)
Appends to @metadata log -------------> Fetches @metadata ---------> Fetches @metadata

Instant Controller Failover

If the Active Controller node crashes:

Remaining KRaft Controller nodes elect a new Active Controller using Raft consensus in less than 1 second.
Because all brokers have already consumed and applied @metadata records up to the latest offset, zero metadata reloading is required.

Controller failover time drops from 15 minutes down to less than 100 milliseconds, enabling Kafka clusters to scale effortlessly to 1,000,000+ partitions.

Migrating to KRaft

Modern Kafka deployments specify controller quorums directly in server.properties:

# KRaft Node Configuration
process.roles = broker,controller
node.id = 1
controller.quorum.voters = 1@kafka1:9093,2@kafka2:9093,3@kafka3:9093

# Storage Data Directory
metadata.log.dir = /var/lib/kafka/metadata-data

Zero external ZooKeeper processes required.

Quick Summary

External ZooKeeper management created dual-state bugs and limited cluster scalability to ~200,000 partitions.
KRaft (KIP-500) replaces ZooKeeper with an internal Raft consensus quorum running directly inside Kafka brokers.
Cluster metadata is stored as an event log in the internal @metadata topic.
All brokers consume the @metadata log asynchronously, reducing controller failover time from minutes to milliseconds and unlocking million-partition scale.

References & Further Reading

Confluent Inc. Confluent Schema Registry Overview & Compatibility Rules. Confluent Docs.
Apache Software Foundation. Apache Avro Specification v1.11.1. Apache Avro Docs.
Google Developers. Protocol Buffers Developer Guide (proto3). Google Docs.