Kafka Architecture Deep Dive: Topics, Partitions, and Offset Ordering Rules

In Part 3, we learned that Kafka stores data in an append-only log file.

Now imagine a real-world production system: an ride-sharing platform like Uber processing GPS location telemetry from 5,000,000 drivers every second.

If you store all GPS records inside a single log file on one server node, you hit physical hardware ceilings:

1. Storage Bound: A single server hard drive fills up within days.
2. Throughput Bound: A single network card and disk controller top out at ~500MB/s write throughput.
3. Consumer Bottleneck: Only one consumer thread can read a single file sequentially at a time.

To scale write throughput, storage capacity, and consumer reading speed across hundreds of servers, Kafka introduces three core storage abstractions: Topics, Partitions, and Offsets.

---

The Hierarchy: Topic $\rightarrow$ Partitions $\rightarrow$ Segment Logs

Kafka organizes data into a clean three-tier storage hierarchy:

TOPIC: "driver-gps-locations" (Logical Category)
  |
  +---> PARTITION 0 (Physical Log File on Broker 1) [ Offset 0 | 1 | 2 | 3 ... ]
  +---> PARTITION 1 (Physical Log File on Broker 2) [ Offset 0 | 1 | 2 | 3 ... ]
  +---> PARTITION 2 (Physical Log File on Broker 3) [ Offset 0 | 1 | 2 | 3 ... ]

1. Topic (Logical Concept)

A Topic is a logical category or stream name to which records are published (e.g. payment-events, order-created, user-signups). It functions like a table name in a database.

2. Partition (Physical Storage Unit)

A Topic is split into one or more Partitions. A partition is an actual physical directory on a Kafka broker node containing an append-only log file.

Partitioning is Kafka’s mechanism for horizontal scaling:

• Each partition can live on a separate physical server (broker) in the cluster.
• A topic with 100 partitions can spread disk storage and I/O write operations across 100 machines simultaneously.

3. Offset (Immutable Address)

Within a single partition, every record is assigned a sequential, monotonically increasing integer called an Offset.

An offset uniquely identifies a record within its partition:

$\text{Global Record Identifier} = (\text{Topic}, \text{Partition}, \text{Offset})$

---

Partition Ordering Guarantee Rules

One of the most misunderstood aspects of Kafka is its message ordering guarantees:

Kafka guarantees strict message ordering ONLY within a single partition. There is NO global ordering guarantee across different partitions of a topic.

PARTITION 0: [ Msg 0 ("UserA") ] -> [ Msg 1 ("UserA") ] -> [ Msg 2 ("UserA") ]  (Strict Order!)
PARTITION 1: [ Msg 0 ("UserB") ] -> [ Msg 1 ("UserC") ] -> [ Msg 2 ("UserB") ]  (Strict Order!)

If UserA sends Event 1 and Event 2 to Partition 0, Kafka guarantees any consumer reading Partition 0 will process Event 1 before Event 2.

However, if Event 1 lands in Partition 0 and Event 2 lands in Partition 1, two separate consumers may process them out of chronological order.

If your domain logic requires strict message ordering (e.g. processing bank account operations for User 981 in strict sequence), all events for User 981 must be routed to the same partition.

---

On-Disk Directory Anatomy

When you create a topic named orders with 3 partitions on a Kafka broker, Kafka creates physical folders inside its configured data directory (log.dirs=/var/lib/kafka/data):

/var/lib/kafka/data/
├── orders-0/
│   ├── 00000000000000000000.log
│   ├── 00000000000000000000.index
│   └── 00000000000000000000.timeindex
├── orders-1/
│   ├── 00000000000000000000.log
│   └── ...
└── orders-2/
    ├── 00000000000000000000.log
    └── ...

Each partition folder contains the actual .log data segment file and binary .index files.

---

How Many Partitions Should You Create?

Choosing the partition count for a topic determines your system’s maximum parallelism throughput ceiling.

Consider two key operational constraints when sizing partitions:

1. Target Throughput Formula: $\text{Partitions} = \max\left(\frac{\text{Target Producer Throughput}}{\text{Single Partition Write Speed}}, \frac{\text{Target Consumer Throughput}}{\text{Single Consumer Read Speed}}\right)$

   If your producers write at 100MB/s and a single consumer thread processes data at 10MB/s, you need at least 10 partitions to allow 10 consumer threads to process the topic in parallel.

2. Consumer Group Limit: A single partition can be read by at most one consumer thread inside a Consumer Group at any given time. If you have a topic with 3 partitions and spawn 10 consumer instances, 7 consumers sit completely idle!

---

Quick Summary

• A Topic is a logical stream name; a Partition is a physical append-only log directory on disk.
• Partitions enable horizontal scaling by spreading storage and I/O across multiple broker nodes.
• Kafka guarantees strict message ordering within a single partition, but not across partitions.
• Offsets provide an immutable 64-bit coordinate system for records within a partition.

References & Further Reading

1. Linux Man Pages. sendfile(2) — Transfer data between file descriptors. Linux Kernel Docs.
2. Oracle Corporation. OpenJDK FileChannel Documentation: FileChannel.transferTo(). Oracle Docs.
3. Stevens, W. R., & Rago, S. A. (2013). Advanced Programming in the UNIX Environment (3rd Edition). Addison-Wesley.

---