The Append-Only Log Abstraction: Why Immutability Rules Event Streaming

Suppose you are auditing a traditional financial bank ledger.

An accountant records a deposit of $500 at 9:00 AM. At 10:00 AM, the customer withdraws $200.

How does the bank ledger record this change? Does the accountant erase the original $500 entry with an eraser and overwrite the paper page with $300?

No. Erasing historical paper records is illegal in financial accounting.

Instead, the accountant appends a new entry to the end of the journal: Withdrawal: -$200. Current account balance is computed by reading the ordered sequence of historical transactions from the start of the log to the end.

This fundamental data structure is the Append-Only Log.

Why You Need This in Real Life

In software engineering, developers often confuse a log file (app.log text lines) with the Log Data Structure.

The Log as an abstract data structure is an ordered, immutable, append-only sequence of records.

THE APPEND-ONLY LOG ABSTRACTION

Offset:      0        1        2        3        4        5
        +--------+--------+--------+--------+--------+--------+
        | Record | Record | Record | Record | Record | Record | ---> Append New Records
        +--------+--------+--------+--------+--------+--------+

Every incoming record receives a monotonically increasing 64-bit integer identifier called an Offset.

Once a record is written to a log, it becomes immutable. It cannot be edited, overwritten, or re-ordered.

The Power of Immutability in Distributed Systems

Mutating state in place is the source of concurrency bugs in software development.

Consider what happens when multiple services access a mutable database row:

MUTABLE DATABASE STATE (Row ID 42)
Thread A reads row 42 -> Thread B overwrites row 42 -> Thread A writes stale data!
(Requires heavy lock synchronization across threads)

When data is stored inside an immutable append-only log, lock contention vanishes:

Zero Lock Concurrency: Readers can scan past log records at high speed while a writer appends new records to the tail of the file. Readers and writers do not block each other.
Simplified Replication: Replicating state across servers reduces to copying log bytes sequentially from offset $N$ to offset $N+K$ .
Deterministic Replay: If an application service crashes mid-processing, it can recover its exact state by replaying records from offset 0 up to its last committed checkpoint offset.

Multiple Independent Consumers with Custom Pointers

In a traditional message queue (e.g. JMS, RabbitMQ), the broker tracks which messages have been consumed and deletes them from memory.

In an append-only log, the log remains untouched by readers.

Instead of the broker tracking read state, every consumer maintains its own independent offset read pointer.

LOG OFFSET DATA
Offset:    0      1      2      3      4      5      6      7
        +------+------+------+------+------+------+------+------+
        | Msg  | Msg  | Msg  | Msg  | Msg  | Msg  | Msg  | Msg  |
        +------+------+------+------+------+------+------+------+
                   ^                    ^                    ^
                   |                    |                    |
        Consumer Group A Offset: 1   Consumer Group B Offset: 4   Producer Write Tail: 7

This simple design delivers massive operational flexibility:

Independent Reading Speeds: Consumer A can process real-time events at offset 7, while Consumer B (a batch analytics job) slowly processes historical events at offset 1.
Zero Reader Interference: Adding 50 new consumer services to read the log adds zero write lock contention.
Reprocessing & Bug Recovery: If a bug corrupts downstream data processing on Tuesday, developers can reset Consumer Group A’s read offset pointer back to Monday’s offset and replay the entire stream.

Building a Toy Append-Only Log in Java

To understand how an append-only log functions on disk, let’s build a minimal ToyLogManager that writes fixed-length records and reads entries by offset index:

import java.io.File;
import java.io.RandomAccessFile;
import java.nio.ByteBuffer;
import java.nio.channels.FileChannel;

public class ToyLogManager {
    private static final int RECORD_HEADER_SIZE = 12; // 8 bytes offset + 4 bytes length
    private final FileChannel logChannel;
    private long currentOffset = 0;

    public ToyLogManager(File logFile) throws Exception {
        @SuppressWarnings("resource")
        RandomAccessFile raf = new RandomAccessFile(logFile, "rw");
        this.logChannel = raf.getChannel();
        this.currentOffset = calculateOffsetCount();
    }

    public synchronized long append(byte[] payload) throws Exception {
        long assignedOffset = currentOffset;

        ByteBuffer buffer = ByteBuffer.allocate(RECORD_HEADER_SIZE + payload.length);
        buffer.putLong(assignedOffset);       // 8 bytes: Offset
        buffer.putInt(payload.length);        // 4 bytes: Payload Length
        buffer.put(payload);                 // Payload bytes
        buffer.flip();

        logChannel.write(buffer, logChannel.size()); // Append to physical end of file
        currentOffset++;
        return assignedOffset;
    }

    public byte[] readRecord(long targetOffset) throws Exception {
        logChannel.position(0);
        ByteBuffer headerBuffer = ByteBuffer.allocate(RECORD_HEADER_SIZE);

        while (logChannel.position() < logChannel.size()) {
            headerBuffer.clear();
            int read = logChannel.read(headerBuffer);
            if (read < RECORD_HEADER_SIZE) break;

            headerBuffer.flip();
            long offset = headerBuffer.getLong();
            int length = headerBuffer.getInt();

            if (offset == targetOffset) {
                ByteBuffer payloadBuffer = ByteBuffer.allocate(length);
                logChannel.read(payloadBuffer);
                return payloadBuffer.array();
            } else {
                // Skip payload bytes to reach next record header
                logChannel.position(logChannel.position() + length);
            }
        }
        return null; // Offset not found
    }

    private long calculateOffsetCount() throws Exception {
        // Counts existing records in log file on startup
        long count = 0;
        logChannel.position(0);
        ByteBuffer headerBuffer = ByteBuffer.allocate(RECORD_HEADER_SIZE);
        while (logChannel.position() < logChannel.size()) {
            headerBuffer.clear();
            if (logChannel.read(headerBuffer) < RECORD_HEADER_SIZE) break;
            headerBuffer.flip();
            headerBuffer.getLong();
            int length = headerBuffer.getInt();
            logChannel.position(logChannel.position() + length);
            count++;
        }
        return count;
    }
}

This toy implementation highlights two core log characteristics:

Records append to the end of the channel file without seeking or mutating existing bytes.
Every record has a permanent offset position.

In production Kafka, scanning records from the start of a file is optimized using sparse index files, which we will build in Part 06.

Quick Summary

An append-only log is an ordered, immutable sequence of records where each record receives a sequential 64-bit offset.
Immutability eliminates lock contention between readers and writers, simplifying multi-node replication.
Consumers maintain their own independent offset read pointers, enabling multi-team data sharing, independent reading speeds, and historical reprocessing.

References & Further Reading

Apache Software Foundation. Apache Kafka Documentation: Log Anatomy & Segment Configuration. Apache Kafka Docs.
Shapira, G., et al. (2021). Kafka: The Definitive Guide (2nd Edition) — Chapter 5: Reliable Data Delivery. O’Reilly Media.
Kreps, J. (2013). The Log. LinkedIn Engineering Blog.