Book Notes: Designing Data-Intensive Applications by Martin Kleppmann
Martin Kleppmann’s Designing Data-Intensive Applications (DDIA) is widely regarded as the definitive guide to distributed systems storage, replication, and query engines.
Here are my top architectural takeaways and notes structured by fundamental data system dimensions.
1. Reliability, Scalability, and Maintainability
- Reliability: Continuing to work correctly even when hardware faults, software bugs, or human errors occur.
- Scalability: The system’s ability to cope with increased load (throughput, response times, percentile tail latencies like p99/p99.9).
- Maintainability: Operability, simplicity (removing accidental complexity), and evolvability.
2. Storage Engines: B-Trees vs LSM-Trees
| Storage Engine Dimension | B-Trees | Log-Structured Merge (LSM) Trees |
|---|---|---|
| Primary Workload | Read-Heavy OLTP Workloads | Write-Heavy Streaming & Analytics |
| Write Operation | Overwrites fixed-size pages (4KB-8KB) in-place | Append-only to MemTable & SSTables |
| Write Amplification | Higher (Page writes + WAL log) | Lower (Sequential disk writes) |
| Fragmentation | Page fragmentation requires vacuuming | Background compaction merges SSTables |
3. Replication & Consensus
- Single-Leader Replication: All writes go to the leader. Easy to implement, but vulnerable to leader failover delays.
- Multi-Leader Replication: Useful for multi-datacenter deployments. Requires conflict resolution strategies (Last-Write-Wins, CRDTs).
- Leaderless Replication (Dynamo-style): Relies on Quorum reads/writes () and read-repair mechanisms.
Key Reflections
Building resilient data systems requires accepting that hardware will fail, networks will partition, and clocks will drift. Architecture is the art of defining clean boundaries and fallback semantics when these failures occur.