Multi-Region Distributed Databases: Active-Active vs Active-Passive Cross-Data-Center Replication
Deconstructing cross-datacenter replication, write conflict resolution, split-brain recovery, and TrueTime commit wait
Part 17 in Series — Catch up on the previous article: Distributed Storage Engines: LSM-Trees (Cassandra/RocksDB) vs B+ Trees (Spanner/CockroachDB) (Part 16) before diving into this post.
Why You Need This in Real Life
A global e-commerce application serves users in New York (US-East) and London (EU-West).
If all database writes are routed to a single primary database in AWS US-East-1, European users experience 140ms cross-Atlantic network round-trip latency on every single database write operation.
To provide sub-10ms response latency globally and survive entire datacenter outages, engineering teams deploy Multi-Region Geographical Database Replication.
However, replicating data across continents introduces physical speed-of-light constraints: light in fiber-optic cable takes to travel from New York to London. Synchronous cross-region writes introduce unacceptable latency, while asynchronous cross-region writes risk write conflict collisions and split-brain data loss.
To architect globally distributed platforms, you must master Active-Passive Replication, Active-Active Replication, and Google Spanner’s TrueTime Commit Wait.
Part 1: Active-Passive (Primary-Standby) Replication
In an Active-Passive architecture, a single datacenter accepts all write traffic (Primary), while read-only Standby replicas in secondary regions replicate changes asynchronously:
US-East (Active Primary) EU-West (Passive Standby)
Client Writes ---> [ Primary DB ] [ Standby DB ]
| ^
| Async Cross-Region Replication (~70ms) |
+----------------------------------------------+
Properties
- Writes: Route to US-East Primary.
- Reads: Can be served locally by EU-West Standby (eventual consistency).
- Failover: If US-East crashes, an operator or automated failover orchestrator promotes EU-West Standby to Primary.
- Pros: Zero write conflict risk (single writer).
- Cons: High write latency for non-primary regions; potential data loss during sudden primary region failure (asynchronous replication gap).
Part 2: Active-Active (Multi-Primary) Replication
In an Active-Active architecture, database instances in all regions accept concurrent read and write operations locally:
US-East (Active Region 1) EU-West (Active Region 2)
US Client Writes ---> [ Region 1 DB ] <-------------> [ Region 2 DB ] <--- EU Client Writes
Bidirectional Async
Replication (~70ms)
The Cross-Region Write Conflict Problem
User A in New York updates their email address to [email protected] on Region 1 DB at 12:00:00.000. At 12:00:00.010 (before Region 1 replication reaches Region 2), User A updates their email to [email protected] on Region 2 DB.
Both regions accept the local write. When the replication payloads cross the Atlantic, Region 1 and Region 2 have conflicting values for the same row!
Part 3: Conflict Resolution Strategies
Distributed Active-Active databases resolve write conflicts using four primary techniques:
1. Last-Write-Wins (LWW)
Uses physical timestamps to pick the latest write.
- Risk: Clock skew between region servers can overwrite a newer write with an older one.
2. Conflict-Free Replicated Data Types (CRDTs)
Mathematically structured data types (e.g., PN-Counters, LWW-Element-Set) that automatically merge concurrent state updates deterministically regardless of arrival order.
3. Application-Level Merge Handlers
Saves both conflicting write versions as siblings (e.g., Vector Clocks in DynamoDB) and forces application code to resolve the conflict upon next read.
4. TrueTime Commit Wait (Google Spanner)
Google Spanner uses GPS receivers and atomic clocks in every datacenter to bound clock uncertainty to . When a transaction commits, Spanner waits for before releasing locks (Commit Wait), guaranteeing globally linearizable ordering across continents without cross-region locks!
Part 4: Architectural Trade-Off Summary
| Dimension | Active-Passive | Active-Active | Global Consensus (Spanner) |
|---|---|---|---|
| Write Latency | Low in Primary region; High in Secondary regions. | Ultra-low locally in all regions (). | High (Bounded by cross-region Paxos round-trips). |
| Read Latency | Low globally. | Low globally. | Low globally. |
| Write Conflicts | None (Single writer). | High (Requires CRDTs / LWW / Vector Clocks). | None (Globally linearizable via TrueTime). |
| Disaster Recovery | Manual/Automated failover; minor data loss window. | Instant zero-downtime failover. | Automatic zero-downtime failover. |
Next Steps
Now that we understand multi-region replication and conflict resolution, we will explore Distributed Tracing & Observability in Part 18: dissecting W3C Trace Context headers and OpenTelemetry.
References & Further Reading
- Twitter Engineering. (2010). Announcing Snowflake: Distributed ID Generator. Twitter Tech Blog.
- IETF. RFC 9562 — Universally Unique IDentifiers (UUID). Internet Engineering Task Force.
- Instagram Engineering. (2012). Sharded IDs at Scale. Instagram Tech Blog.
Part 18: Distributed Tracing & Observability: Trace Context Propagation, OpenTelemetry, and W3C Headers
Continue to Part 18 →