Adetayo Akinsanya unkletayo.dev

Multi-Region Distributed Databases: Active-Active vs Active-Passive Cross-Data-Center Replication

Deconstructing cross-datacenter replication, write conflict resolution, split-brain recovery, and TrueTime commit wait

Part 17 in Series — Catch up on the previous article: Distributed Storage Engines: LSM-Trees (Cassandra/RocksDB) vs B+ Trees (Spanner/CockroachDB) (Part 16) before diving into this post.

Why You Need This in Real Life

A global e-commerce application serves users in New York (US-East) and London (EU-West).

If all database writes are routed to a single primary database in AWS US-East-1, European users experience 140ms cross-Atlantic network round-trip latency on every single database write operation.

To provide sub-10ms response latency globally and survive entire datacenter outages, engineering teams deploy Multi-Region Geographical Database Replication.

However, replicating data across continents introduces physical speed-of-light constraints: light in fiber-optic cable takes 70ms\sim 70\text{ms} to travel from New York to London. Synchronous cross-region writes introduce unacceptable latency, while asynchronous cross-region writes risk write conflict collisions and split-brain data loss.

To architect globally distributed platforms, you must master Active-Passive Replication, Active-Active Replication, and Google Spanner’s TrueTime Commit Wait.


Part 1: Active-Passive (Primary-Standby) Replication

In an Active-Passive architecture, a single datacenter accepts all write traffic (Primary), while read-only Standby replicas in secondary regions replicate changes asynchronously:

US-East (Active Primary)                              EU-West (Passive Standby)
Client Writes ---> [ Primary DB ]                                [ Standby DB ]
                       |                                              ^
                       | Async Cross-Region Replication (~70ms)       |
                       +----------------------------------------------+

Properties

  • Writes: Route to US-East Primary.
  • Reads: Can be served locally by EU-West Standby (eventual consistency).
  • Failover: If US-East crashes, an operator or automated failover orchestrator promotes EU-West Standby to Primary.
  • Pros: Zero write conflict risk (single writer).
  • Cons: High write latency for non-primary regions; potential data loss during sudden primary region failure (asynchronous replication gap).

Part 2: Active-Active (Multi-Primary) Replication

In an Active-Active architecture, database instances in all regions accept concurrent read and write operations locally:

US-East (Active Region 1)                             EU-West (Active Region 2)
US Client Writes ---> [ Region 1 DB ] <-------------> [ Region 2 DB ] <--- EU Client Writes
                                     Bidirectional Async
                                     Replication (~70ms)

The Cross-Region Write Conflict Problem

User A in New York updates their email address to [email protected] on Region 1 DB at 12:00:00.000. At 12:00:00.010 (before Region 1 replication reaches Region 2), User A updates their email to [email protected] on Region 2 DB.

Both regions accept the local write. When the replication payloads cross the Atlantic, Region 1 and Region 2 have conflicting values for the same row!


Part 3: Conflict Resolution Strategies

Distributed Active-Active databases resolve write conflicts using four primary techniques:

1. Last-Write-Wins (LWW)

Uses physical timestamps to pick the latest write.

  • Risk: Clock skew between region servers can overwrite a newer write with an older one.

2. Conflict-Free Replicated Data Types (CRDTs)

Mathematically structured data types (e.g., PN-Counters, LWW-Element-Set) that automatically merge concurrent state updates deterministically regardless of arrival order.

3. Application-Level Merge Handlers

Saves both conflicting write versions as siblings (e.g., Vector Clocks in DynamoDB) and forces application code to resolve the conflict upon next read.

4. TrueTime Commit Wait (Google Spanner)

Google Spanner uses GPS receivers and atomic clocks in every datacenter to bound clock uncertainty to ϵ1ms\epsilon \le 1\text{ms}. When a transaction commits, Spanner waits for 2ϵ2\epsilon before releasing locks (Commit Wait), guaranteeing globally linearizable ordering across continents without cross-region locks!


Part 4: Architectural Trade-Off Summary

DimensionActive-PassiveActive-ActiveGlobal Consensus (Spanner)
Write LatencyLow in Primary region; High in Secondary regions.Ultra-low locally in all regions (<5ms< 5\text{ms}).High (Bounded by cross-region Paxos round-trips).
Read LatencyLow globally.Low globally.Low globally.
Write ConflictsNone (Single writer).High (Requires CRDTs / LWW / Vector Clocks).None (Globally linearizable via TrueTime).
Disaster RecoveryManual/Automated failover; minor data loss window.Instant zero-downtime failover.Automatic zero-downtime failover.

Next Steps

Now that we understand multi-region replication and conflict resolution, we will explore Distributed Tracing & Observability in Part 18: dissecting W3C Trace Context headers and OpenTelemetry.

References & Further Reading

  1. Twitter Engineering. (2010). Announcing Snowflake: Distributed ID Generator. Twitter Tech Blog.
  2. IETF. RFC 9562 — Universally Unique IDentifiers (UUID). Internet Engineering Task Force.
  3. Instagram Engineering. (2012). Sharded IDs at Scale. Instagram Tech Blog.

Up Next in Series →

Part 18: Distributed Tracing & Observability: Trace Context Propagation, OpenTelemetry, and W3C Headers

Continue to Part 18 →