Building a Custom Distributed Rate Limiter & Resilience Gateway in Java: The System Design Capstone
Synthesizing rate limiting algorithms, token buckets, sliding window counters, circuit breakers, and singleflight cache locks into a runnable custom framework
Adetayo Akinsanya (unkletayo) • 2026-10-23
Part 20 in Series — Catch up on the previous article: The Master System Design Framework: 4-Step Methodology for Senior & Staff Architect Interviews (Part 19) before diving into this post.
Why You Need This in Real Life
Throughout this 20-part series, we have dissected System Design and Distributed Systems internals: the fallacies of distributed computing, PACELC and CAP trade-offs, Lamport and Vector Clocks, Consistent Hashing with Virtual Nodes, database sharding, Gossip protocols, 2PC vs Sagas, Paxos and Raft consensus, distributed locking with fencing tokens, rate limiting algorithms, Circuit Breakers, load balancing, distributed caching, CQRS, and OpenTelemetry tracing.
However, theoretical knowledge can remain abstract until you build these systems yourself.
In this final capstone post, we synthesize everything learned across the previous 19 parts by constructing a fully functional, runnable Distributed Rate Limiter & Resilience Gateway (MiniDistributedResilienceGateway) in plain Java without external third-party framework dependencies.
Our custom gateway will feature:
- Token Bucket Rate Limiter: Enforces configurable per-client request quotas with automatic token refilling based on elapsed time.
- Circuit Breaker State Engine: Manages Closed, Open, and Half-Open state transitions to isolate failing downstream services and fail fast under load.
- Singleflight Cache Loader: Prevents Cache Stampedes (Thundering Herd Problem) by ensuring only one thread queries the database on a cache miss while parallel requests wait safely.
- Resilience Gateway Dispatcher: Synthesizes rate limiting, circuit breaking, and caching into a unified execution pipeline.
Part 1: Architecture of MiniDistributedResilienceGateway
Our gateway consists of four integrated components:
+-----------------------------------------------------------------------------+
| MiniDistributedResilienceGateway Architecture |
| |
| Incoming HTTP Request |
| | |
| v |
| 1. TokenBucketRateLimiter (Per-Client Rate Check) |
| - Allowed? Proceed. Rejected? Throw 429 Too Many Requests. |
| | |
| v |
| 2. CircuitBreakerEngine (Downstream Dependency Guard) |
| - OPEN? Fail-Fast immediately. CLOSED/HALF-OPEN? Proceed. |
| | |
| v |
| 3. SingleflightCacheLoader (Deduplicated Data Retrieval) |
| - Cache Hit? Return instantly. Cache Miss? Single thread loads DB. |
| | |
| v |
| 4. Downstream Microservice Database |
+-----------------------------------------------------------------------------+
Part 2: Complete Runnable Capstone Source Code
Create a single file named MiniDistributedResilienceGateway.java and execute it with standard javac and java JDK tools:
package com.example.capstone;
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicReference;
import java.util.function.Function;
public class MiniDistributedResilienceGateway {
// =========================================================================
// 1. TOKEN BUCKET RATE LIMITER ENGINE
// =========================================================================
public static class TokenBucketRateLimiter {
private final int capacity;
private final double refillRatePerMs; // Tokens per millisecond
private double tokens;
private long lastRefillTimestamp;
public TokenBucketRateLimiter(int capacity, int refillTokensPerSecond) {
this.capacity = capacity;
this.refillRatePerMs = refillTokensPerSecond / 1000.0;
this.tokens = capacity;
this.lastRefillTimestamp = System.currentTimeMillis();
}
public synchronized boolean allowRequest() {
long now = System.currentTimeMillis();
long delta = Math.max(0, now - lastRefillTimestamp);
tokens = Math.min(capacity, tokens + (delta * refillRatePerMs));
lastRefillTimestamp = now;
if (tokens >= 1.0) {
tokens -= 1.0;
return true; // Request Allowed
}
return false; // Rate Limit Exceeded (HTTP 429)
}
public synchronized double getRemainingTokens() {
return tokens;
}
}
// =========================================================================
// 2. CIRCUIT BREAKER STATE MACHINE
// =========================================================================
public static class CircuitBreaker {
public enum State { CLOSED, OPEN, HALF_OPEN }
private final int failureThresholdPercent;
private final long sleepWindowMs;
private final int sampleSize = 10;
private final Boolean[] ringBuffer = new Boolean[sampleSize];
private int bufferIndex = 0;
private final AtomicReference<State> state = new AtomicReference<>(State.CLOSED);
private long lastStateChangeTimestamp = System.currentTimeMillis();
public CircuitBreaker(int failureThresholdPercent, long sleepWindowMs) {
this.failureThresholdPercent = failureThresholdPercent;
this.sleepWindowMs = sleepWindowMs;
}
public synchronized boolean allowExecution() {
long now = System.currentTimeMillis();
if (state.get() == State.OPEN) {
if (now - lastStateChangeTimestamp > sleepWindowMs) {
state.set(State.HALF_OPEN);
lastStateChangeTimestamp = now;
System.out.println(" [CIRCUIT BREAKER] Transitioned OPEN -> HALF-OPEN (Testing probe traffic)");
return true;
}
return false; // Reject fail-fast!
}
return true;
}
public synchronized void recordResult(boolean success) {
ringBuffer[bufferIndex % sampleSize] = success;
bufferIndex++;
if (state.get() == State.HALF_OPEN) {
if (success) {
state.set(State.CLOSED);
lastStateChangeTimestamp = System.currentTimeMillis();
System.out.println(" [CIRCUIT BREAKER] Probe Success! Transitioned HALF-OPEN -> CLOSED");
} else {
state.set(State.OPEN);
lastStateChangeTimestamp = System.currentTimeMillis();
System.out.println(" [CIRCUIT BREAKER] Probe Failed! Resetting to OPEN");
}
return;
}
if (state.get() == State.CLOSED) {
int failures = 0;
int count = 0;
for (Boolean b : ringBuffer) {
if (b != null) {
count++;
if (!b) failures++;
}
}
if (count >= sampleSize) {
int rate = (failures * 100) / count;
if (rate >= failureThresholdPercent) {
state.set(State.OPEN);
lastStateChangeTimestamp = System.currentTimeMillis();
System.out.println(" [CIRCUIT BREAKER] Failure rate (" + rate + "%) >= threshold (" + failureThresholdPercent + "%). TRIP OPEN!");
}
}
}
}
public State getState() {
return state.get();
}
}
// =========================================================================
// 3. SINGLEFLIGHT CACHE LOADER (CACHE STAMPEDE MITIGATION)
// =========================================================================
public static class SingleflightCache<K, V> {
private final ConcurrentHashMap<K, V> cache = new ConcurrentHashMap<>();
private final ConcurrentHashMap<K, CompletableFuture<V>> inFlight = new ConcurrentHashMap<>();
public V get(K key, Function<K, V> dbLoader) {
V cached = cache.get(key);
if (cached != null) {
return cached;
}
CompletableFuture<V> future = inFlight.computeIfAbsent(key, k -> {
CompletableFuture<V> f = new CompletableFuture<>();
CompletableFuture.runAsync(() -> {
try {
System.out.println(" [SINGLEFLIGHT DB LOAD] Querying Database for Key: " + k);
V loaded = dbLoader.apply(k);
if (loaded != null) cache.put(k, loaded);
f.complete(loaded);
} catch (Exception ex) {
f.completeExceptionally(ex);
} finally {
inFlight.remove(k);
}
});
return f;
});
return future.join();
}
}
// =========================================================================
// 4. UNIFIED RESILIENCE GATEWAY DISPATCHER
// =========================================================================
public static class ResilienceGateway {
private final ConcurrentHashMap<String, TokenBucketRateLimiter> clientLimiters = new ConcurrentHashMap<>();
private final CircuitBreaker circuitBreaker = new CircuitBreaker(50, 3000); // 50% fail, 3s sleep
private final SingleflightCache<String, String> cache = new SingleflightCache<>();
public String handleRequest(String clientId, String resourceId, Function<String, String> dbLoader) {
// Step 1: Enforce Rate Limiting
TokenBucketRateLimiter limiter = clientLimiters.computeIfAbsent(clientId, id -> new TokenBucketRateLimiter(5, 2));
if (!limiter.allowRequest()) {
throw new RuntimeException("HTTP 429 Too Many Requests for Client: " + clientId);
}
// Step 2: Enforce Circuit Breaker Guard
if (!circuitBreaker.allowExecution()) {
throw new RuntimeException("HTTP 503 Service Unavailable: Downstream Circuit OPEN");
}
// Step 3: Execute via Singleflight Cache
try {
String result = cache.get(resourceId, dbLoader);
circuitBreaker.recordResult(true);
return result;
} catch (Exception ex) {
circuitBreaker.recordResult(false);
throw ex;
}
}
public CircuitBreaker.State getCircuitState() {
return circuitBreaker.getState();
}
}
// =========================================================================
// 5. MAIN DEMONSTRATION EXECUTABLE
// =========================================================================
public static void main(String[] args) throws Exception {
System.out.println("==================================================================");
System.out.println(" Booting Custom Distributed Rate Limiter & Resilience Gateway ");
System.out.println("==================================================================");
ResilienceGateway gateway = new ResilienceGateway();
Function<String, String> dbLoader = key -> {
try { Thread.sleep(200); } catch (InterruptedException e) {} // Simulate DB latency
return "{\"product_id\": \"" + key + "\", \"price\": 199.99}";
};
// 1. Demonstrate Rate Limiting
System.out.println("\n--- 1. Testing Rate Limiter (5 Request Burst Capacity) ---");
for (int i = 1; i <= 7; i++) {
try {
String response = gateway.handleRequest("client_app_1", "prod_101", dbLoader);
System.out.println("Request " + i + ": SUCCESS -> " + response);
} catch (Exception e) {
System.err.println("Request " + i + ": REJECTED -> " + e.getMessage());
}
}
// 2. Demonstrate Singleflight Cache Stampede Protection
System.out.println("\n--- 2. Testing Singleflight Cache Stampede Protection (10 Parallel Threads) ---");
ExecutorService threadPool = Executors.newFixedThreadPool(10);
List<Future<?>> futures = new ArrayList<>();
for (int i = 0; i < 5; i++) {
futures.add(threadPool.submit(() -> {
try {
// Refill delay
Thread.sleep(1000);
String res = gateway.handleRequest("client_app_2", "prod_202", dbLoader);
System.out.println("Parallel Thread Received: " + res);
} catch (Exception e) {
System.err.println("Parallel Thread Error: " + e.getMessage());
}
}));
}
for (Future<?> f : futures) f.get();
threadPool.shutdown();
System.out.println("\n==================================================================");
System.out.println(" Capstone Execution Completed Successfully! ");
System.out.println("==================================================================");
}
}
Part 3: Framework Execution & Runtime Console Trace
When you run main(), MiniDistributedResilienceGateway outputs the following startup console logs:
==================================================================
Booting Custom Distributed Rate Limiter & Resilience Gateway
==================================================================
--- 1. Testing Rate Limiter (5 Request Burst Capacity) ---
[SINGLEFLIGHT DB LOAD] Querying Database for Key: prod_101
Request 1: SUCCESS -> {"product_id": "prod_101", "price": 199.99}
Request 2: SUCCESS -> {"product_id": "prod_101", "price": 199.99}
Request 3: SUCCESS -> {"product_id": "prod_101", "price": 199.99}
Request 4: SUCCESS -> {"product_id": "prod_101", "price": 199.99}
Request 5: SUCCESS -> {"product_id": "prod_101", "price": 199.99}
Request 6: REJECTED -> HTTP 429 Too Many Requests for Client: client_app_1
Request 7: REJECTED -> HTTP 429 Too Many Requests for Client: client_app_1
--- 2. Testing Singleflight Cache Stampede Protection (10 Parallel Threads) ---
[SINGLEFLIGHT DB LOAD] Querying Database for Key: prod_202
Parallel Thread Received: {"product_id": "prod_202", "price": 199.99}
Parallel Thread Received: {"product_id": "prod_202", "price": 199.99}
Parallel Thread Received: {"product_id": "prod_202", "price": 199.99}
Parallel Thread Received: {"product_id": "prod_202", "price": 199.99}
Parallel Thread Received: {"product_id": "prod_202", "price": 199.99}
==================================================================
Capstone Execution Completed Successfully!
==================================================================
Notice that during the 5 parallel requests for prod_202, only a single database query was executed ([SINGLEFLIGHT DB LOAD]), completely eliminating the Cache Stampede threat!
Conclusion & Series Master Summary
Congratulations! You have completed the System Design & Distributed Systems from First Principles master series.
Throughout this 20-part series, we have covered:
- Module 1: Deutsch’s 8 Fallacies of Distributed Computing, CAP Theorem trade-offs, PACELC model, physical clock skew, NTP drift, Lamport Timestamps, Vector Clocks, and causal consistency.
- Module 2: Consistent Hashing, Virtual Nodes (VNodes), database sharding (Range vs Hash), Snowflake 64-bit ID generation, dynamic shard splitting, and Gossip epidemic protocols (SWIM).
- Module 3: Distributed transactions (2PC vs Saga Pattern), Paxos Phase 1/2 voting, Raft leader election and log replication, Redlock mechanics, and ZooKeeper ephemeral fencing tokens.
- Module 4: Token Bucket, Leaky Bucket, and Sliding Window rate limiters, atomic Redis Lua scripts, Circuit Breaker state machines (Closed, Open, Half-Open), Layer 4 vs Layer 7 load balancing, and Power of Two Random Choices (P2C).
- Module 5: Distributed caching patterns (Cache-Aside, Write-Through, Write-Back), Singleflight mutex locks, XFetch probabilistic expiration, CQRS, Event Sourcing replay, and Kafka vs RabbitMQ message queues.
- Module 6: LSM-Trees vs B+ Trees, Bloom Filters, Multi-Region Active-Active replication, Google Spanner TrueTime commit wait, W3C
traceparentheaders, OpenTelemetry, and the 4-step System Design interview framework. - Module 7: Synthesizing all concepts into a custom, runnable Java Distributed Rate Limiter & Resilience Gateway capstone project.
References & Further Reading
- IETF. RFC 6585 — Additional HTTP Status Codes (429 Too Many Requests). Internet Engineering Task Force.
- Redis Ltd. Redis Lua Scripting & Atomic Memory Operations. Redis Docs.
- Stripe Engineering. Scaling your API with Rate Limiters and Load Shedders. Stripe Engineering Blog.
Series Status
Part 21 in this series is scheduled for upcoming release on the daily publication roadmap.