Load Balancing Architectures: Layer 4 vs Layer 7, Consistent Hash, and Power of Two Choices

Why You Need This in Real Life

A microservices platform migrates its internal communication from REST (HTTP/1.1) to gRPC (HTTP/2) to improve throughput. They keep their existing Layer 4 AWS Network Load Balancer (NLB) in front of 10 backend pods.

Under load tests, Node 1 CPU utilization hits 100%, while the remaining 9 backend nodes sit completely idle at 2% CPU.

Why did 90% of backend hardware sit idle?

Because Layer 4 load balancers operate at the TCP transport layer. They route IP sockets, not individual HTTP requests. Since gRPC multiplexes thousands of RPC calls over a single long-lived TCP connection, the L4 load balancer bound the single TCP connection to Node 1. Every single gRPC request traveled to Node 1!

To build scalable infrastructure, you must understand the trade-offs between Layer 4 (Transport) and Layer 7 (Application) load balancing, gRPC connection pooling traps, and advanced balancing algorithms like Power of Two Random Choices (P2C).

Part 1: Layer 4 vs Layer 7 Load Balancing

Layer 4 Load Balancer (TCP / IP Level)
Client TCP Socket ---------------> [ L4 Load Balancer ] ---------------> Backend Server 1 (TCP Connection)
                                   (Inspects IP & Port Only)

Layer 7 Load Balancer (HTTP / gRPC Level)
Client Request 1 ---\
Client Request 2 ------> [ L7 Load Balancer ] ---> Request 1 -> Backend Server 1
Client Request 3 ---/   (Terminates TLS & Parses HTTP Headers) -> Request 2 -> Backend Server 2
                                                               -> Request 3 -> Backend Server 3

Comparative Breakdown

Feature	Layer 4 (Transport Load Balancer)	Layer 7 (Application Load Balancer)
OSI Layer	Layer 4 (TCP / UDP Packets).	Layer 7 (HTTP / HTTPS / gRPC / WebSockets).
Packet Inspection	Inspects IP address, TCP port, SYN/ACK flags.	Inspects URL path, HTTP headers, cookies, payload.
TLS Termination	Passes raw encrypted TCP bytes (Direct Server Return).	Terminates TLS, decrypts payload, inspects headers.
Performance & CPU	Ultra-high throughput ( $1,000,000+$ packets/sec).	Higher CPU overhead (TLS decryption, HTTP parsing).
gRPC Multiplexing	Fails to load balance gRPC requests (TCP pin).	Successfully balances individual gRPC frames.

Part 2: The gRPC HTTP/2 Load Balancing Dilemma

HTTP/1.1 opens a fresh TCP connection per request (or reuses a pool of short-lived connections). L4 load balancers handle HTTP/1.1 acceptably because connections open and close frequently.

HTTP/2 multiplexes hundreds of concurrent streams over a single persistent TCP connection.

gRPC Client ------------ Single Long-Lived TCP Connection ------------> L4 Load Balancer
                                                                               |
                                                                  Bound to Server A ONLY!
                                                                  (Servers B & C receive 0 requests!)

Solutions for gRPC Load Balancing

Layer 7 Reverse Proxy (Envoy / NGINX): The L7 proxy terminates the client’s HTTP/2 TCP connection, inspects individual gRPC frames, and distributes requests across backend servers over a pool of upstream connections.
Client-Side Load Balancing: The gRPC client queries a service discovery engine (e.g., CoreDNS, Consul), obtains IPs for all 10 backend pods, and maintains a direct HTTP/2 connection to every pod, balancing requests locally.

Part 3: Load Balancing Algorithms

1. Round Robin & Weighted Round Robin

Routes requests sequentially across servers ( $S_1 \rightarrow S_2 \rightarrow S_3 \rightarrow S_1$ ). Weighted Round Robin assigns higher proportions of traffic to servers with more CPU/RAM.

2. Least Connections

Routes requests to the server with the fewest active TCP connections. Ideal for long-running connections (WebSockets, database pools).

3. Least Response Time

Routes traffic to the server with the lowest combination of active connections and lowest response latency.

4. Power of Two Random Choices (P2C)

Randomly selects two servers from the cluster, compares their active load, and routes the request to the less loaded of the two!

Total Servers: [ S1, S2, S3, S4, S5, S6, S7, S8, S9, S10 ]

1. Randomly pick 2 servers: [ S3 (Load: 15 req), S7 (Load: 4 req) ]
2. Compare load: S7 (4 req) < S3 (15 req)
3. Route request to S7!

Why P2C Beats Pure Least Connections at Scale

Michael Mitzenmacher proved mathematically that selecting the best of two random choices dramatically reduces maximum server load compared to pure random routing ( $\mathcal{O}(\log \log N)$ vs $\mathcal{O}(\frac{\log N}{\log \log N})$ ), while avoiding the centralized load metrics synchronization bottleneck required by global Least Connections algorithms.

Part 4: Runnable Java Power of Two Choices (P2C) Implementation

package com.example.loadbalancer;

import java.util.*;
import java.util.concurrent.ThreadLocalRandom;

public class PowerOfTwoLoadBalancer {

    public record ServerNode(String address, int activeConnections) {}

    public ServerNode selectServer(List<ServerNode> servers) {
        if (servers == null || servers.isEmpty()) {
            throw new IllegalArgumentException("No backend servers available");
        }

        if (servers.size() == 1) {
            return servers.get(0);
        }

        // 1. Pick two distinct random indices
        int idx1 = ThreadLocalRandom.current().nextInt(servers.size());
        int idx2;
        do {
            idx2 = ThreadLocalRandom.current().nextInt(servers.size());
        } while (idx1 == idx2);

        ServerNode server1 = servers.get(idx1);
        ServerNode server2 = servers.get(idx2);

        // 2. Route to the node with fewer active connections
        System.out.printf("[P2C SELECTOR] Comparing Node %s (%d conns) vs Node %s (%d conns)%n",
                server1.address(), server1.activeConnections(),
                server2.address(), server2.activeConnections());

        return (server1.activeConnections() <= server2.activeConnections()) ? server1 : server2;
    }
}

Next Steps

Now that we understand load balancing architectures, L4 vs L7, and Power of Two Choices, we will enter Module 5 (Distributed Caching & Message Queuing): starting with Distributed Caching Patterns in Part 13.

References & Further Reading

O’Neil, P., O’Neil, E., & Weikum, G. (1996). The Log-Structured Merge-Tree (LSM-Tree). Acta Informatica, 33(4), 351–385.
Comer, D. (1979). The Ubiquitous B-Tree. ACM Computing Surveys, 11(2), 121–137.
Manning, C. D., Raghavan, P., & Schütze, H. (2008). Introduction to Information Retrieval (Chapter 1: Inverted Indexes). Cambridge University Press.