The Kubernetes Pod Networking Model: Flat IP-per-Pod Networks and CNI Plugins

A microservice Pod running on Worker Node 01 (Pod A, IP 10.244.1.15) opens a TCP socket connection to a target Pod running on Worker Node 04 (Pod B, IP 10.244.2.88).

The connection succeeds instantly.

When Pod B inspects the incoming TCP connection header, it sees the source IP address:

Source IP: 10.244.1.15 (Pod A's exact IP!)

There is no Network Address Translation (NAT) modifying the IP header as the packet travels across nodes.

If you inspect Pod A’s routing table, it sees a flat, un-segmented network where every Pod in the cluster possesses a unique, routable IP address.

How does Kubernetes enforce a flat, cluster-wide IP network across hundreds of distinct physical worker nodes?

The answer lies in The Kubernetes Pod Networking Specification and Container Network Interface (CNI) Plugins.

---

1. The Four Rules of Kubernetes Pod Networking

Unlike traditional Docker installations (where containers on separate hosts use isolated 172.17.0.0/16 bridge subnets requiring port forwarding), Kubernetes enforces a strict IP-per-Pod Networking Specification.

[ Worker Node 01 (Host IP: 192.168.1.10) ]      [ Worker Node 02 (Host IP: 192.168.1.20) ]
+----------------------------------------+      +----------------------------------------+
| Pod A (10.244.1.15)                    |      | Pod B (10.244.2.88)                    |
+----------------------------------------+      +----------------------------------------+
                     \                              /
                      v                            v
      DIRECT CROSS-NODE POD-TO-POD COMMUNICATION WITHOUT NAT!
      (Pod A sends packet directly to 10.244.2.88 across host network)

Every Kubernetes network implementation must satisfy four fundamental rules:

1. Every Pod gets a unique IP address: Pods are treated like independent physical hosts on a network.
2. Pod-to-Pod Communication Without NAT: All Pods can communicate with all other Pods on any node without using Network Address Translation.
3. Node-to-Pod Communication Without NAT: Node daemons (like kubelet) can communicate directly with all Pods on their host.
4. IP Address Symmetry: The IP address a Pod sees as its own (ip addr) is identical to the IP address every other Pod sees it as.

---

2. The Container Network Interface (CNI)

Kubernetes does not contain a built-in default network implementation inside the kubelet binary.

Instead, Kubernetes defines an open specification called the Container Network Interface (CNI).

A CNI plugin is an executable binary stored in /opt/cni/bin/ (e.g., calico, cilium, flannel, aws-k8s-cni).

Kubelet Daemon                                     CNI Plugin (/opt/cni/bin/cilium)
  |                                                          |
  |--- 1. Executes CNI ADD Command via JSON stdio ---------->|
  |    Env: CNI_COMMAND=ADD, CNI_NETNS=/proc/14092/ns/net   |
  |                                                          |
  |                                 [ CNI Plugin Operations ]
  |                                 1. Allocates Pod IP from CIDR
  |                                 2. Creates veth pair
  |                                 3. Attaches veth to container netns
  |                                 4. Configures node routing table
  |                                                          |
  |<-- 2. Returns JSON Result (IP: 10.244.1.15, Gateway...) --|

The CNI Execution Contract:

When Kubelet prepares a Pod sandbox:

1. It invokes the CNI plugin executable passing environment variables (CNI_COMMAND=ADD, CNI_NETNS=<path>) and JSON configuration via stdin.
2. The CNI plugin configures the container’s network namespace, assigns an IP address, sets up veth virtual patch cables, and configures host routing tables.
3. The plugin returns a JSON response to Kubelet confirming the assigned IP and gateway.

---

3. How Cross-Node Pod Traffic Moves: Overlays vs Direct Routing

How do packets move from Pod A (10.244.1.15 on Node 1) to Pod B (10.244.2.88 on Node 2) across physical host networks (192.168.1.0/24)?

CNI plugins use two primary implementation strategies:

Strategy A: Overlay Networks (VXLAN / Geneve)

Used when physical host routers cannot be configured with custom Pod CIDR routes (e.g., heterogeneous cloud networks).

[ Pod A (10.244.1.15) ] ---> Sends IP Packet to Dst: 10.244.2.88
                                   |
                                   v
[ VXLAN CNI Device (Node 1) ] Encapsulates packet in UDP Packet:
                              Outer Src: 192.168.1.10 (Node 1)
                              Outer Dst: 192.168.1.20 (Node 2)
                              Inner Payload: 10.244.1.15 -> 10.244.2.88
                                   |
                                   v (Transmits over physical UDP network)
[ VXLAN CNI Device (Node 2) ] Decapsulates UDP packet, extracts Inner Payload,
                              and delivers raw packet to Pod B!

• Pros: Works on any underlying host network without special network hardware configuration.
• Cons: Minor CPU overhead due to packet encapsulation and decapsulation headers (UDP port 4789).

---

Strategy B: Direct Native Routing (BGP / AWS VPC CNI)

Used when physical or cloud network routers support direct IP route announcements (e.g., Calico BGP mode or AWS VPC CNI).

[ Node 1 Router (BGP) ] Announces: "Subnet 10.244.1.0/24 is reachable via 192.168.1.10"
[ Node 2 Router (BGP) ] Announces: "Subnet 10.244.2.0/24 is reachable via 192.168.1.20"

• Pros: Zero encapsulation overhead! Packets travel across physical network switches at native 100Gbps line rate.
• Cons: Requires underlying network infrastructure to support custom route table limits or BGP peer routing.

---

CNI Architecture Comparison Matrix

CNI Plugin	Default Routing Method	Encapsulation Overhead	NetworkPolicy Engine	Best Used For
Flannel	VXLAN Overlay	Low (UDP encapsulation)	❌ None (Requires Flannel + Calico)	Simple development clusters & light workloads
Calico	Direct BGP Routing / VXLAN	Zero (in BGP mode)	✅ Advanced eBPF / iptables rules	Enterprise hybrid-cloud production clusters
Cilium	eBPF Direct / VXLAN	Zero (in eBPF native mode)	✅ Layer 7 (HTTP/gRPC) eBPF Security	High-performance, high-security eBPF clusters
AWS VPC CNI	Native AWS ENI Allocation	Zero (Uses native AWS Subnet IPs)	✅ Native Security Groups	AWS EKS clusters requiring native VPC IP integration

---

Summary & Next Steps

The Kubernetes Pod Networking Model creates a unified cluster network:

• The IP-per-Pod Model treats Pods as independent network hosts, requiring direct pod-to-pod communication without NAT.
• CNI Plugins run as binaries executed by Kubelet to configure container network namespaces and IP allocations.
• Overlay Networks (VXLAN) encapsulate Pod traffic inside UDP packets to cross unmanaged host networks.
• Direct Native Routing (BGP / eBPF) advertises Pod subnets directly to host routers for native wire-speed performance.

In the next article, we examine Service Proxies Under the Hood: Virtual IPs, kube-proxy iptables, and IPVS Routing.

References & Further Reading

1. Cloud Native Computing Foundation. CNCF Container Storage Interface (CSI) Specification v1.5.0. CNCF GitHub.
2. CNCF. Kubernetes Storage: PersistentVolumes and Dynamic Provisioning. CNCF Docs.
3. CNCF Rook Project. Rook Cloud-Native Storage Orchestration for Kubernetes. Rook Docs.

---