Enforcing Container Resource Limits: cgroups v1 vs v2 Memory and CPU Limits

A Java microservice container configured with --memory=512m restarts continuously in production.

Application logs show zero Java exceptions.

However, running docker inspect reveals the container exited with:

"State": {
    "Status": "exited",
    "ExitCode": 137,
    "OOMKilled": true
}

Exit Code 137 ( $128 + 9$ for SIGKILL) indicates the Linux kernel Out-Of-Memory (OOM) Killer terminated the container process.

Investigation reveals that the legacy JVM runtime inspected host system memory via /proc/meminfo, saw 64 Gigabytes of physical host RAM, and calculated its default maximum heap size as 16 Gigabytes—completely unaware of the 512MB container boundary!

How does the Linux kernel enforce CPU and RAM memory limits on containers?

To manage containerized workloads reliably, we must explore Control Groups (cgroups v1 vs v2) and the Completely Fair Scheduler (CFS).

1. cgroups v1 vs cgroups v2 Architecture

Control Groups (cgroups) are the Linux kernel subsystem responsible for metering, limiting, and accounting for process resource utilization (CPU, memory, disk I/O, network bandwidth).

cgroups v1 (Per-Subsystem Hierarchy Trees)       cgroups v2 (Unified Single Tree Hierarchy)

/sys/fs/cgroup/                                 /sys/fs/cgroup/
├── memory/                                     ├── cgroup.controllers
│   └── docker/container_1/                     ├── cgroup.procs
├── cpu/                                        └── docker-container_1/
│   └── docker/container_1/                         ├── memory.max
└── blkio/                                          ├── cpu.max
    └── docker/container_1/                         └── io.max

Key Differences:

cgroups v1 (Legacy): Maintained independent, fragmented hierarchy trees for each resource controller (/sys/fs/cgroup/memory, /sys/fs/cgroup/cpu). A process could belong to one node in the memory hierarchy and a completely different node in the CPU hierarchy, creating race conditions during page allocations.
cgroups v2 (Modern Unified Hierarchy): Replaced multi-tree controllers with a single unified tree structure in /sys/fs/cgroup/. All resource limits (CPU, memory, I/O) for a container process are managed under a single unified directory node.

2. Enforcing Memory Limits & OOM Killer Mechanics

When you launch a container with memory bounds:

docker run -d --name web --memory=512m --memory-swap=1g nginx

The container runtime creates a cgroup directory node and writes the byte threshold to the control files:

# cgroups v2 control path:
/sys/fs/cgroup/docker-<container_id>/memory.max   # Set to 536870912 (512MB)
/sys/fs/cgroup/docker-<container_id>/memory.swap.max # Set to 536870912 (512MB swap)

Memory Threshold Controls in cgroups v2:

memory.min: Hard memory protection. If memory usage is below memory.min, the kernel will never reclaim pages from the container.
memory.high: Throttle threshold. If memory exceeds memory.high, the kernel forces the container processes to slow down and perform synchronous page reclamation.
memory.max: Hard maximum limit. If memory usage exceeds memory.max and page reclamation fails, the kernel triggers the Out-Of-Memory (OOM) Killer.

How the OOM Killer Works

When a container exceeds memory.max:

The kernel ranks container processes by their oom_score.
It selects the process consuming the most memory and sends a uncatchable SIGKILL (signal 9).
The process terminates immediately, and Docker records OOMKilled: true with Exit Code 137.

3. CPU Allocation Mechanics: CFS Bandwidth Control

CPU allocation in Linux containers does not use dedicated CPU core pinning by default. It uses the kernel’s Completely Fair Scheduler (CFS) Bandwidth Control.

When you set --cpus=1.5:

docker run -d --name worker --cpus=1.5 my-app

The runtime configures two variables in /sys/fs/cgroup/.../cpu.max:

$\text{Quota} = \text{Period} \times \text{CPU Count}$

# Reads from /sys/fs/cgroup/docker-<container_id>/cpu.max:
# Format: <quota> <period>
150000 100000

period (100,000 $\mu s$ = 100 ms): The time window over which CPU utilization is measured.
quota (150,000 $\mu s$ = 150 ms): The total CPU execution time the container is permitted to consume across all host cores within each 100ms period.

If the container multi-threading code consumes 150ms of total CPU time within a 40ms real-time burst, the CFS scheduler throttles the container, forcing all container threads to sleep for the remaining 60ms of the period!

4. The JVM Container Awareness Problem

Historically, applications running inside Java 8 (prior to update 191) queried system resources via /proc/meminfo and /proc/cpuinfo.

Because /proc virtual files reflect the host system’s total hardware, a JVM running inside a 512MB container on a 64GB host calculated its heap size based on 64GB, triggering inevitable OOM Killer crashes.

Modern runtimes (Java 11+, Node 16+, Go 1.19+) inspect cgroup paths (/sys/fs/cgroup/memory.max) directly to determine true container limits.

For legacy Java runtimes, explicitly enable container awareness flags:

java -XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0 -jar app.jar

cgroups v1 vs cgroups v2 Comparison

Feature	cgroups v1	cgroups v2
Directory Tree Structure	Per-subsystem separate hierarchies (`/sys/fs/cgroup/memory`)	Single unified tree (`/sys/fs/cgroup/`)
Memory Limit Control File	`memory.limit_in_bytes`	`memory.max`
CPU Limit Control File	`cpu.cfs_quota_us` / `cpu.cfs_period_us`	`cpu.max` (unified `<quota> <period>`)
OOM Handling	Coarse per-subsystem kill	Granular container group kill (`memory.oom.group`)
Swap Control	Managed via `memory.memsw.limit_in_bytes`	Cleanly separated via `memory.swap.max`

Summary & Next Steps

Linux cgroups provide the metering boundary that makes multi-tenant container execution safe:

cgroups v2 replaces fragmented v1 controllers with a unified filesystem tree in /sys/fs/cgroup/.
memory.max enforces hard memory caps; exceeding it triggers the Linux OOM Killer (Exit Code 137).
CFS Bandwidth Control (cpu.max) enforces fractional CPU limits using quota/period ratios (e.g., 150ms per 100ms period).
Runtimes must be container-aware to inspect cgroup paths rather than raw /proc host files.

In the next article, we transition to Module 4 and explore Container Virtual Networking: Bridge Networks, veth Pairs, and Network Namespaces.

References & Further Reading

Docker Inc. Best Practices for Writing Dockerfiles & Multi-Stage Builds. Docker Docs.
Moby Project. BuildKit Concurrent Execution Engine Architecture. GitHub.
Open Container Initiative. OCI Image Format Specification v1.0.2. OCI Standard.