Production Debugging and Operations: Event Loops, Pending Pods, and OOMKilled Analysis

At 02:15 AM, a critical microservice deployment crashes in production.

Running kubectl get pods shows:

NAME                      READY   STATUS             RESTARTS   AGE
payment-5f8b9c4d2-x9k42   0/1     CrashLoopBackOff   8          12m

A junior operations engineer attempts to fix the outage by blindly restarting worker nodes, deleting Pods, and re-applying random deployment YAML files.

The system outage extends for two hours.

When a senior site reliability engineer (SRE) logs in, they follow a Systematic 5-Step Debugging Framework:

kubectl describe pod payment-5f8b9c4d2-x9k42
kubectl logs payment-5f8b9c4d2-x9k42 --previous

Within 45 seconds, the root cause is identified: the application suffered an OOMKilled (Exit Code 137) failure because its cgroup memory limit (256Mi) was lower than the JVM initial heap footprint (300Mi).

Updating memory limits to 512Mi restores system stability immediately.

How do experienced engineers troubleshoot production Kubernetes workloads systematically using cluster events, exit codes, and diagnostic logs?

---

1. The Systematic 5-Step Debugging Framework

When a workload fails, avoid guessing or deleting cluster resources blindly. Follow the 5-step diagnostic escalation tree:

[ 1. STATUS OBSERVATION ] ---------> `kubectl get pods -o wide --all-namespaces`
      Check STATUS column, Node assignment, and RESTARTS count
            |
            v
[ 2. EVENT INSPECTION ] -----------> `kubectl describe pod <pod-name>`
      Inspect the `Events` array for FailedScheduling, OOMKilled, or Mount errors
            |
            v
[ 3. LOG DIAGNOSTICS ] ------------> `kubectl logs <pod-name> --previous`
      Read stdout/stderr stack traces from the PREVIOUS crashed container instance
            |
            v
[ 4. CLUSTER EVENT TIMELINE ] -----> `kubectl get events --sort-by='.metadata.creationTimestamp'`
      Cross-reference cluster-wide node, storage, and networking events
            |
            v
[ 5. IN-CONTAINER INSPECTION ] -----> `kubectl exec -it <pod-name> -- /bin/sh`
      Debug active network namespaces, environment variables, and local filesystems

---

2. Common Pod Failure States & Root Causes

A. Status: Pending

The Pod has been created in etcd, but the kube-scheduler cannot assign it to any worker node.

Events:
  Type     Reason            Age   From                Message
  ----     ------            ----  ----                -------
  Warning  FailedScheduling  2m    default-scheduler   0/10 nodes are available: 10 Insufficient memory.

• Root Causes:
  1. Insufficient Node Capacity: The Pod’s resource requests exceed available CPU/RAM on all nodes.
  2. Unmatched Taints/Affinities: Nodes are tainted, and the Pod lacks matching tolerations.
  3. Unbound PVC: The Pod requests a PersistentVolumeClaim that is still waiting for dynamic storage provisioning.

---

B. Status: CrashLoopBackOff

The container process starts, fails/crashes, exits with a non-zero exit code, and is restarted repeatedly by Kubelet. The restart delay grows exponentially ($10s, 20s, 40s \dots$ up to 5 minutes).

# Read logs of the container instance that JUST CRASHED:
kubectl logs <pod-name> --previous

• Root Causes:
  1. Application runtime exceptions (uncaught null-pointer exceptions, missing environment variables).
  2. Failed database connection timeouts.
  3. Incorrect CMD or ENTRYPOINT executable path in Dockerfile.

---

C. Status: OOMKilled (Exit Code 137)

The Linux kernel Out-Of-Memory Killer forcefully terminated the container process because it attempted to allocate more RAM than allowed by its cgroup limit (limits.memory).

State:          Terminated
  Reason:       OOMKilled
  Exit Code:    137

• Mathematical Proof: Exit Code $137 = 128 + 9$ (Signal 9 = SIGKILL).
• Fix: Increase limits.memory in the PodSpec or fix memory leaks in application code.

---

D. Status: ImagePullBackOff / ErrImagePull

The Kubelet cannot pull the specified container image from the registry.

• Root Causes:
  1. Image tag typo (my-app:v1.0.0-typo).
  2. Private registry authentication failure (missing imagePullSecrets).
  3. Registry rate limiting (HTTP 429).

---

E. Status: Terminating (Stuck)

A Pod remains stuck in Terminating status for hours and refuses to disappear.

• Root Cause: The resource object has active Finalizers (finalizers: [kubernetes.io/pv-protection]). Kubelet is waiting for a background resource (like a volume unmount) to complete before deleting the object.
• Emergency Fix (Use with caution):

  kubectl get pod <pod-name> -o json | jq '.metadata.finalizers = []' | kubectl replace -f -
  

---

Diagnostic Matrix by Error Indicator

Observed Symptom	Primary Diagnostic Command	Most Likely Cause	Corrective Action
Pending	kubectl describe pod <name>	Node resource exhaustion or Taints	Reduce resource requests or add worker nodes
CrashLoopBackOff	kubectl logs <name> --previous	Application exception on startup	Fix code bugs / config values shown in logs
OOMKilled (137)	kubectl describe pod <name>	Memory allocation exceeded cgroup limit	Increase limits.memory in PodSpec
ImagePullBackOff	kubectl describe pod <name>	Invalid image tag or registry auth error	Fix image name or add imagePullSecrets
Stuck Terminating	kubectl get pod <name> -o yaml	Finalizers blocking deletion	Inspect mount dependencies or clear finalizers

---

Summary & Next Steps

Systematic production debugging replaces guesswork with empirical evidence:

• Follow the 5-Step Escalation Tree: Status $\to$ Events $\to$ Previous Logs $\to$ Cluster Events $\to$ In-Container Inspection.
• kubectl describe pod exposes Events like FailedScheduling, FailedMount, and OOMKilled.
• kubectl logs --previous captures stack traces from crashed container instances.
• Exit Code 137 confirms kernel OOM Killer termination.

In the final article of this master series—Post 20: Building a Custom Kubernetes Operator in Java—we tie together everything we’ve learned by building a working Kubernetes Operator!

References & Further Reading

1. CNCF etcd Project. etcd Disaster Recovery, Backup, and Restore Operations. etcd Docs.
2. VMware / CNCF. Velero Disaster Recovery & Cluster Snapshot Architecture. Velero Docs.
3. Cloud Native Computing Foundation. Operating etcd Clusters for Kubernetes. CNCF Docs.

---