Operating Kafka in Production: Critical JMX Metrics, Kernel Tuning & Runbooks

Suppose it’s 2:00 AM on Cyber Monday.

Your site reliability engineering (SRE) dashboard flashes red. Consumer lag on your primary order-processing topic has spiked from 100 messages to 5,000,000 messages.

Customer credit card charges succeed, but order confirmation emails and warehouse fulfillment workflows freeze.

Is the bottleneck caused by a slow database downstream, a degraded consumer pod, broker disk skew, or an under-replicated partition?

Operating Kafka in production requires knowing which metrics signal impending failure and how to troubleshoot cluster degradation under pressure.

---

Part A: Partition Sizing & Capacity Planning

A common mistake in production is creating topics with arbitrary partition counts.

Rule of Thumb Guidelines:

• Maximum Partitions Per Broker: Keep total partition count under 4,000 partitions per broker node (under KRaft, clusters can scale higher, but individual broker memory overhead remains an issue).
• Maximum Partition Disk Size: Keep individual partition log directories under 25GB to 50GB. Re-syncing a failed 500GB partition across the network during node recovery consumes hours of network bandwidth.

$\text{Total Cluster Partitions} = \text{Brokers} \times \text{Target Partitions Per Broker (e.g. 2000)}$

---

Part B: The 5 Critical Production JMX Metrics

Set up alerts on your monitoring system (Datadog, Prometheus/Grafana) for these five metrics:

1. UnderReplicatedPartitions (Priority: CRITICAL P1)

• JMX MBean: kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions
• What it Means: The number of partitions where the active In-Sync Replica (ISR) count is less than the configured replication factor.
• Alert Condition: > 0 for more than 5 minutes. Signals broker node failure, disk corruption, or severe network degradation.

2. RecordsLag / ConsumerLag (Priority: HIGH P2)

• JMX MBean: kafka.consumer:type=consumer-fetch-manager-metrics,name=records-lag
• What it Means: The delta between the highest offset produced to a partition and the current offset committed by a consumer group.
• Alert Condition: Sustained upward trend over time. Indicates consumers cannot process incoming message volume.

3. IsrShrinksPerSec & IsrExpandsPerSec

• JMX MBean: kafka.server:type=ReplicaManager,name=IsrShrinksPerSec
• What it Means: The rate at which follower replicas drop out of the ISR pool.
• Alert Condition: > 0. Indicates follower brokers are suffering GC pauses or network packet drops.

4. OfflinePartitionsCount (Priority: CRITICAL P1)

• JMX MBean: kafka.controller:type=KafkaController,name=OfflinePartitionsCount
• What it Means: The number of partitions that have no active leader broker.
• Alert Condition: > 0. Total data read/write outage for affected partitions!

5. RequestHandlerAvgIdlePercent

• JMX MBean: kafka.server:type=KafkaRequestHandlerPool,name=RequestHandlerAvgIdlePercent
• What it Means: The percentage of time Kafka network request handler threads sit idle.
• Alert Condition: < 0.2 (20% idle). Signals the broker CPU is severely overloaded.

---

Part C: Linux OS Kernel Parameters for Kafka

Standard Linux kernel default parameters are tuned for desktop workloads, not high-throughput I/O servers.

Configure these kernel settings in /etc/sysctl.conf on every Kafka broker node:

# 1. Disable Swap (Prevents JVM memory swapping to disk!)
vm.swappiness = 1

# 2. Increase Max Memory-Mapped Files (Required for sparse index mmap calls)
vm.max_map_count = 1048576

# 3. Background Page Cache Flushes (Starts background flushes at 5% dirty memory)
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10

# 4. Increase Socket Read/Write Buffers for High-Bandwidth Networking
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

# 5. File Descriptor Limits (/etc/security/limits.conf)
# kafka soft nofile 100000
# kafka hard nofile 100000

---

Part D: Production Incident Troubleshooting Runbook

Incident Scenario A: Consumer Lag Spikes Exponentially

DIAGNOSTIC FLOWCHART: CONSUMER LAG SPIKE

1. Is Consumer Lag occurring across ALL partitions or just ONE partition?
   |
   +---> ALL Partitions: Downstream database/API issue. Check consumer thread pools.
   |
   +---> ONE Partition: Key Hotspotting! One producer key is over-subscribing 
                        a single partition (e.g. "NULL" key or huge enterprise account ID).

Remediation:

• If a single partition is lagging due to key hotspotting, check partitioner code or re-balance partition counts.
• If all partitions are lagging, scale out the consumer group by adding more consumer pods (up to total partition count).

---

Incident Scenario B: Broker Disk Skew (One Broker Hits 99% Disk Usage)

Over time, topics created with key-based partitioning develop uneven storage allocation across physical broker disks.

Remediation:

Do not attempt to move partition directories manually using OS mv commands!

Use Kafka’s kafka-reassign-partitions.sh tool with a re-throttled bandwidth limit:

# 1. Generate partition reassignment JSON plan
kafka-reassign-partitions.sh --bootstrap-server kafka1:9092 \
  --topics-to-move-json-file topics.json \
  --broker-list "1,2,3,4" --generate > reassignment.json

# 2. Execute reassignment with 50MB/s network throttle to prevent production lag!
kafka-reassign-partitions.sh --bootstrap-server kafka1:9092 \
  --reassignment-json-file reassignment.json \
  --throttle 50000000 --execute

---

Master Series Summary (Parts 1–18)

Over these 18 parts, we deconstructed Apache Kafka from bare-metal hardware fundamentals up to production operations:

1. Distributed Data Crisis: Solving $N^2$ point-to-point integration spaghetti with durable logs (Part 01).
2. Hardware Mechanics: Sequential disk I/O, OS Page Cache, and append-only log architecture (Parts 02-03).
3. Core Storage: Topics, Partitions, Offsets, Zero-Copy sendfile(), Segment Files, and Compaction (Parts 04-06).
4. Producer Internals: RecordAccumulator, linger.ms, MurmurHash2 keys, idempotence, and acks=all durability (Parts 07-09).
5. Consumer Groups: Pull model, __consumer_offsets, delivery semantics, and Cooperative Sticky rebalancing (Parts 10-12).
6. Replication & Consensus: High Watermark, ISR tracking, KRaft metadata quorums, and Exactly-Once Semantics (Parts 13-15).
7. Ecosystem & Operations: Avro Schema Registry, Kafka Connect vs Streams, OS kernel tuning, and production monitoring (Parts 16-18).

You now possess the foundational mental models to design, build, and operate resilient Kafka event-driven architectures.

References & Further Reading

1. Gregg, B. (2020). Systems Performance: Enterprise and the Cloud (2nd Edition) — Chapter 10: Network. Addison-Wesley.
2. Apache Software Foundation. Apache Kafka Documentation: Performance Tuning & OS Level Optimizations. Apache Kafka Docs.
3. Confluent Inc. Benchmarking Apache Kafka: 2 Million Writes Per Second. Confluent Tech Paper.

---