Kafka Producer Reliability: Balancing acks=all, min.insync.replicas & Data Loss

Suppose you are configuring Kafka producers for two different microservices inside your company:

1. Service A (Financial Payments): Ingests credit card authorization events. Losing a single message results in un-billed customer transactions and financial loss.
2. Service B (UI Clickstream Analytics): Ingests mouse movement data. Losing 5 messages out of 100,000 has zero impact on analytics dashboards.

If you configure both services using identical producer settings, you either waste cluster bandwidth on low-value telemetry or lose real money on payment failures.

Kafka allows developers to tune data durability and acknowledgment speed across a wide spectrum using three settings: acks, min.insync.replicas, and unclean.leader.election.enable.

---

The acks Spectrum Explained

The acks setting controls how many partition replicas must store a message before the broker sends an acknowledgment back to the producer.

PRODUCER                    LEADER BROKER                     FOLLOWER BROKER
   |                             |                                  |
   |--- producer.send(msg) ----->|                                  |
   |                             |--- Replicates to Follower ------>|
   |                             |<-- Ack from Follower ------------|
   |<-- Ack to Producer ---------|                                  |

---

1. acks = 0 (Fire-and-Forget / Maximum Throughput)

The producer sends a record batch over the network socket and immediately considers it successful without waiting for any response from the broker.

Producer ---> Network Socket (Zero waiting for broker ack!)

• Latency: Lowest possible latency.
• Throughput: Maximum throughput.
• Data Loss Risk: High. If the broker crashes, network switches drop packets, or disk space runs out, the producer never knows. Messages vanish silently.
• Use Case: High-frequency metrics, IoT telemetry, mouse movement tracking.

---

2. acks = 1 (Leader Acknowledgment / Balanced Default)

The producer waits for the partition leader broker to write the message batch to its local OS page cache before returning an acknowledgment.

Producer ---> Leader Broker (Writes to local page cache & returns Ack!)

• Latency: Medium latency (one network round trip to partition leader).
• Throughput: High throughput.
• Data Loss Risk: Moderate. If the partition leader node crashes after acknowledging the write but before follower replicas fetch the record, the un-replicated record is lost when a follower becomes the new leader.
• Use Case: Non-critical user notifications, audit logs, general event streaming.

---

3. acks = all or acks = -1 (Quorum Acknowledgment / Zero Data Loss)

The producer waits until the partition leader AND all active In-Sync Replicas (ISR) confirm they have written the record batch.

Producer ---> Leader Broker ---> Follower 1 (ISR) (Writes & Acks)
                   |-----------> Follower 2 (ISR) (Writes & Acks)
                   |<----------- All ISR Replicas Acked! Returns Ack to Producer!

• Latency: Higher latency (waits for multi-node network round trips).
• Throughput: Moderated throughput.
• Data Loss Risk: Zero (when combined with min.insync.replicas).
• Use Case: Financial transactions, order processing, inventory adjustments.

---

The Missing Link: min.insync.replicas

A common production mistake is setting acks = all without configuring min.insync.replicas.

Suppose a topic has a replication factor of 3 (1 Leader + 2 Followers). Over time, two follower brokers experience network disconnects and drop out of the In-Sync Replica (ISR) pool.

The ISR pool shrinks to just 1 node (the leader).

If min.insync.replicas is left at its default value of 1, acks = all succeeds after writing to only the single leader node! If that leader node dies, data is lost despite using acks = all.

To guarantee multi-node redundancy, configure min.insync.replicas on the topic:

# Topic / Broker Configuration
min.insync.replicas = 2

When acks = all and min.insync.replicas = 2:

• The broker requires at least 2 ISR nodes to confirm the write.
• If the ISR pool drops to 1 node, the leader rejects writes and returns a NotEnoughReplicasException to the producer, preventing silent single-node writes.

---

Dirty Leader Elections: unclean.leader.election.enable

What happens if the partition leader broker dies and all remaining follower replicas in the ISR pool are out-of-sync (behind by thousands of messages)?

Kafka presents a choice:

1. Option A: unclean.leader.election.enable = false (Default) Do not elect an out-of-sync follower as leader. The partition remains offline until the original leader recovers. Prioritizes Data Consistency over Availability.

2. Option B: unclean.leader.election.enable = true Elect an out-of-sync follower as the new leader immediately. Prioritizes Availability over Data Consistency. Un-replicated historical messages on the old leader are permanently discarded.

For financial systems, always leave unclean.leader.election.enable = false.

---

Durability Matrix Summary

Goal	acks	min.insync.replicas	Replication Factor	unclean.leader.election
Zero Data Loss (Financial)	all (-1)	2	3	false
High Performance (General)	1	1	3	false
Maximum Throughput (Telemetry)	0	1	1	true

---

Quick Summary

• acks=0 provides maximum speed with no broker confirmation; acks=1 confirms leader write; acks=all confirms full ISR write.
• acks=all requires min.insync.replicas = 2 to prevent single-node write degradation when followers fail.
• unclean.leader.election.enable = false prevents out-of-sync followers from taking over as leader, preserving log integrity.

References & Further Reading

1. Apache Software Foundation. Apache Kafka Documentation: Consumer Offset Management & __consumer_offsets Internal Topic. Apache Kafka Docs.
2. Shapira, G., et al. (2021). Kafka: The Definitive Guide (2nd Edition) — Chapter 4: Commits and Offsets. O’Reilly Media.
3. Kleppmann, M. (2017). Designing Data-Intensive Applications (Chapter 11: Stream Processing). O’Reilly Media.

---