Kafka Schema Registry & Avro: Guarding Against Breaking Payload Changes

Suppose a producer engineering team deploys a new version of an Order Payment service.

The team decides to rename a payload field inside event JSON messages:

// OLD PAYLOAD (Version 1):
{ "order_id": 101, "user_email": "[email protected]", "total_amount": 49.99 }

// NEW PAYLOAD (Version 2 deployed by Producer team):
{ "order_id": 101, "email": "[email protected]", "amount_cents": 4999 }

The producer team deploys its code without notifying downstream teams.

Ten minutes later, downstream analytics services, billing systems, and warehouse consumers crash with NullPointerException errors because they expect user_email and total_amount.

In a decoupled event-driven architecture, producers and consumers deploy independently. Without a centralized schema contract enforcement engine, payload changes break downstream systems.

Confluent Schema Registry and Apache Avro solve data corruption across independent team deployments.

---

Why Plain JSON Fails at Enterprise Scale

JSON is human-readable, but creates three major production problems:

1. Verbose Payload Size: String field names ("user_email", "total_amount") repeat across every single message, wasting network bandwidth and disk storage.
2. Weak Type Enforcement: Numbers can arrive as strings ("49.99"), integers (4999), or floats (49.99), forcing consumer code to handle dynamic parsing edge cases.
3. Zero Backward-Compatibility Checks: Producers can remove mandatory fields or change field types without triggering build compiler errors.

---

Apache Avro: Binary Compact Serialization

Apache Avro is a binary serialization format that relies on a defined JSON schema definition.

{
  "type": "record",
  "name": "OrderPayment",
  "namespace": "com.company.events",
  "fields": [
    { "name": "order_id", "type": "long" },
    { "name": "user_email", "type": "string" },
    { "name": "total_amount", "type": "double" }
  ]
}

Because Avro messages do NOT include field names inside the binary payload, message payloads are extremely small:

JSON PAYLOAD (78 bytes):
{"order_id":101,"user_email":"[email protected]","total_amount":49.99}

AVRO BINARY PAYLOAD (21 bytes):
[ 0x65 ][ 0x10 ][ 0x61 0x6C 0x65 0x78 ... ][ 0x40 0x48 0xFE ... ]

Avro payload sizes are typically 70% to 80% smaller than JSON.

---

Confluent Schema Registry Architecture

To deserialize an Avro binary payload, a consumer must possess the exact Avro schema used by the producer when writing the message.

Shipping the full schema inside every message payload would ruin compression gains.

Instead, the Schema Registry stores schemas centrally and assigns an integer ID to each schema version.

PRODUCER CLIENT                                       SCHEMA REGISTRY
      |                                                      |
      |--- 1. Register Schema (OrderPayment.avsc) ----------->|
      |<-- 2. Return Schema ID: 42 --------------------------|
      |
      v
Build 5-Byte Wire Format Header:
[ Magic Byte 0x00 ][ 4-Byte Schema ID: 42 ][ Avro Binary Payload... ]
      |
      v
Publish to Kafka Topic

---

The 5-Byte Kafka Avro Wire Format Header

Every message payload serialized using Kafka’s Avro Serializer begins with a 5-byte header:

Byte 0:       Magic Byte (always 0x00)
Bytes 1 - 4:  Schema ID (4-byte Big-Endian Integer)
Bytes 5+:     Raw Avro Binary Payload Data

When a consumer reads a message:

1. It reads the 4-byte Schema ID (42) from the header.
2. It fetches Schema ID 42 from its local in-memory cache (or requests it from Schema Registry if missing).
3. It uses Schema 42 to deserialize the raw binary payload into a Java GenericRecord or compiled class object.

---

Schema Evolution Compatibility Rules

Schema Registry acts as a gatekeeper during producer.send(). If a developer attempts to register a breaking schema change, Schema Registry rejects the HTTP request and blocks the producer build.

Schema Registry supports multiple compatibility enforcement modes:

1. BACKWARD Compatibility (Default)

Consumers using the NEW schema can read messages written by producers using the OLD schema.

• Rule: You can delete fields or add optional fields (with default values).
• Upgrade Path: Upgrade consumers first, then upgrade producers.

2. FORWARD Compatibility

Consumers using the OLD schema can read messages written by producers using the NEW schema.

• Rule: You can add new fields or delete optional fields.
• Upgrade Path: Upgrade producers first, then upgrade consumers.

3. FULL Compatibility

Schemas are both BACKWARD and FORWARD compatible.

• Rule: You can only add or remove fields that have explicit default values defined.
• Upgrade Path: Upgrade producers or consumers in any arbitrary order.

---

Code Example: Using Avro with Kafka Producer

import com.company.events.OrderPayment; // Auto-generated from Avro schema file
import io.confluent.kafka.serializers.KafkaAvroSerializer;
import org.apache.kafka.clients.producer.*;

import java.util.Properties;

public class AvroProducerDemo {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka1:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class.getName());
        props.put("schema.registry.url", "http://schemaregistry:8081");

        Producer<String, OrderPayment> producer = new KafkaProducer<>(props);

        OrderPayment payment = OrderPayment.newBuilder()
                .setOrderId(101L)
                .setUserEmail("[email protected]")
                .setTotalAmount(49.99)
                .build();

        producer.send(new ProducerRecord<>("orders", "USER-101", payment), (metadata, exception) -> {
            if (exception == null) {
                System.out.println("Published Avro Record to Offset: " + metadata.offset());
            }
        });

        producer.close();
    }
}

---

Quick Summary

• Plain JSON payloads waste bandwidth and lack compile-time data contract enforcement.
• Apache Avro serializes data into compact binary payloads up to 80% smaller than JSON.
• Confluent Schema Registry manages schema versions centrally, prefixing payloads with a 5-byte header containing a 4-byte Schema ID.
• Compatibility modes (BACKWARD, FORWARD, FULL) prevent producers from deploying breaking payload changes that crash downstream consumers.

References & Further Reading

1. Bejeck, B. (2018). Kafka Streams in Action. Manning Publications.
2. Apache Software Foundation. Apache Kafka Documentation: Kafka Streams Architecture & RocksDB State Stores. Apache Kafka Docs.
3. Sax, M. J., et al. (2018). Topology Optimization in Kafka Streams. Confluent Blog.

---