Kafka Connect vs Kafka Streams: Zero-Code ELT Pipelines vs Real-Time Analytics

Suppose you are the lead data architect for an e-commerce platform.

Your leadership team gives you two integration projects:

1. Project A (Database Integration): Capture every row update from a production PostgreSQL database and stream it continuously into an Elasticsearch search cluster and an AWS S3 data lake.
2. Project B (Real-Time Stream Analytics): Consume a stream of raw clickstream events, join them with user profile data, compute a running 5-minute sliding count of page views per user, and output fraudulent activity alerts.

If you write custom KafkaProducer and KafkaConsumer Java boilerplate code for Project A, you waste months writing database connection pool handlers, error retry loops, and offset tracking logic.

If you attempt Project B using plain consumer loops, managing in-memory aggregation state across server restarts becomes a complex engineering challenge.

Kafka provides two specialized frameworks to solve these distinct problems: Kafka Connect (for Project A) and Kafka Streams (for Project B).

---

When to Use Which Framework

+---------------------------------------------------------------------------------+
|                       KAFKA ECOSYSTEM SELECTION MATRIX                          |
|                                                                                 |
|  DATA MOVEMENT & INTEGRATION                 REAL-TIME COMPUTATION & TRANSFORMS |
|  (No custom code required)                   (Java / Scala event stream code)   |
|                                                                                 |
|  +---------------------------+               +-------------------------------+  |
|  |     KAFKA CONNECT         |               |        KAFKA STREAMS          |  |
|  | Databases -> Kafka -> S3  |               | Aggregations, Joins, Windows  |  |
|  +---------------------------+               +-------------------------------+  |
+---------------------------------------------------------------------------------+

---

Part A: Kafka Connect — Zero-Code Data Pipelines

Kafka Connect is a scalable, fault-tolerant framework for streaming data between Kafka and external storage systems (RDBMS databases, Elasticsearch, S3, Snowflake, Mongo).

Key Characteristics:

• Declarative Configuration: You do not write Java code. You submit a JSON/YAML configuration file via REST API to a Kafka Connect cluster.
• Source vs Sink Connectors:
  • Source Connector: Pulls data from external systems (e.g. Debezium PostgreSQL CDC) and writes records into Kafka topics.
  • Sink Connector: Reads records from Kafka topics and writes data into external systems (e.g. Elasticsearch Sink).
• Distributed Worker Cluster: Connect runs as an independent cluster of worker nodes (connect-distributed) that automatically balances task execution and offset tracking.

// Example: Registering an Elasticsearch Sink Connector via REST API
{
  "name": "elasticsearch-sink",
  "config": {
    "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
    "tasks.max": "3",
    "topics": "orders",
    "key.ignore": "false",
    "connection.url": "http://elasticsearch:9200",
    "type.name": "_doc"
  }
}

---

Part B: Kafka Streams — In-Process Stream Processing

Kafka Streams is a client library for building real-time stream processing applications in Java or Scala.

Key Characteristics:

• No Separate Cluster Required: Kafka Streams runs inside your standard Java application process (Spring Boot, Quarkus, plain Java main()). It does not require deploying a separate processing cluster like Apache Flink or Spark.
• Fault-Tolerant State Stores (RocksDB): Computations that aggregate data over time (e.g. count(), reduce(), windowed joins) store local state in an embedded RocksDB key-value database on local disk.
• Changelog Topics: Every local state update is backed by a replicated Kafka Changelog Topic. If your application pod dies, a new pod reads the changelog topic to restore its embedded RocksDB state automatically.

---

The Core Abstraction: KStream vs KTable Duality

Kafka Streams unifies event streams and database tables through Stream-Table Duality:

KSTREAM (Stream of Events / Facts)
Offset 0: [ Key: "UserA", Value: "Login" ]
Offset 1: [ Key: "UserA", Value: "PageViews" ]
Offset 2: [ Key: "UserA", Value: "Logout" ]

KTABLE (Current State / Change Log Snapshot)
Key: "UserA" ===> Value: "Logout" (Latest state ONLY!)

• KStream: Represents an append-only stream of independent immutable events (e.g. every click, every purchase).
• KTable: Represents the current latest state for each key, behaving like a database table updated via primary key.

---

Code Example: Real-Time Word Count in Kafka Streams

import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.streams.*;
import org.apache.kafka.streams.kstream.*;

import java.util.Arrays;
import java.util.Properties;

public class StreamsDemo {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, "wordcount-app");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka1:9092");
        props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
        props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());

        StreamsBuilder builder = new StreamsBuilder();

        // 1. Read input stream from "text-lines" topic
        KStream<String, String> textLines = builder.stream("text-lines");

        // 2. Transform: Tokenize sentences -> Group by word -> Count in 5-min windows
        KTable<String, Long> wordCounts = textLines
                .flatMapValues(text -> Arrays.asList(text.toLowerCase().split("\\W+")))
                .groupBy((key, word) -> word)
                .count(Materialized.as("counts-store")); // Embedded RocksDB state store!

        // 3. Write results out to "word-counts" topic
        wordCounts.toStream().to("word-counts", Produced.with(Serdes.String(), Serdes.Long()));

        KafkaStreams streams = new KafkaStreams(builder.build(), props);
        streams.start();
    }
}

---

Summary Comparison

Feature	Kafka Connect	Kafka Streams
Primary Goal	Move data between Kafka & external systems	Process, aggregate, & join event streams
Code Requirement	Zero code (JSON/YAML declarations)	Custom Java / Scala code
Runtime Model	Separate Kafka Connect cluster	Embedded inside user Java application
State Storage	None (delegates to target systems)	Embedded RocksDB + Kafka Changelog Topics
Ideal For	Database CDC, S3 archival, Search indexing	Fraud detection, Session windows, Real-time metrics

References & Further Reading

1. Apache Kafka Wiki. KIP-382: MirrorMaker 2.0 Specification (MM2). Kafka Improvement Proposals.
2. Confluent Inc. Multi-Region Cluster Topologies & Disaster Recovery. Confluent Docs.
3. Kleppmann, M. (2017). Designing Data-Intensive Applications (Chapter 5: Replication). O’Reilly Media.

---