Introduction to Apache Kafka

Apache Kafka is an open-source distributed event streaming platform built for high-throughput, fault-tolerant, real-time data pipelines and event-driven applications. Originally created at LinkedIn in 2010 and open-sourced in 2011, Kafka is now an Apache top-level project powering data infrastructure at thousands of companies worldwide.

What problems does Kafka solve?

Modern architectures involve many services that need to share data in real time — user activity, transactions, logs, metrics, IoT sensor readings. Without a dedicated streaming layer, teams typically wire services directly together, creating a fragile web of point-to-point connections that grows as O(n²).

Kafka acts as a central nervous system: producers write events once, and any number of consumers read them independently, at their own pace, and even replay past events.

Core concepts at a glance

Every Kafka deployment revolves around five key abstractions:

Common use cases

• Event-driven microservices — services communicate via Kafka topics instead of direct HTTP calls, enabling loose coupling and independent deployment.
• Real-time analytics — clickstream data, purchase events, or sensor readings flow into Kafka and are aggregated by Kafka Streams or Apache Flink in seconds.
• Log aggregation — application logs from hundreds of instances are centralized in Kafka and forwarded to Elasticsearch, S3, or Splunk via Kafka Connect.
• Event sourcing — Kafka's durable, replayable log serves as the source of truth for state reconstruction.
• Change Data Capture (CDC) — database row changes (via Debezium) are streamed into Kafka, keeping downstream systems in sync in real time.
• Metrics and monitoring — Kafka aggregates metrics from distributed systems at high volume before routing them to time-series databases like InfluxDB or Prometheus.

Kafka vs traditional message brokers

Kafka differs from classic brokers like RabbitMQ or ActiveMQ in several fundamental ways:

Your first Kafka setup

The quickest way to run Kafka locally is with Docker Compose. The example below uses KRaft mode — Kafka without ZooKeeper, available and recommended since Kafka 3.3.

# docker-compose.yml
services:
  kafka:
    image: apache/kafka:3.7.0
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"

# Start the cluster
docker compose up -d

# Create a topic
docker exec kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --create --topic my-first-topic --partitions 3 --replication-factor 1

# List topics
docker exec kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 --list

# Produce some events
docker exec -it kafka /opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server localhost:9092 --topic my-first-topic

# In a second terminal, consume events
docker exec -it kafka /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 --topic my-first-topic --from-beginning

How Kafka stores data

Each topic partition is stored as an append-only log on the broker's disk. Events are never modified or deleted immediately — they are retained for a configurable period (default 7 days) or until a size limit is reached. This log-structured approach is what gives Kafka its high write throughput and replay capability.

# Inspect the partition log on disk (inside the broker container)
ls /var/kafka-logs/my-first-topic-0/
# 00000000000000000000.log   ← event data
# 00000000000000000000.index ← byte offset index
# 00000000000000000000.timeindex ← timestamp index

How this tutorial is organized

1. Introduction (this page) — core concepts and first steps.
2. Architecture — brokers, partitions, replication, KRaft internals.
3. Producers — sending events reliably with acks, batching, and compression.
4. Consumers & Consumer Groups — reading events, offset management, and rebalancing.
5. Kafka Streams — stream processing with KStream, KTable, and windowing.
6. Kafka Connect — source and sink connectors for zero-code integration.
7. Schema Registry & Avro — enforcing schemas and evolving them safely.
8. CLI Cheat Sheet — all the commands you'll reach for daily.