# Kafka Data Streaming Platform

> Apache Kafka with Zookeeper, Schema Registry, Connect, and AKHQ management UI

- Category: Message Queues & Brokers
- Difficulty: advanced
- Components: kafka, zookeeper, schema-registry, kafka-connect, akhq
- Tags: kafka, streaming, data-pipeline, zookeeper, schema-registry, event-driven
- Source: https://docker.recipes/messaging/kafka-streaming-platform

## Overview

Apache Kafka is a distributed event streaming platform originally developed at LinkedIn and open-sourced in 2011, designed to handle high-throughput, fault-tolerant streaming of real-time data feeds. Built as a distributed commit log, Kafka excels at ingesting millions of events per second while providing durable storage and replay capabilities that traditional message queues cannot match. Unlike conventional messaging systems, Kafka treats data as an immutable sequence of events, making it ideal for event sourcing, real-time analytics, and building reactive architectures. This comprehensive streaming platform combines Kafka with Apache ZooKeeper for distributed coordination, Confluent Schema Registry for data governance, Kafka Connect for seamless data integration, and AKHQ as a web-based management interface. Together, these components create a complete event-driven architecture capable of connecting databases, applications, and microservices through standardized data pipelines. This stack is particularly valuable for organizations implementing event-driven microservices, real-time analytics platforms, or change data capture (CDC) workflows where data consistency, replay capability, and horizontal scalability are critical requirements.

## Features

- High-throughput event processing capable of handling millions of messages per second with low latency
- Durable message persistence with configurable retention policies and log compaction for event sourcing
- Consumer groups enabling parallel processing and automatic load balancing across multiple consumers
- Schema evolution and compatibility checking through Confluent Schema Registry with Avro support
- Pre-configured Kafka Connect workers for building data pipelines between Kafka and external systems
- AKHQ web interface providing topic management, consumer group monitoring, and message browsing
- Exactly-once delivery semantics ensuring data integrity in mission-critical applications
- Horizontal partitioning across brokers for linear scalability and fault tolerance

## Use cases

- Real-time data pipeline orchestration between microservices, databases, and analytics platforms
- Change Data Capture (CDC) implementation to stream database changes to downstream applications
- Event sourcing architectures where business events need to be stored and replayed chronologically
- Log aggregation from distributed applications and infrastructure components for centralized monitoring
- IoT data ingestion platforms handling sensor data streams from thousands of connected devices
- Financial trading systems requiring ultra-low latency order processing and market data distribution
- E-commerce platforms tracking user behavior, inventory changes, and order fulfillment events in real-time

## Prerequisites

- Minimum 6GB available RAM (1GB for Kafka, 512MB for ZooKeeper, remaining for other components)
- Docker Engine 20.10+ and Docker Compose V2 for proper container orchestration support
- Available ports 8080 (AKHQ), 8081 (Schema Registry), 8083 (Connect), and 9092 (Kafka broker)
- Basic understanding of distributed systems concepts and event-driven architecture patterns
- Sufficient disk space for persistent volumes as Kafka stores all messages until retention period expires
- Network configuration allowing inter-container communication on default Docker bridge network

## docker-compose.yml

```yaml
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    container_name: zookeeper
    restart: unless-stopped
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    volumes:
      - zookeeper_data:/var/lib/zookeeper/data
      - zookeeper_logs:/var/lib/zookeeper/log

  kafka:
    image: confluentinc/cp-kafka:latest
    container_name: kafka
    restart: unless-stopped
    ports:
      - "${KAFKA_PORT:-9092}:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:${KAFKA_PORT:-9092}
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    volumes:
      - kafka_data:/var/lib/kafka/data
    depends_on:
      - zookeeper

  schema-registry:
    image: confluentinc/cp-schema-registry:latest
    container_name: schema-registry
    restart: unless-stopped
    ports:
      - "${SCHEMA_PORT:-8081}:8081"
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: kafka:29092
    depends_on:
      - kafka

  kafka-connect:
    image: confluentinc/cp-kafka-connect:latest
    container_name: kafka-connect
    restart: unless-stopped
    ports:
      - "${CONNECT_PORT:-8083}:8083"
    environment:
      CONNECT_BOOTSTRAP_SERVERS: kafka:29092
      CONNECT_REST_ADVERTISED_HOST_NAME: kafka-connect
      CONNECT_GROUP_ID: compose-connect-group
      CONNECT_CONFIG_STORAGE_TOPIC: docker-connect-configs
      CONNECT_OFFSET_STORAGE_TOPIC: docker-connect-offsets
      CONNECT_STATUS_STORAGE_TOPIC: docker-connect-status
      CONNECT_KEY_CONVERTER: io.confluent.connect.avro.AvroConverter
      CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter
      CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL: http://schema-registry:8081
      CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: http://schema-registry:8081
      CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
    depends_on:
      - kafka
      - schema-registry

  akhq:
    image: tchiotludo/akhq:latest
    container_name: akhq
    restart: unless-stopped
    ports:
      - "${AKHQ_PORT:-8080}:8080"
    environment:
      AKHQ_CONFIGURATION: |
        akhq:
          connections:
            docker-kafka-server:
              properties:
                bootstrap.servers: "kafka:29092"
              schema-registry:
                url: "http://schema-registry:8081"
              connect:
                - name: "connect"
                  url: "http://kafka-connect:8083"
    depends_on:
      - kafka
      - schema-registry
      - kafka-connect

volumes:
  zookeeper_data:
  zookeeper_logs:
  kafka_data:
```

## .env

```bash
# Kafka Streaming Platform
KAFKA_PORT=9092
SCHEMA_PORT=8081
CONNECT_PORT=8083
AKHQ_PORT=8080
```

## Usage notes

- AKHQ management UI at http://localhost:8080
- Kafka accessible at localhost:9092
- Schema Registry at localhost:8081
- Kafka Connect REST API at localhost:8083
- Create topics via AKHQ or kafka-topics CLI
- Install connectors in kafka-connect for CDC, S3, etc.

## Troubleshooting

- Error 'Broker may not be available': Verify ZooKeeper is running and accessible before starting Kafka broker
- Schema Registry connection refused: Ensure Kafka broker is fully started before Schema Registry attempts connection
- AKHQ shows empty cluster: Check that AKHQ_CONFIGURATION environment variable is properly formatted YAML
- Kafka Connect worker fails to start: Verify internal topics (configs, offsets, status) are created with correct replication factor
- OutOfMemoryError in Kafka container: Increase Docker memory limits or add KAFKA_HEAP_OPTS environment variable
- Producer/Consumer timeout errors: Adjust KAFKA_ADVERTISED_LISTENERS to match your network configuration for external access

