# Apache Kafka Full Stack

> Complete Kafka setup with Zookeeper, Schema Registry, Connect, and KSQL.

- Category: Message Queues & Brokers
- Difficulty: advanced
- Components: zookeeper, kafka, schema-registry, kafka-connect, ksqldb
- Tags: kafka, streaming, event-driven, zookeeper, ksql
- Source: https://docker.recipes/messaging/apache-kafka-full-stack

## Overview

Apache ZooKeeper serves as the distributed coordination backbone for Apache Kafka, providing essential services like configuration management, leader election, and cluster membership through its hierarchical namespace and strong consistency guarantees. Originally developed at Yahoo for managing distributed systems, ZooKeeper has become the de facto standard for coordinating Kafka brokers, tracking partition assignments, and managing consumer group offsets. This comprehensive Kafka ecosystem combines ZooKeeper's coordination services with Kafka's high-throughput streaming platform, Confluent Schema Registry for Avro schema evolution, and ksqlDB for real-time stream processing and analytics. The stack creates a complete event streaming architecture where ZooKeeper maintains cluster state, Kafka handles message durability and partitioning, Schema Registry ensures data compatibility across services, and ksqlDB enables SQL-based stream transformations. Organizations building event-driven architectures, real-time analytics platforms, or microservices requiring reliable message ordering will find this stack invaluable for handling millions of events per second while maintaining exactly-once processing semantics. The combination particularly excels in scenarios requiring both high-throughput data ingestion and complex stream processing, making it ideal for financial trading systems, IoT data pipelines, and large-scale log aggregation where data consistency and replay capabilities are critical.

## Features

- Hierarchical namespace in ZooKeeper for organized configuration and service coordination
- High-throughput message streaming with Kafka's distributed log architecture supporting millions of events per second
- Avro schema evolution and compatibility checking through Confluent Schema Registry
- SQL-based stream processing with ksqlDB for real-time analytics and transformations
- Consumer group coordination enabling parallel processing across multiple application instances
- Exactly-once processing semantics for mission-critical data pipelines
- Log compaction and retention policies for efficient storage and event sourcing patterns
- Distributed partitioning strategy for horizontal scaling and fault tolerance

## Use cases

- Financial trading platforms requiring microsecond message ordering and replay capabilities
- IoT data ingestion pipelines processing sensor data from thousands of connected devices
- E-commerce platforms implementing event sourcing for order management and inventory tracking
- Real-time fraud detection systems analyzing transaction streams with complex event patterns
- Microservices architectures using event-driven communication for service decoupling
- Log aggregation and monitoring systems collecting application logs from distributed services
- Change data capture (CDC) implementations for database synchronization and data lake population

## Prerequisites

- Minimum 6GB RAM available (1GB for Kafka, 1GB for ZooKeeper, 2GB for ksqlDB, plus overhead)
- Docker Engine 20.10+ and Docker Compose v2 for proper networking and volume management
- Available ports 9092 (Kafka), 8081 (Schema Registry), and 8088 (ksqlDB) on the host system
- Basic understanding of Apache Kafka concepts including topics, partitions, and consumer groups
- Knowledge of Avro schema design principles for effective Schema Registry utilization
- Familiarity with SQL syntax for ksqlDB stream processing queries and transformations

## docker-compose.yml

```yaml
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    container_name: zookeeper
    restart: unless-stopped
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    volumes:
      - zk_data:/var/lib/zookeeper/data
      - zk_logs:/var/lib/zookeeper/log
    networks:
      - kafka-network

  kafka:
    image: confluentinc/cp-kafka:latest
    container_name: kafka
    restart: unless-stopped
    depends_on:
      - zookeeper
    ports:
      - "${KAFKA_PORT:-9092}:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    volumes:
      - kafka_data:/var/lib/kafka/data
    networks:
      - kafka-network

  schema-registry:
    image: confluentinc/cp-schema-registry:latest
    container_name: schema-registry
    restart: unless-stopped
    depends_on:
      - kafka
    ports:
      - "${SCHEMA_REGISTRY_PORT:-8081}:8081"
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: kafka:29092
    networks:
      - kafka-network

  ksqldb:
    image: confluentinc/ksqldb-server:latest
    container_name: ksqldb
    restart: unless-stopped
    depends_on:
      - kafka
      - schema-registry
    ports:
      - "${KSQL_PORT:-8088}:8088"
    environment:
      KSQL_BOOTSTRAP_SERVERS: kafka:29092
      KSQL_LISTENERS: http://0.0.0.0:8088
      KSQL_KSQL_SCHEMA_REGISTRY_URL: http://schema-registry:8081
    networks:
      - kafka-network

volumes:
  zk_data:
  zk_logs:
  kafka_data:

networks:
  kafka-network:
    driver: bridge
```

## .env

```bash
# Kafka Full Stack
KAFKA_PORT=9092
SCHEMA_REGISTRY_PORT=8081
KSQL_PORT=8088
```

## Usage notes

- Kafka broker at localhost:9092
- Schema Registry at http://localhost:8081
- KSQL at http://localhost:8088
- Use kafka-topics CLI to manage topics

## Troubleshooting

- ZooKeeper connection errors 'java.net.ConnectException': Ensure ZooKeeper container is fully started before Kafka attempts connection, check container logs for port binding issues
- Kafka 'Not enough replicas' warnings: Set KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR to 1 for single-broker development setups, increase for production clusters
- Schema Registry 'Schema not found' errors: Verify schema compatibility settings and ensure subjects are properly registered before producing/consuming with schemas
- ksqlDB 'Topic does not exist' failures: Create Kafka topics explicitly using kafka-topics CLI before referencing them in ksqlDB streams or tables
- OutOfMemory errors in Kafka: Increase container memory limits and tune JVM heap settings using KAFKA_HEAP_OPTS environment variable
- Consumer lag issues: Monitor partition distribution and increase consumer instances within groups, check for hot partitions causing processing bottlenecks

