# Kafka + Zookeeper + Kafka UI

> Distributed streaming platform with web UI.

- Category: Message Queues & Brokers
- Difficulty: advanced
- Components: kafka, zookeeper, kafka-ui, schema-registry
- Tags: kafka, streaming, event-streaming, zookeeper, message-queue
- Source: https://docker.recipes/messaging/kafka-complete

## Overview

Apache Kafka is a distributed event streaming platform originally developed by LinkedIn and later open-sourced, designed to handle real-time data feeds with high throughput and fault tolerance. It serves as the backbone for event-driven architectures, enabling applications to produce, consume, and process streams of records in real-time across distributed systems. Kafka's unique approach to messaging through persistent, partitioned logs makes it ideal for scenarios requiring message replay, event sourcing, and horizontal scalability.

This stack combines Kafka with Apache ZooKeeper for distributed coordination, Confluent Schema Registry for managing Avro schemas and data evolution, and Kafka UI for visual management and monitoring. ZooKeeper handles cluster metadata, leader election, and configuration management for Kafka brokers, while Schema Registry ensures data compatibility across producers and consumers by managing schema versions. Kafka UI provides a web-based interface for monitoring topics, consumer groups, partitions, and message flows without requiring command-line tools.

This configuration is ideal for developers building event-driven applications, data engineers implementing streaming pipelines, and DevOps teams needing a complete Kafka development environment. The inclusion of Schema Registry makes it particularly valuable for organizations requiring strict data governance and schema evolution, while Kafka UI reduces the operational complexity of managing topics and monitoring cluster health. This stack provides the foundation for implementing microservices communication, real-time analytics, and data integration patterns.

## Features

- High-throughput message processing capable of handling millions of events per second
- Durable message storage with configurable retention policies and log compaction
- Consumer groups enabling parallel processing and automatic partition rebalancing
- Exactly-once delivery semantics for critical data processing workflows
- Avro schema management with backward and forward compatibility checking
- Web-based cluster monitoring with topic, partition, and consumer group visibility
- Event sourcing capabilities through persistent, immutable message logs
- Multi-partition topic support for horizontal scaling and load distribution

## Use cases

- Real-time data pipeline development for streaming analytics and ETL processes
- Event-driven microservices architecture with reliable inter-service communication
- Change data capture (CDC) for synchronizing databases and maintaining data consistency
- Log aggregation from distributed applications and infrastructure components
- IoT data ingestion and processing for sensor networks and telemetry systems
- Financial transaction processing requiring audit trails and event replay capabilities
- E-commerce activity tracking for recommendation engines and user behavior analysis

## Prerequisites

- Minimum 6GB RAM available (1GB ZooKeeper + 4GB Kafka + 1GB other components)
- Docker Engine 20.10+ and Docker Compose V2 for proper service orchestration
- Ports 8080, 8081, and 29092 available on the host system
- Basic understanding of event streaming concepts and message queue patterns
- Familiarity with Avro schema design for Schema Registry integration
- SSD storage recommended for optimal Kafka log segment performance

## docker-compose.yml

```yaml
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
      - ZOOKEEPER_CLIENT_PORT=2181
      - ZOOKEEPER_TICK_TIME=2000
    volumes:
      - zookeeper-data:/var/lib/zookeeper/data
      - zookeeper-log:/var/lib/zookeeper/log
    networks:
      - kafka-network
    restart: unless-stopped

  kafka:
    image: confluentinc/cp-kafka:latest
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      - KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT
      - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
    volumes:
      - kafka-data:/var/lib/kafka/data
    ports:
      - "29092:29092"
    depends_on:
      - zookeeper
    networks:
      - kafka-network
    restart: unless-stopped

  schema-registry:
    image: confluentinc/cp-schema-registry:latest
    environment:
      - SCHEMA_REGISTRY_HOST_NAME=schema-registry
      - SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS=kafka:9092
    ports:
      - "8081:8081"
    depends_on:
      - kafka
    networks:
      - kafka-network
    restart: unless-stopped

  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    environment:
      - KAFKA_CLUSTERS_0_NAME=local
      - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka:9092
      - KAFKA_CLUSTERS_0_SCHEMAREGISTRY=http://schema-registry:8081
    ports:
      - "8080:8080"
    depends_on:
      - kafka
      - schema-registry
    networks:
      - kafka-network
    restart: unless-stopped

volumes:
  zookeeper-data:
  zookeeper-log:
  kafka-data:

networks:
  kafka-network:
    driver: bridge
```

## .env

```bash
# Kafka
# Kafka at localhost:29092 (external)
# Kafka UI at http://localhost:8080
# Schema Registry at http://localhost:8081
```

## Usage notes

- Kafka broker at localhost:29092
- Kafka UI at http://localhost:8080
- Schema Registry at http://localhost:8081
- Topic management via UI
- Avro schema support

## Troubleshooting

- Error 'Connection to node -1 could not be established': Verify KAFKA_ADVERTISED_LISTENERS includes correct hostname and port mapping
- ZooKeeper connection timeouts: Increase ZOOKEEPER_TICK_TIME to 4000 for slower systems or networks
- Schema Registry 'Subject not found' errors: Ensure topics exist before registering schemas and verify KAFKASTORE_BOOTSTRAP_SERVERS connectivity
- Kafka UI showing 'Cluster unavailable': Check that kafka service is fully started before kafka-ui by adding health checks or delays
- High memory usage warnings: Adjust JVM heap settings using KAFKA_HEAP_OPTS environment variable
- Topic creation failures: Verify KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR matches available broker count in cluster

