Apache Kafka

advanced

Distributed event streaming platform.

Overview

Apache Kafka is a distributed event streaming platform designed for high-throughput data pipelines, real-time analytics, and event-driven architectures. Originally developed by LinkedIn and open-sourced in 2011, Kafka revolutionized how organizations handle streaming data by providing durable message storage, replay capabilities, and the ability to process millions of events per second. This stack combines Kafka with Apache ZooKeeper for distributed coordination and Kafka UI for visual management. ZooKeeper handles Kafka's metadata, cluster coordination, and leader election, while Kafka UI provides a web interface for monitoring topics, consumer groups, and message flows. Together, they create a complete event streaming platform that can handle everything from log aggregation to real-time analytics. This combination is ideal for developers building event-driven microservices, data engineers creating real-time pipelines, and organizations implementing change data capture or log aggregation systems. The inclusion of Kafka UI makes this stack particularly valuable for teams who need visual monitoring and management capabilities without relying solely on command-line tools.

Key Features

High-throughput message processing capable of millions of events per second
Durable message storage with configurable retention periods for event replay
Partitioned topics for horizontal scalability and parallel processing
Consumer groups enabling multiple consumers to process messages in parallel
ZooKeeper coordination for cluster management and metadata storage
Kafka UI web interface for topic management and consumer monitoring
Log compaction support for event sourcing patterns
Exactly-once delivery semantics for critical data processing

Common Use Cases

1Real-time data pipelines for streaming ETL and data integration
2Event-driven microservices architectures with reliable message delivery
3Log aggregation from multiple applications and services
4Change data capture (CDC) for database synchronization
5Real-time analytics and stream processing applications
6Activity tracking and user behavior analytics
7IoT data collection and processing from sensor networks

Prerequisites

Minimum 6GB RAM available (1GB ZooKeeper + 4GB Kafka + 1GB overhead)
Ports 2181, 8080, and 9092 available on the host system
Understanding of event streaming concepts and message brokers
Familiarity with topic partitioning and consumer group concepts
Basic knowledge of ZooKeeper's role in distributed systems
Docker and Docker Compose installed with sufficient disk space for message storage

For development & testing. Review security settings, change default credentials, and test thoroughly before production use. See Terms

docker-compose.yml

1services: 
2  zookeeper: 
3    image: confluentinc/cp-zookeeper:latest
4    container_name: zookeeper
5    environment: 
6      ZOOKEEPER_CLIENT_PORT: 2181
7    volumes: 
8      - zookeeper_data:/var/lib/zookeeper/data
9    networks: 
10      - kafka
11
12  kafka: 
13    image: confluentinc/cp-kafka:latest
14    container_name: kafka
15    environment: 
16      KAFKA_BROKER_ID: 1
17      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
18      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
19      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
20    volumes: 
21      - kafka_data:/var/lib/kafka/data
22    ports: 
23      - "9092:9092"
24    depends_on: 
25      - zookeeper
26    networks: 
27      - kafka
28
29  kafka-ui: 
30    image: provectuslabs/kafka-ui:latest
31    container_name: kafka-ui
32    environment: 
33      KAFKA_CLUSTERS_0_NAME: local
34      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:9092
35    ports: 
36      - "8080:8080"
37    depends_on: 
38      - kafka
39    networks: 
40      - kafka
41
42volumes: 
43  zookeeper_data: 
44  kafka_data: 
45
46networks: 
47  kafka: 
48    driver: bridge

.env Template

.env

1# No additional config needed

Usage Notes

1Docs: https://kafka.apache.org/documentation/
2Kafka UI at http://localhost:8080, broker on port 9092
3Create topics: docker exec kafka kafka-topics --create --topic test --bootstrap-server localhost:9092
4List topics: docker exec kafka kafka-topics --list --bootstrap-server localhost:9092
5Use Confluent Platform images for production features
6Consider KRaft mode (kafka-kraft recipe) to eliminate Zookeeper

Individual Services(3 services)

Copy individual services to mix and match with your existing compose files.

zookeeper

zookeeper:
  image: confluentinc/cp-zookeeper:latest
  container_name: zookeeper
  environment:
    ZOOKEEPER_CLIENT_PORT: 2181
  volumes:
    - zookeeper_data:/var/lib/zookeeper/data
  networks:
    - kafka

kafka

kafka:
  image: confluentinc/cp-kafka:latest
  container_name: kafka
  environment:
    KAFKA_BROKER_ID: 1
    KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
    KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
  volumes:
    - kafka_data:/var/lib/kafka/data
  ports:
    - "9092:9092"
  depends_on:
    - zookeeper
  networks:
    - kafka

kafka-ui

kafka-ui:
  image: provectuslabs/kafka-ui:latest
  container_name: kafka-ui
  environment:
    KAFKA_CLUSTERS_0_NAME: local
    KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:9092
  ports:
    - "8080:8080"
  depends_on:
    - kafka
  networks:
    - kafka

Quick Start

terminal

1# 1. Create the compose file
2cat > docker-compose.yml << 'EOF'
3services:
4  zookeeper:
5    image: confluentinc/cp-zookeeper:latest
6    container_name: zookeeper
7    environment:
8      ZOOKEEPER_CLIENT_PORT: 2181
9    volumes:
10      - zookeeper_data:/var/lib/zookeeper/data
11    networks:
12      - kafka
13
14  kafka:
15    image: confluentinc/cp-kafka:latest
16    container_name: kafka
17    environment:
18      KAFKA_BROKER_ID: 1
19      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
20      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
21      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
22    volumes:
23      - kafka_data:/var/lib/kafka/data
24    ports:
25      - "9092:9092"
26    depends_on:
27      - zookeeper
28    networks:
29      - kafka
30
31  kafka-ui:
32    image: provectuslabs/kafka-ui:latest
33    container_name: kafka-ui
34    environment:
35      KAFKA_CLUSTERS_0_NAME: local
36      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:9092
37    ports:
38      - "8080:8080"
39    depends_on:
40      - kafka
41    networks:
42      - kafka
43
44volumes:
45  zookeeper_data:
46  kafka_data:
47
48networks:
49  kafka:
50    driver: bridge
51EOF
52
53# 2. Create the .env file
54cat > .env << 'EOF'
55# No additional config needed
56EOF
57
58# 3. Start the services
59docker compose up -d
60
61# 4. View logs
62docker compose logs -f

One-Liner

Run this command to download and set up the recipe in one step:

terminal

1curl -fsSL https://docker.recipes/api/recipes/kafka/run | bash

Troubleshooting

Kafka fails to start with 'Connection to node -1 could not be established': Ensure ZooKeeper is running and KAFKA_ZOOKEEPER_CONNECT points to correct address
Topics not visible in Kafka UI: Check KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS environment variable matches Kafka service name and port
OutOfMemoryError in Kafka container: Increase Docker memory limits or add KAFKA_HEAP_OPTS environment variable with appropriate heap size
Messages not being consumed: Verify consumer group configuration and check if consumers are subscribed to correct topic partitions
ZooKeeper connection timeout errors: Increase ZOOKEEPER_CLIENT_PORT_TIMEOUT and ensure sufficient resources allocated to ZooKeeper container
Kafka UI shows 'Cluster is not available': Verify network connectivity between kafka-ui and kafka containers using correct service names

Community Notes

Loading notes...

Download Recipe Kit

Get all files in a ready-to-deploy package

Includes docker-compose.yml, .env template, README, and license

Components

kafkazookeeper

Related Recipes

Ad Space

Shortcuts: C CopyF FavoriteD Download