Kafka (KRaft Mode)

intermediate

Apache Kafka without Zookeeper dependency.

Overview

Apache Kafka is a distributed event streaming platform designed for high-throughput data pipelines and real-time analytics. Originally developed by LinkedIn and open-sourced in 2011, Kafka has become the backbone of modern event-driven architectures, capable of handling millions of messages per second while providing durability, fault tolerance, and exactly-once processing guarantees. The platform excels at decoupling data producers from consumers, enabling scalable microservices architectures and real-time data processing workflows. This deployment runs Kafka in KRaft mode (Kafka Raft), which eliminates the traditional dependency on Apache Zookeeper by using Kafka's own consensus protocol for cluster coordination. The single kafka container operates as both a broker and controller, handling message storage and cluster metadata management internally. This architecture significantly simplifies operations, reduces infrastructure complexity, and provides faster failover times compared to Zookeeper-based deployments. This configuration is ideal for developers building event-driven applications, data engineers creating streaming pipelines, and organizations transitioning from legacy messaging systems to modern event streaming platforms. The KRaft mode deployment is production-ready as of Kafka 3.3+ and offers improved operational efficiency, making it perfect for both development environments and production workloads that don't require multi-broker clustering.

Key Features

KRaft consensus protocol eliminates Zookeeper dependency for simplified cluster management
High-throughput message processing capable of handling millions of events per second
Durable message storage with configurable retention policies and log compaction
Consumer groups enabling parallel processing and automatic load balancing
Exactly-once processing semantics for mission-critical data integrity
Built-in partitioning for horizontal scalability and parallel consumption
Topic-based pub/sub messaging with support for complex routing patterns
Native support for event sourcing patterns with message replay capabilities

Common Use Cases

1Real-time data pipeline orchestration for ETL and stream processing workflows
2Event-driven microservices architecture with reliable inter-service communication
3Log aggregation and centralized logging infrastructure for distributed systems
4Change data capture (CDC) for database synchronization and data replication
5IoT device telemetry collection and real-time sensor data processing
6Financial transaction processing with audit trails and fraud detection
7Website activity tracking and real-time analytics for user behavior analysis

Prerequisites

Docker Engine 20.10+ with Docker Compose v2 support
Minimum 2GB RAM available (4GB+ recommended for production workloads)
Port 9092 available for Kafka broker connections
Basic understanding of pub/sub messaging patterns and event streaming concepts
Familiarity with topic-partition concepts for effective message routing
Java 11+ knowledge helpful for custom client development and debugging

For development & testing. Review security settings, change default credentials, and test thoroughly before production use. See Terms

docker-compose.yml

1services: 
2  kafka: 
3    image: bitnami/kafka:latest
4    container_name: kafka-kraft
5    restart: unless-stopped
6    environment: 
7      KAFKA_CFG_NODE_ID: 1
8      KAFKA_CFG_PROCESS_ROLES: broker,controller
9      KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
10      KAFKA_CFG_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
11      KAFKA_CFG_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
12      KAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLER
13      KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
14    volumes: 
15      - kafka_data:/bitnami/kafka
16    ports: 
17      - "9092:9092"
18
19volumes: 
20  kafka_data:

.env Template

.env

1# No Zookeeper needed

Usage Notes

1Docs: https://kafka.apache.org/documentation/#kraft
2Kafka broker on port 9092, KRaft controller on 9093
3No Zookeeper required - single container deployment
4Create topics: docker exec kafka-kraft kafka-topics.sh --create --topic test --bootstrap-server localhost:9092
5KRaft is ready since Kafka 3.3+
6Simpler operations and faster failover than Zookeeper mode

Quick Start

terminal

1# 1. Create the compose file
2cat > docker-compose.yml << 'EOF'
3services:
4  kafka:
5    image: bitnami/kafka:latest
6    container_name: kafka-kraft
7    restart: unless-stopped
8    environment:
9      KAFKA_CFG_NODE_ID: 1
10      KAFKA_CFG_PROCESS_ROLES: broker,controller
11      KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
12      KAFKA_CFG_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
13      KAFKA_CFG_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
14      KAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLER
15      KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
16    volumes:
17      - kafka_data:/bitnami/kafka
18    ports:
19      - "9092:9092"
20
21volumes:
22  kafka_data:
23EOF
24
25# 2. Create the .env file
26cat > .env << 'EOF'
27# No Zookeeper needed
28EOF
29
30# 3. Start the services
31docker compose up -d
32
33# 4. View logs
34docker compose logs -f

One-Liner

Run this command to download and set up the recipe in one step:

terminal

1curl -fsSL https://docker.recipes/api/recipes/kafka-kraft/run | bash

Troubleshooting

Connection refused on port 9092: Ensure KAFKA_CFG_ADVERTISED_LISTENERS matches your client connection hostname
Topics not created or visible: Verify bootstrap server configuration and use kafka-topics.sh with correct broker address
High memory usage or OutOfMemoryError: Increase container memory limits and tune JVM heap settings via KAFKA_OPTS
Message delivery failures: Check topic partition count and replication factor settings for your use case
Slow startup or controller election timeouts: Ensure sufficient disk I/O performance and consider adjusting controller.quorum.election.timeout.ms
Client authentication errors: Verify KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP configuration matches client security settings

Community Notes

Loading notes...

Download Recipe Kit

Get all files in a ready-to-deploy package

Includes docker-compose.yml, .env template, README, and license

Components

kafka

Related Recipes

Ad Space

Shortcuts: C CopyF FavoriteD Download