# Kafka (KRaft Mode)

> Apache Kafka without Zookeeper dependency.

- Category: Message Queues & Brokers
- Difficulty: intermediate
- Components: kafka
- Tags: kafka, kraft, streaming, no-zookeeper
- Source: https://docker.recipes/messaging/kafka-kraft

## Overview

Apache Kafka is a distributed event streaming platform designed for high-throughput data pipelines and real-time analytics. Originally developed by LinkedIn and open-sourced in 2011, Kafka has become the backbone of modern event-driven architectures, capable of handling millions of messages per second while providing durability, fault tolerance, and exactly-once processing guarantees. The platform excels at decoupling data producers from consumers, enabling scalable microservices architectures and real-time data processing workflows.

This deployment runs Kafka in KRaft mode (Kafka Raft), which eliminates the traditional dependency on Apache Zookeeper by using Kafka's own consensus protocol for cluster coordination. The single kafka container operates as both a broker and controller, handling message storage and cluster metadata management internally. This architecture significantly simplifies operations, reduces infrastructure complexity, and provides faster failover times compared to Zookeeper-based deployments.

This configuration is ideal for developers building event-driven applications, data engineers creating streaming pipelines, and organizations transitioning from legacy messaging systems to modern event streaming platforms. The KRaft mode deployment is production-ready as of Kafka 3.3+ and offers improved operational efficiency, making it perfect for both development environments and production workloads that don't require multi-broker clustering.

## Features

- KRaft consensus protocol eliminates Zookeeper dependency for simplified cluster management
- High-throughput message processing capable of handling millions of events per second
- Durable message storage with configurable retention policies and log compaction
- Consumer groups enabling parallel processing and automatic load balancing
- Exactly-once processing semantics for mission-critical data integrity
- Built-in partitioning for horizontal scalability and parallel consumption
- Topic-based pub/sub messaging with support for complex routing patterns
- Native support for event sourcing patterns with message replay capabilities

## Use cases

- Real-time data pipeline orchestration for ETL and stream processing workflows
- Event-driven microservices architecture with reliable inter-service communication
- Log aggregation and centralized logging infrastructure for distributed systems
- Change data capture (CDC) for database synchronization and data replication
- IoT device telemetry collection and real-time sensor data processing
- Financial transaction processing with audit trails and fraud detection
- Website activity tracking and real-time analytics for user behavior analysis

## Prerequisites

- Docker Engine 20.10+ with Docker Compose v2 support
- Minimum 2GB RAM available (4GB+ recommended for production workloads)
- Port 9092 available for Kafka broker connections
- Basic understanding of pub/sub messaging patterns and event streaming concepts
- Familiarity with topic-partition concepts for effective message routing
- Java 11+ knowledge helpful for custom client development and debugging

## docker-compose.yml

```yaml
services:
  kafka:
    image: bitnami/kafka:latest
    container_name: kafka-kraft
    restart: unless-stopped
    environment:
      KAFKA_CFG_NODE_ID: 1
      KAFKA_CFG_PROCESS_ROLES: broker,controller
      KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_CFG_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
      KAFKA_CFG_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
    volumes:
      - kafka_data:/bitnami/kafka
    ports:
      - "9092:9092"

volumes:
  kafka_data:
```

## .env

```bash
# No Zookeeper needed
```

## Usage notes

- Docs: https://kafka.apache.org/documentation/#kraft
- Kafka broker on port 9092, KRaft controller on 9093
- No Zookeeper required - single container deployment
- Create topics: docker exec kafka-kraft kafka-topics.sh --create --topic test --bootstrap-server localhost:9092
- KRaft is ready since Kafka 3.3+
- Simpler operations and faster failover than Zookeeper mode

## Troubleshooting

- Connection refused on port 9092: Ensure KAFKA_CFG_ADVERTISED_LISTENERS matches your client connection hostname
- Topics not created or visible: Verify bootstrap server configuration and use kafka-topics.sh with correct broker address
- High memory usage or OutOfMemoryError: Increase container memory limits and tune JVM heap settings via KAFKA_OPTS
- Message delivery failures: Check topic partition count and replication factor settings for your use case
- Slow startup or controller election timeouts: Ensure sufficient disk I/O performance and consider adjusting controller.quorum.election.timeout.ms
- Client authentication errors: Verify KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP configuration matches client security settings

