# Apache Pulsar Cluster

> Cloud-native distributed messaging and streaming platform with multi-tenancy.

- Category: Message Queues & Brokers
- Difficulty: advanced
- Components: pulsar-broker, pulsar-bookie, zookeeper, pulsar-manager
- Tags: pulsar, messaging, streaming, multi-tenant, cloud-native
- Source: https://docker.recipes/messaging/pulsar-cluster

## Overview

Apache Pulsar is a cloud-native, distributed messaging and streaming platform originally developed by Yahoo and later open-sourced through the Apache Software Foundation. It provides unified queuing and streaming capabilities with multi-tenancy, geo-replication, and tiered storage support, making it a powerful alternative to traditional messaging systems like Apache Kafka and RabbitMQ. Pulsar's architecture separates serving from storage, enabling independent scaling and better resource utilization.

This deployment creates a complete Pulsar cluster with six interconnected services: a ZooKeeper instance for coordination, an initialization container for cluster metadata setup, two BookKeeper bookies (bookie1 and bookie2) for persistent message storage, a Pulsar broker for message routing and client connections, and Pulsar Manager for web-based administration. The setup demonstrates Pulsar's distributed architecture with proper separation of concerns between metadata management, storage, and serving layers.

This configuration is ideal for organizations evaluating Pulsar's capabilities, development teams building event-driven applications, and system architects requiring a messaging platform with strong multi-tenancy and geo-replication features. The cluster provides both the Pulsar binary protocol for high-performance messaging and HTTP admin APIs for management operations, making it suitable for both development and production evaluation scenarios.

## Features

- Multi-tenant messaging with namespace isolation and resource quotas
- Unified publish-subscribe and message queuing in a single platform
- BookKeeper integration for durable message storage across multiple bookies
- Pulsar Manager web interface for cluster monitoring and administration
- Schema registry support for message schema evolution and validation
- Geo-replication capabilities for cross-region message distribution
- Tiered storage support for cost-effective long-term message retention
- Pulsar Functions framework for serverless stream processing

## Use cases

- Event-driven microservices architecture with guaranteed message delivery
- Real-time analytics pipelines requiring both streaming and queuing semantics
- Multi-tenant SaaS platforms needing isolated messaging per customer
- IoT data ingestion with high-throughput message processing requirements
- Financial services requiring exactly-once message delivery guarantees
- Content distribution networks with geo-replicated message flows
- Log aggregation and centralized logging infrastructure

## Prerequisites

- Docker Engine 20.10+ and Docker Compose V2 for container orchestration
- Minimum 3GB RAM available for the cluster (ZooKeeper 256MB, bookies 512MB each, broker 512MB)
- Ports 6650 (Pulsar binary protocol), 8080 (admin API), 9527 and 7750 (Pulsar Manager) available
- Basic understanding of messaging concepts like topics, subscriptions, and producers/consumers
- Familiarity with BookKeeper storage concepts for message persistence configuration
- Knowledge of ZooKeeper for cluster metadata and coordination troubleshooting

## docker-compose.yml

```yaml
services:
  zookeeper:
    image: apachepulsar/pulsar:latest
    hostname: zookeeper
    command: bin/pulsar zookeeper
    environment:
      - PULSAR_MEM=-Xms256m -Xmx256m
    volumes:
      - zookeeper_data:/pulsar/data/zookeeper
    networks:
      - pulsar-net
    restart: unless-stopped

  init-cluster:
    image: apachepulsar/pulsar:latest
    command: bin/pulsar initialize-cluster-metadata --cluster pulsar-cluster --zookeeper zookeeper:2181 --configuration-store zookeeper:2181 --web-service-url http://broker:8080 --broker-service-url pulsar://broker:6650
    depends_on:
      - zookeeper
    networks:
      - pulsar-net

  bookie1:
    image: apachepulsar/pulsar:latest
    hostname: bookie1
    command: bin/pulsar bookie
    environment:
      - PULSAR_MEM=-Xms512m -Xmx512m
      - clusterName=pulsar-cluster
    volumes:
      - bookie1_data:/pulsar/data/bookkeeper
    depends_on:
      - zookeeper
      - init-cluster
    networks:
      - pulsar-net
    restart: unless-stopped

  bookie2:
    image: apachepulsar/pulsar:latest
    hostname: bookie2
    command: bin/pulsar bookie
    environment:
      - PULSAR_MEM=-Xms512m -Xmx512m
      - clusterName=pulsar-cluster
    volumes:
      - bookie2_data:/pulsar/data/bookkeeper
    depends_on:
      - zookeeper
      - init-cluster
    networks:
      - pulsar-net
    restart: unless-stopped

  broker:
    image: apachepulsar/pulsar:latest
    hostname: broker
    command: bin/pulsar broker
    ports:
      - "6650:6650"
      - "8080:8080"
    environment:
      - PULSAR_MEM=-Xms512m -Xmx512m
      - clusterName=pulsar-cluster
      - zookeeperServers=zookeeper:2181
      - configurationStoreServers=zookeeper:2181
    depends_on:
      - bookie1
      - bookie2
    networks:
      - pulsar-net
    restart: unless-stopped

  pulsar-manager:
    image: apachepulsar/pulsar-manager:latest
    ports:
      - "9527:9527"
      - "7750:7750"
    environment:
      SPRING_CONFIGURATION_FILE: /pulsar-manager/pulsar-manager/application.properties
    depends_on:
      - broker
    networks:
      - pulsar-net
    restart: unless-stopped

volumes:
  zookeeper_data:
  bookie1_data:
  bookie2_data:

networks:
  pulsar-net:
    driver: bridge
```

## .env

```bash
# Pulsar Configuration
PULSAR_CLUSTER_NAME=pulsar-cluster

# Pulsar Manager
PULSAR_MANAGER_ADMIN_USER=admin
PULSAR_MANAGER_ADMIN_PASSWORD=secure_manager_password
```

## Usage notes

- Pulsar broker at pulsar://localhost:6650
- Admin API at http://localhost:8080
- Pulsar Manager at http://localhost:9527
- Initialize manager: curl -X PUT http://localhost:7750/pulsar-manager/users/superuser

## Troubleshooting

- Broker fails to start with 'Unable to connect to zookeeper': Ensure ZooKeeper is fully started before broker initialization by checking container logs
- BookKeeper bookie registration errors: Verify the init-cluster container completed successfully and cluster metadata was properly initialized
- Pulsar Manager shows 'No clusters found': Use curl -X PUT http://localhost:7750/pulsar-manager/users/superuser to initialize the management interface
- Client connection refused on port 6650: Check that broker container is running and firewall allows connections to the Pulsar binary protocol port
- High memory usage warnings in logs: Adjust PULSAR_MEM environment variables for each service based on available system resources
- Topic creation fails with metadata errors: Restart the broker service after ensuring ZooKeeper and bookies are healthy and accessible

