docker.recipes

Apache Pulsar Cluster

advanced

Cloud-native distributed messaging and streaming platform with multi-tenancy.

Overview

Apache Pulsar is a cloud-native, distributed messaging and streaming platform originally developed by Yahoo and later open-sourced through the Apache Software Foundation. It provides unified queuing and streaming capabilities with multi-tenancy, geo-replication, and tiered storage support, making it a powerful alternative to traditional messaging systems like Apache Kafka and RabbitMQ. Pulsar's architecture separates serving from storage, enabling independent scaling and better resource utilization. This deployment creates a complete Pulsar cluster with six interconnected services: a ZooKeeper instance for coordination, an initialization container for cluster metadata setup, two BookKeeper bookies (bookie1 and bookie2) for persistent message storage, a Pulsar broker for message routing and client connections, and Pulsar Manager for web-based administration. The setup demonstrates Pulsar's distributed architecture with proper separation of concerns between metadata management, storage, and serving layers. This configuration is ideal for organizations evaluating Pulsar's capabilities, development teams building event-driven applications, and system architects requiring a messaging platform with strong multi-tenancy and geo-replication features. The cluster provides both the Pulsar binary protocol for high-performance messaging and HTTP admin APIs for management operations, making it suitable for both development and production evaluation scenarios.

Key Features

  • Multi-tenant messaging with namespace isolation and resource quotas
  • Unified publish-subscribe and message queuing in a single platform
  • BookKeeper integration for durable message storage across multiple bookies
  • Pulsar Manager web interface for cluster monitoring and administration
  • Schema registry support for message schema evolution and validation
  • Geo-replication capabilities for cross-region message distribution
  • Tiered storage support for cost-effective long-term message retention
  • Pulsar Functions framework for serverless stream processing

Common Use Cases

  • 1Event-driven microservices architecture with guaranteed message delivery
  • 2Real-time analytics pipelines requiring both streaming and queuing semantics
  • 3Multi-tenant SaaS platforms needing isolated messaging per customer
  • 4IoT data ingestion with high-throughput message processing requirements
  • 5Financial services requiring exactly-once message delivery guarantees
  • 6Content distribution networks with geo-replicated message flows
  • 7Log aggregation and centralized logging infrastructure

Prerequisites

  • Docker Engine 20.10+ and Docker Compose V2 for container orchestration
  • Minimum 3GB RAM available for the cluster (ZooKeeper 256MB, bookies 512MB each, broker 512MB)
  • Ports 6650 (Pulsar binary protocol), 8080 (admin API), 9527 and 7750 (Pulsar Manager) available
  • Basic understanding of messaging concepts like topics, subscriptions, and producers/consumers
  • Familiarity with BookKeeper storage concepts for message persistence configuration
  • Knowledge of ZooKeeper for cluster metadata and coordination troubleshooting

For development & testing. Review security settings, change default credentials, and test thoroughly before production use. See Terms

docker-compose.yml

docker-compose.yml
1services: 
2  zookeeper: 
3    image: apachepulsar/pulsar:latest
4    hostname: zookeeper
5    command: bin/pulsar zookeeper
6    environment: 
7      - PULSAR_MEM=-Xms256m -Xmx256m
8    volumes: 
9      - zookeeper_data:/pulsar/data/zookeeper
10    networks: 
11      - pulsar-net
12    restart: unless-stopped
13
14  init-cluster: 
15    image: apachepulsar/pulsar:latest
16    command: bin/pulsar initialize-cluster-metadata --cluster pulsar-cluster --zookeeper zookeeper:2181 --configuration-store zookeeper:2181 --web-service-url http://broker:8080 --broker-service-url pulsar://broker:6650
17    depends_on: 
18      - zookeeper
19    networks: 
20      - pulsar-net
21
22  bookie1: 
23    image: apachepulsar/pulsar:latest
24    hostname: bookie1
25    command: bin/pulsar bookie
26    environment: 
27      - PULSAR_MEM=-Xms512m -Xmx512m
28      - clusterName=pulsar-cluster
29    volumes: 
30      - bookie1_data:/pulsar/data/bookkeeper
31    depends_on: 
32      - zookeeper
33      - init-cluster
34    networks: 
35      - pulsar-net
36    restart: unless-stopped
37
38  bookie2: 
39    image: apachepulsar/pulsar:latest
40    hostname: bookie2
41    command: bin/pulsar bookie
42    environment: 
43      - PULSAR_MEM=-Xms512m -Xmx512m
44      - clusterName=pulsar-cluster
45    volumes: 
46      - bookie2_data:/pulsar/data/bookkeeper
47    depends_on: 
48      - zookeeper
49      - init-cluster
50    networks: 
51      - pulsar-net
52    restart: unless-stopped
53
54  broker: 
55    image: apachepulsar/pulsar:latest
56    hostname: broker
57    command: bin/pulsar broker
58    ports: 
59      - "6650:6650"
60      - "8080:8080"
61    environment: 
62      - PULSAR_MEM=-Xms512m -Xmx512m
63      - clusterName=pulsar-cluster
64      - zookeeperServers=zookeeper:2181
65      - configurationStoreServers=zookeeper:2181
66    depends_on: 
67      - bookie1
68      - bookie2
69    networks: 
70      - pulsar-net
71    restart: unless-stopped
72
73  pulsar-manager: 
74    image: apachepulsar/pulsar-manager:latest
75    ports: 
76      - "9527:9527"
77      - "7750:7750"
78    environment: 
79      SPRING_CONFIGURATION_FILE: /pulsar-manager/pulsar-manager/application.properties
80    depends_on: 
81      - broker
82    networks: 
83      - pulsar-net
84    restart: unless-stopped
85
86volumes: 
87  zookeeper_data: 
88  bookie1_data: 
89  bookie2_data: 
90
91networks: 
92  pulsar-net: 
93    driver: bridge

.env Template

.env
1# Pulsar Configuration
2PULSAR_CLUSTER_NAME=pulsar-cluster
3
4# Pulsar Manager
5PULSAR_MANAGER_ADMIN_USER=admin
6PULSAR_MANAGER_ADMIN_PASSWORD=secure_manager_password

Usage Notes

  1. 1Pulsar broker at pulsar://localhost:6650
  2. 2Admin API at http://localhost:8080
  3. 3Pulsar Manager at http://localhost:9527
  4. 4Initialize manager: curl -X PUT http://localhost:7750/pulsar-manager/users/superuser

Individual Services(6 services)

Copy individual services to mix and match with your existing compose files.

zookeeper
zookeeper:
  image: apachepulsar/pulsar:latest
  hostname: zookeeper
  command: bin/pulsar zookeeper
  environment:
    - PULSAR_MEM=-Xms256m -Xmx256m
  volumes:
    - zookeeper_data:/pulsar/data/zookeeper
  networks:
    - pulsar-net
  restart: unless-stopped
init-cluster
init-cluster:
  image: apachepulsar/pulsar:latest
  command: bin/pulsar initialize-cluster-metadata --cluster pulsar-cluster --zookeeper zookeeper:2181 --configuration-store zookeeper:2181 --web-service-url http://broker:8080 --broker-service-url pulsar://broker:6650
  depends_on:
    - zookeeper
  networks:
    - pulsar-net
bookie1
bookie1:
  image: apachepulsar/pulsar:latest
  hostname: bookie1
  command: bin/pulsar bookie
  environment:
    - PULSAR_MEM=-Xms512m -Xmx512m
    - clusterName=pulsar-cluster
  volumes:
    - bookie1_data:/pulsar/data/bookkeeper
  depends_on:
    - zookeeper
    - init-cluster
  networks:
    - pulsar-net
  restart: unless-stopped
bookie2
bookie2:
  image: apachepulsar/pulsar:latest
  hostname: bookie2
  command: bin/pulsar bookie
  environment:
    - PULSAR_MEM=-Xms512m -Xmx512m
    - clusterName=pulsar-cluster
  volumes:
    - bookie2_data:/pulsar/data/bookkeeper
  depends_on:
    - zookeeper
    - init-cluster
  networks:
    - pulsar-net
  restart: unless-stopped
broker
broker:
  image: apachepulsar/pulsar:latest
  hostname: broker
  command: bin/pulsar broker
  ports:
    - "6650:6650"
    - "8080:8080"
  environment:
    - PULSAR_MEM=-Xms512m -Xmx512m
    - clusterName=pulsar-cluster
    - zookeeperServers=zookeeper:2181
    - configurationStoreServers=zookeeper:2181
  depends_on:
    - bookie1
    - bookie2
  networks:
    - pulsar-net
  restart: unless-stopped
pulsar-manager
pulsar-manager:
  image: apachepulsar/pulsar-manager:latest
  ports:
    - "9527:9527"
    - "7750:7750"
  environment:
    SPRING_CONFIGURATION_FILE: /pulsar-manager/pulsar-manager/application.properties
  depends_on:
    - broker
  networks:
    - pulsar-net
  restart: unless-stopped

Quick Start

terminal
1# 1. Create the compose file
2cat > docker-compose.yml << 'EOF'
3services:
4  zookeeper:
5    image: apachepulsar/pulsar:latest
6    hostname: zookeeper
7    command: bin/pulsar zookeeper
8    environment:
9      - PULSAR_MEM=-Xms256m -Xmx256m
10    volumes:
11      - zookeeper_data:/pulsar/data/zookeeper
12    networks:
13      - pulsar-net
14    restart: unless-stopped
15
16  init-cluster:
17    image: apachepulsar/pulsar:latest
18    command: bin/pulsar initialize-cluster-metadata --cluster pulsar-cluster --zookeeper zookeeper:2181 --configuration-store zookeeper:2181 --web-service-url http://broker:8080 --broker-service-url pulsar://broker:6650
19    depends_on:
20      - zookeeper
21    networks:
22      - pulsar-net
23
24  bookie1:
25    image: apachepulsar/pulsar:latest
26    hostname: bookie1
27    command: bin/pulsar bookie
28    environment:
29      - PULSAR_MEM=-Xms512m -Xmx512m
30      - clusterName=pulsar-cluster
31    volumes:
32      - bookie1_data:/pulsar/data/bookkeeper
33    depends_on:
34      - zookeeper
35      - init-cluster
36    networks:
37      - pulsar-net
38    restart: unless-stopped
39
40  bookie2:
41    image: apachepulsar/pulsar:latest
42    hostname: bookie2
43    command: bin/pulsar bookie
44    environment:
45      - PULSAR_MEM=-Xms512m -Xmx512m
46      - clusterName=pulsar-cluster
47    volumes:
48      - bookie2_data:/pulsar/data/bookkeeper
49    depends_on:
50      - zookeeper
51      - init-cluster
52    networks:
53      - pulsar-net
54    restart: unless-stopped
55
56  broker:
57    image: apachepulsar/pulsar:latest
58    hostname: broker
59    command: bin/pulsar broker
60    ports:
61      - "6650:6650"
62      - "8080:8080"
63    environment:
64      - PULSAR_MEM=-Xms512m -Xmx512m
65      - clusterName=pulsar-cluster
66      - zookeeperServers=zookeeper:2181
67      - configurationStoreServers=zookeeper:2181
68    depends_on:
69      - bookie1
70      - bookie2
71    networks:
72      - pulsar-net
73    restart: unless-stopped
74
75  pulsar-manager:
76    image: apachepulsar/pulsar-manager:latest
77    ports:
78      - "9527:9527"
79      - "7750:7750"
80    environment:
81      SPRING_CONFIGURATION_FILE: /pulsar-manager/pulsar-manager/application.properties
82    depends_on:
83      - broker
84    networks:
85      - pulsar-net
86    restart: unless-stopped
87
88volumes:
89  zookeeper_data:
90  bookie1_data:
91  bookie2_data:
92
93networks:
94  pulsar-net:
95    driver: bridge
96EOF
97
98# 2. Create the .env file
99cat > .env << 'EOF'
100# Pulsar Configuration
101PULSAR_CLUSTER_NAME=pulsar-cluster
102
103# Pulsar Manager
104PULSAR_MANAGER_ADMIN_USER=admin
105PULSAR_MANAGER_ADMIN_PASSWORD=secure_manager_password
106EOF
107
108# 3. Start the services
109docker compose up -d
110
111# 4. View logs
112docker compose logs -f

One-Liner

Run this command to download and set up the recipe in one step:

terminal
1curl -fsSL https://docker.recipes/api/recipes/pulsar-cluster/run | bash

Troubleshooting

  • Broker fails to start with 'Unable to connect to zookeeper': Ensure ZooKeeper is fully started before broker initialization by checking container logs
  • BookKeeper bookie registration errors: Verify the init-cluster container completed successfully and cluster metadata was properly initialized
  • Pulsar Manager shows 'No clusters found': Use curl -X PUT http://localhost:7750/pulsar-manager/users/superuser to initialize the management interface
  • Client connection refused on port 6650: Check that broker container is running and firewall allows connections to the Pulsar binary protocol port
  • High memory usage warnings in logs: Adjust PULSAR_MEM environment variables for each service based on available system resources
  • Topic creation fails with metadata errors: Restart the broker service after ensuring ZooKeeper and bookies are healthy and accessible

Community Notes

Loading...
Loading notes...

Download Recipe Kit

Get all files in a ready-to-deploy package

Includes docker-compose.yml, .env template, README, and license

Components

pulsar-brokerpulsar-bookiezookeeperpulsar-manager

Tags

#pulsar#messaging#streaming#multi-tenant#cloud-native

Category

Message Queues & Brokers

Related Recipes

Ad Space