Apache Pulsar Cluster

advanced

Apache Pulsar messaging and streaming platform with manager UI.

Overview

Apache Pulsar is a cloud-native distributed messaging and event-streaming platform originally developed at Yahoo and later open-sourced to the Apache Software Foundation. Unlike traditional messaging systems, Pulsar combines the best of both worlds by providing unified queuing and streaming capabilities in a single platform, with built-in multi-tenancy, geo-replication, and tiered storage. Its unique architecture separates computation from storage, using Apache BookKeeper as the underlying storage layer, which enables infinite message retention and horizontal scaling without data rebalancing. Pulsar Manager serves as a comprehensive web-based administration interface that simplifies cluster management, tenant configuration, and monitoring. Together, they form a complete messaging infrastructure that handles everything from real-time stream processing to traditional message queuing, with advanced features like schema evolution, transactions, and Pulsar Functions for serverless computing. This combination is particularly valuable for organizations building modern, cloud-native applications that require reliable, scalable messaging with enterprise-grade features like multi-tenancy and disaster recovery through geo-replication.

Key Features

Unified queuing and streaming model supporting both pub-sub and message queuing patterns
Multi-tenant architecture with namespace isolation and resource quotas
Geo-replication for disaster recovery and global message distribution
Tiered storage for automatic data archiving to cloud storage systems
Built-in schema registry with schema evolution support
Pulsar Functions for serverless stream processing and transformations
Topic compaction for maintaining latest state per message key
Web-based Pulsar Manager UI for cluster administration and monitoring

Common Use Cases

1Event-driven microservices architectures requiring reliable message delivery
2Multi-tenant SaaS platforms needing isolated messaging per customer
3IoT data ingestion and real-time analytics for sensor networks
4Financial trading systems requiring low-latency message processing
5Log aggregation and centralized logging for distributed applications
6Stream processing pipelines for real-time data transformation
7Cross-datacenter replication for global application deployments

Prerequisites

Minimum 4GB RAM recommended for standalone Pulsar deployment
Docker Engine 19.03 or higher with Docker Compose v2 support
Ports 6650, 8080, 9527, and 7750 available on the host system
Understanding of messaging patterns and pub-sub concepts
Basic knowledge of Apache BookKeeper storage concepts
Familiarity with tenant and namespace management in multi-tenant systems

For development & testing. Review security settings, change default credentials, and test thoroughly before production use. See Terms

docker-compose.yml

1services: 
2  pulsar: 
3    image: apachepulsar/pulsar:latest
4    container_name: pulsar
5    restart: unless-stopped
6    command: bin/pulsar standalone
7    ports: 
8      - "${PULSAR_PORT:-6650}:6650"
9      - "${PULSAR_HTTP_PORT:-8080}:8080"
10    volumes: 
11      - pulsar_data:/pulsar/data
12      - pulsar_conf:/pulsar/conf
13    networks: 
14      - pulsar-network
15
16  pulsar-manager: 
17    image: apachepulsar/pulsar-manager:latest
18    container_name: pulsar-manager
19    restart: unless-stopped
20    ports: 
21      - "${MANAGER_PORT:-9527}:9527"
22      - "7750:7750"
23    depends_on: 
24      - pulsar
25    environment: 
26      SPRING_CONFIGURATION_FILE: /pulsar-manager/pulsar-manager/application.properties
27    networks: 
28      - pulsar-network
29
30volumes: 
31  pulsar_data: 
32  pulsar_conf: 
33
34networks: 
35  pulsar-network: 
36    driver: bridge

.env Template

.env

1# Apache Pulsar
2PULSAR_PORT=6650
3PULSAR_HTTP_PORT=8080
4MANAGER_PORT=9527

Usage Notes

1Pulsar broker at localhost:6650
2Admin API at http://localhost:8080
3Pulsar Manager at http://localhost:9527
4Multi-tenant messaging system

Individual Services(2 services)

Copy individual services to mix and match with your existing compose files.

pulsar

pulsar:
  image: apachepulsar/pulsar:latest
  container_name: pulsar
  restart: unless-stopped
  command: bin/pulsar standalone
  ports:
    - ${PULSAR_PORT:-6650}:6650
    - ${PULSAR_HTTP_PORT:-8080}:8080
  volumes:
    - pulsar_data:/pulsar/data
    - pulsar_conf:/pulsar/conf
  networks:
    - pulsar-network

pulsar-manager

pulsar-manager:
  image: apachepulsar/pulsar-manager:latest
  container_name: pulsar-manager
  restart: unless-stopped
  ports:
    - ${MANAGER_PORT:-9527}:9527
    - "7750:7750"
  depends_on:
    - pulsar
  environment:
    SPRING_CONFIGURATION_FILE: /pulsar-manager/pulsar-manager/application.properties
  networks:
    - pulsar-network

Quick Start

terminal

1# 1. Create the compose file
2cat > docker-compose.yml << 'EOF'
3services:
4  pulsar:
5    image: apachepulsar/pulsar:latest
6    container_name: pulsar
7    restart: unless-stopped
8    command: bin/pulsar standalone
9    ports:
10      - "${PULSAR_PORT:-6650}:6650"
11      - "${PULSAR_HTTP_PORT:-8080}:8080"
12    volumes:
13      - pulsar_data:/pulsar/data
14      - pulsar_conf:/pulsar/conf
15    networks:
16      - pulsar-network
17
18  pulsar-manager:
19    image: apachepulsar/pulsar-manager:latest
20    container_name: pulsar-manager
21    restart: unless-stopped
22    ports:
23      - "${MANAGER_PORT:-9527}:9527"
24      - "7750:7750"
25    depends_on:
26      - pulsar
27    environment:
28      SPRING_CONFIGURATION_FILE: /pulsar-manager/pulsar-manager/application.properties
29    networks:
30      - pulsar-network
31
32volumes:
33  pulsar_data:
34  pulsar_conf:
35
36networks:
37  pulsar-network:
38    driver: bridge
39EOF
40
41# 2. Create the .env file
42cat > .env << 'EOF'
43# Apache Pulsar
44PULSAR_PORT=6650
45PULSAR_HTTP_PORT=8080
46MANAGER_PORT=9527
47EOF
48
49# 3. Start the services
50docker compose up -d
51
52# 4. View logs
53docker compose logs -f

One-Liner

Run this command to download and set up the recipe in one step:

terminal

1curl -fsSL https://docker.recipes/api/recipes/apache-pulsar-cluster/run | bash

Troubleshooting

Pulsar Manager shows 'Connection refused' to broker: Ensure pulsar service is fully started and accessible on port 6650 before accessing manager UI
Out of memory errors during startup: Increase Docker memory allocation to at least 4GB or set PULSAR_MEM environment variable
Topics not visible in Pulsar Manager: Initialize manager with proper cluster configuration using the REST API at port 7750
Persistent volume permission denied errors: Ensure Docker has proper permissions to write to mounted volume directories
Slow message consumption: Check if consumer is acknowledging messages properly and consider adjusting receive queue size
BookKeeper write failures: Verify sufficient disk space and that ensemble/quorum settings match available bookies

Community Notes

Loading notes...

Download Recipe Kit

Get all files in a ready-to-deploy package

Includes docker-compose.yml, .env template, README, and license

Components

pulsarpulsar-manager

Related Recipes

Ad Space

Shortcuts: C CopyF FavoriteD Download