# RabbitMQ HA Cluster

> High-availability RabbitMQ cluster with mirrored queues, management UI, and monitoring.

- Category: Message Queues & Brokers
- Difficulty: advanced
- Components: rabbitmq, haproxy, prometheus, grafana
- Tags: rabbitmq, message-queue, ha, cluster, amqp
- Source: https://docker.recipes/messaging/rabbitmq-ha-cluster

## Overview

RabbitMQ is an open-source message broker that implements the Advanced Message Queuing Protocol (AMQP), originally developed by Rabbit Technologies and later acquired by VMware. It serves as a reliable intermediary for message passing between applications, providing features like message persistence, routing flexibility, and delivery acknowledgments. RabbitMQ supports multiple messaging protocols including AMQP, MQTT, and STOMP, making it versatile for various messaging patterns from simple task queues to complex routing scenarios.

This high-availability cluster combines three RabbitMQ nodes with HAProxy load balancing to eliminate single points of failure in message broker infrastructure. The cluster uses Erlang's built-in clustering capabilities with mirrored queues to ensure message durability and availability even when individual nodes fail. HAProxy provides intelligent load balancing for both AMQP connections and management UI access, while Prometheus collects metrics from all components and Grafana visualizes cluster health, message rates, and queue depths.

Production environments running microservices, e-commerce platforms, or IoT systems benefit significantly from this stack when message delivery reliability is critical. The combination addresses scenarios where message loss would impact business operations or user experience, providing automatic failover, queue mirroring across nodes, and comprehensive monitoring to detect issues before they affect applications. Organizations transitioning from single RabbitMQ instances or seeking to improve their messaging infrastructure's resilience will find this cluster configuration essential for maintaining service availability.

## Features

- Three-node RabbitMQ cluster with automatic leader election and node synchronization
- Mirrored queues across all cluster nodes ensuring message persistence during node failures
- HAProxy load balancing for AMQP connections on port 5672 with health checking
- Management UI load balancing with sticky sessions for consistent administrative access
- Prometheus metrics collection from RabbitMQ nodes including queue depths and message rates
- Grafana dashboards for real-time cluster monitoring and historical performance analysis
- Erlang cookie-based cluster authentication for secure inter-node communication
- Support for multiple messaging protocols (AMQP, MQTT, STOMP) across the cluster

## Use cases

- E-commerce platforms requiring guaranteed order processing and inventory updates
- Microservices architectures needing reliable inter-service communication
- IoT data collection systems handling sensor data from thousands of devices
- Financial systems processing payment transactions with zero tolerance for message loss
- Task queue systems distributing background jobs across multiple worker processes
- Event-driven architectures requiring message routing between multiple applications
- Enterprise integration scenarios connecting legacy systems with modern applications

## Prerequisites

- Minimum 4GB RAM for the complete stack (1GB per RabbitMQ node + 1GB for monitoring)
- Docker Engine version 20.10+ with Docker Compose v2 support
- Available ports: 5672 (AMQP), 15672 (Management), 8404 (HAProxy stats), 9090 (Prometheus), 3000 (Grafana)
- Basic understanding of AMQP concepts and RabbitMQ clustering mechanisms
- Familiarity with HAProxy configuration for troubleshooting load balancer issues
- Knowledge of PromQL for customizing Prometheus queries and Grafana dashboards

## docker-compose.yml

```yaml
services:
  rabbitmq1:
    image: rabbitmq:3-management
    hostname: rabbitmq1
    environment:
      RABBITMQ_ERLANG_COOKIE: ${ERLANG_COOKIE}
      RABBITMQ_DEFAULT_USER: ${RABBITMQ_USER}
      RABBITMQ_DEFAULT_PASS: ${RABBITMQ_PASSWORD}
    volumes:
      - rabbitmq1_data:/var/lib/rabbitmq
      - ./rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf:ro
      - ./definitions.json:/etc/rabbitmq/definitions.json:ro
    networks:
      - rabbitmq-net
    restart: unless-stopped

  rabbitmq2:
    image: rabbitmq:3-management
    hostname: rabbitmq2
    environment:
      RABBITMQ_ERLANG_COOKIE: ${ERLANG_COOKIE}
      RABBITMQ_DEFAULT_USER: ${RABBITMQ_USER}
      RABBITMQ_DEFAULT_PASS: ${RABBITMQ_PASSWORD}
    volumes:
      - rabbitmq2_data:/var/lib/rabbitmq
      - ./rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf:ro
    depends_on:
      - rabbitmq1
    networks:
      - rabbitmq-net
    restart: unless-stopped

  rabbitmq3:
    image: rabbitmq:3-management
    hostname: rabbitmq3
    environment:
      RABBITMQ_ERLANG_COOKIE: ${ERLANG_COOKIE}
      RABBITMQ_DEFAULT_USER: ${RABBITMQ_USER}
      RABBITMQ_DEFAULT_PASS: ${RABBITMQ_PASSWORD}
    volumes:
      - rabbitmq3_data:/var/lib/rabbitmq
      - ./rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf:ro
    depends_on:
      - rabbitmq1
    networks:
      - rabbitmq-net
    restart: unless-stopped

  haproxy:
    image: haproxy:latest
    ports:
      - "5672:5672"
      - "15672:15672"
      - "8404:8404"
    volumes:
      - ./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
    depends_on:
      - rabbitmq1
      - rabbitmq2
      - rabbitmq3
    networks:
      - rabbitmq-net
    restart: unless-stopped

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus_data:/prometheus
    networks:
      - rabbitmq-net
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
    volumes:
      - grafana_data:/var/lib/grafana
    depends_on:
      - prometheus
    networks:
      - rabbitmq-net
    restart: unless-stopped

volumes:
  rabbitmq1_data:
  rabbitmq2_data:
  rabbitmq3_data:
  prometheus_data:
  grafana_data:

networks:
  rabbitmq-net:
    driver: bridge
```

## .env

```bash
# RabbitMQ Configuration
ERLANG_COOKIE=secure_erlang_cookie_value
RABBITMQ_USER=admin
RABBITMQ_PASSWORD=secure_rabbitmq_password

# Grafana
GRAFANA_PASSWORD=secure_grafana_password
```

## Usage notes

- RabbitMQ Management at http://localhost:15672
- AMQP connection via HAProxy at localhost:5672
- HAProxy stats at http://localhost:8404
- Create rabbitmq.conf with cluster settings

## Troubleshooting

- RabbitMQ nodes failing to join cluster: Verify ERLANG_COOKIE environment variable is identical across all nodes and check hostname resolution
- HAProxy showing backend servers as DOWN: Check RabbitMQ management plugin is enabled and nodes are responding on port 15672
- Messages not replicating across cluster: Ensure queue mirroring policy is configured in definitions.json or applied via management UI
- High memory usage warnings: Adjust RabbitMQ memory thresholds in rabbitmq.conf and monitor queue depths for unbounded growth
- Prometheus not scraping RabbitMQ metrics: Verify prometheus_rabbitmq_exporter plugin is enabled and accessible on port 15692
- Split-brain cluster scenarios: Implement cluster partition handling strategy and check network connectivity between RabbitMQ nodes

