# HashiCorp Consul Cluster

> Service mesh and service discovery platform for distributed systems.

- Category: DevOps & CI/CD
- Difficulty: advanced
- Components: consul-server, consul-client, consul-envoy
- Tags: service-mesh, consul, hashicorp, service-discovery, networking
- Source: https://docker.recipes/devops/hashicorp-consul

## Overview

HashiCorp Consul is a distributed service mesh and service discovery platform designed to connect and secure services across any runtime platform and public or private cloud. Originally developed by HashiCorp in 2014, Consul provides a full-featured control plane with service discovery, configuration, and segmentation functionality, enabling organizations to build secure, resilient distributed systems with automatic network configurations and service-to-service communication.

This three-server Consul cluster configuration establishes a highly available service registry with automated leader election and consensus through the Raft protocol. The consul-server nodes form a quorum-based cluster that maintains service catalogs, health checks, and key-value data, while the consul-client acts as a lightweight agent for service registration and discovery. Together, they create a distributed system that can handle node failures while maintaining service visibility and enabling secure inter-service communication through built-in service mesh capabilities.

This stack is ideal for platform engineering teams managing microservices architectures, DevOps engineers implementing service discovery in multi-cloud environments, and organizations requiring centralized configuration management with high availability. The combination of multiple server nodes with client agents provides the redundancy needed for production workloads while offering the flexibility to scale service registration across distributed applications.

## Features

- Raft consensus algorithm for distributed leader election and data replication
- Built-in service discovery with DNS and HTTP interfaces on ports 8600 and 8500
- Integrated health checking with configurable check intervals and failure thresholds
- Key-value store for dynamic configuration and service metadata
- Service mesh capabilities with Connect for encrypted service-to-service communication
- Multi-datacenter federation support for cross-region service discovery
- Access Control Lists (ACL) system for fine-grained service and data permissions
- Web UI dashboard for cluster visualization and service monitoring

## Use cases

- Microservices service discovery in Kubernetes or Docker Swarm environments
- Multi-cloud service registry for applications spanning AWS, Azure, and GCP
- Configuration management for distributed applications requiring dynamic updates
- Service mesh implementation for zero-trust networking between microservices
- Load balancer backend discovery for HAProxy, NGINX, or cloud load balancers
- Cross-datacenter service communication for disaster recovery scenarios
- API gateway service registration for Kong, Ambassador, or Istio gateways

## Prerequisites

- Minimum 1.5GB RAM total (512MB per consul-server node recommended)
- Docker Engine 20.10+ with Docker Compose v2 support
- Available ports 8500 (HTTP API/UI) and 8600 (DNS) on the host system
- Understanding of distributed systems concepts like consensus and leader election
- Knowledge of service discovery patterns and health check configurations
- Network connectivity between all cluster nodes for gossip protocol communication

## docker-compose.yml

```yaml
services:
  consul-server1:
    image: hashicorp/consul:latest
    container_name: consul-server1
    command: agent -server -bootstrap-expect=3 -ui -client=0.0.0.0 -retry-join=consul-server2 -retry-join=consul-server3
    environment:
      - CONSUL_BIND_INTERFACE=eth0
    volumes:
      - consul-server1:/consul/data
    ports:
      - "8500:8500"
      - "8600:8600/udp"
    networks:
      - consul-network
    restart: unless-stopped

  consul-server2:
    image: hashicorp/consul:latest
    container_name: consul-server2
    command: agent -server -bootstrap-expect=3 -retry-join=consul-server1 -retry-join=consul-server3
    environment:
      - CONSUL_BIND_INTERFACE=eth0
    volumes:
      - consul-server2:/consul/data
    networks:
      - consul-network
    restart: unless-stopped

  consul-server3:
    image: hashicorp/consul:latest
    container_name: consul-server3
    command: agent -server -bootstrap-expect=3 -retry-join=consul-server1 -retry-join=consul-server2
    environment:
      - CONSUL_BIND_INTERFACE=eth0
    volumes:
      - consul-server3:/consul/data
    networks:
      - consul-network
    restart: unless-stopped

  consul-client:
    image: hashicorp/consul:latest
    container_name: consul-client
    command: agent -retry-join=consul-server1
    environment:
      - CONSUL_BIND_INTERFACE=eth0
    volumes:
      - consul-client:/consul/data
    networks:
      - consul-network
    restart: unless-stopped

volumes:
  consul-server1:
  consul-server2:
  consul-server3:
  consul-client:

networks:
  consul-network:
    driver: bridge
```

## .env

```bash
# HashiCorp Consul Cluster
# 3-node server cluster with 1 client
```

## Usage notes

- Consul UI at http://localhost:8500
- 3-node HA cluster
- Register services via API or config
- DNS interface at :8600
- Health checking built-in

## Troubleshooting

- Error 'No cluster leader': Ensure all three server nodes are running and can communicate on port 8300
- UI shows 'failed' health checks: Verify service definitions include proper health check endpoints and intervals
- Client cannot join cluster: Check that consul-client can resolve consul-server1 hostname via Docker networking
- Split-brain scenario after network partition: Restart minority nodes to rejoin the majority partition
- High memory usage on server nodes: Reduce log level or increase cleanup intervals in consul configuration
- Service registration fails with ACL errors: Ensure proper ACL tokens are configured if ACL system is enabled

