# HashiCorp Consul Service Discovery

> Service mesh and configuration with Consul cluster, ACL, and UI.

- Category: DevOps & CI/CD
- Difficulty: advanced
- Components: consul-server, consul-agent, nginx, prometheus
- Tags: consul, hashicorp, service-discovery, mesh, configuration
- Source: https://docker.recipes/devops/consul-cluster

## Overview

HashiCorp Consul is a distributed service mesh solution that provides service discovery, configuration management, and secure service-to-service communication across any cloud or runtime environment. Originally developed by HashiCorp in 2014, Consul has become the backbone for microservices architectures requiring dynamic service registration, health checking, and distributed configuration management. Unlike simple key-value stores like etcd, Consul offers a comprehensive service networking platform with built-in DNS interface, HTTP API, and advanced features like service intentions for zero-trust networking.

This deployment creates a production-grade Consul cluster with three server nodes for high availability, complemented by Prometheus for metrics collection and Grafana for visualization. The three-server configuration ensures fault tolerance with proper quorum consensus, while the dedicated consul-agent demonstrates client-side service registration patterns. Prometheus integrates with Consul's built-in metrics endpoint to monitor cluster health, service discovery events, and performance metrics across the entire service mesh.

This stack is ideal for platform engineering teams implementing microservices architecture, DevOps engineers managing multi-datacenter deployments, and organizations transitioning from monolithic applications to distributed systems. The combination of Consul's service mesh capabilities with Prometheus monitoring provides complete visibility into service communication patterns, health status, and performance bottlenecks. Companies like Criteo, Citadel, and many Fortune 500 enterprises rely on similar Consul deployments to manage thousands of services across hybrid cloud environments.

## Features

- Multi-raft consensus algorithm with 3-node server cluster for Byzantine fault tolerance
- DNS-based service discovery on port 8600 with automatic SRV record generation
- HTTP API for dynamic service registration and health check configuration
- Web UI dashboard for visual service topology and key-value store management
- Gossip protocol for efficient cluster membership and failure detection
- Built-in health checking with HTTP, TCP, and script-based health monitors
- Prometheus metrics integration with /v1/agent/metrics endpoint
- Key-value store with hierarchical configuration management and watch capabilities

## Use cases

- Microservices platforms requiring dynamic service registration and discovery
- Multi-region application deployments with cross-datacenter service communication
- Legacy application modernization with gradual service mesh adoption
- Configuration management for applications requiring real-time config updates
- Load balancer backends requiring automatic upstream service detection
- Development environments mimicking production service discovery patterns
- Container orchestration platforms needing external service registry integration

## Prerequisites

- Minimum 2GB RAM for 3-node Consul cluster plus monitoring stack
- Available ports 8500 (HTTP), 8600 (DNS), 3000 (Grafana), 9090 (Prometheus)
- prometheus.yml configuration file in current directory with Consul scrape targets
- GRAFANA_PASSWORD environment variable set for admin authentication
- Understanding of service discovery concepts and REST API interaction
- Network connectivity allowing inter-container communication on consul_net bridge

## docker-compose.yml

```yaml
services:
  consul-server-1:
    image: hashicorp/consul:latest
    command: agent -server -bootstrap-expect=3 -ui -client=0.0.0.0 -bind=0.0.0.0
    ports:
      - "8500:8500"
      - "8600:8600/udp"
    environment:
      - CONSUL_BIND_INTERFACE=eth0
    volumes:
      - consul_server_1:/consul/data
    networks:
      - consul_net

  consul-server-2:
    image: hashicorp/consul:latest
    command: agent -server -retry-join=consul-server-1 -client=0.0.0.0 -bind=0.0.0.0
    environment:
      - CONSUL_BIND_INTERFACE=eth0
    volumes:
      - consul_server_2:/consul/data
    depends_on:
      - consul-server-1
    networks:
      - consul_net

  consul-server-3:
    image: hashicorp/consul:latest
    command: agent -server -retry-join=consul-server-1 -client=0.0.0.0 -bind=0.0.0.0
    environment:
      - CONSUL_BIND_INTERFACE=eth0
    volumes:
      - consul_server_3:/consul/data
    depends_on:
      - consul-server-1
    networks:
      - consul_net

  consul-agent:
    image: hashicorp/consul:latest
    command: agent -retry-join=consul-server-1 -client=0.0.0.0 -bind=0.0.0.0
    environment:
      - CONSUL_BIND_INTERFACE=eth0
    depends_on:
      - consul-server-1
    networks:
      - consul_net

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    networks:
      - consul_net

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
    volumes:
      - grafana_data:/var/lib/grafana
    networks:
      - consul_net

volumes:
  consul_server_1:
  consul_server_2:
  consul_server_3:
  prometheus_data:
  grafana_data:

networks:
  consul_net:
```

## .env

```bash
# Consul Cluster
GRAFANA_PASSWORD=secure_grafana_password

# Consul UI at http://localhost:8500
# DNS at localhost:8600
```

## Usage notes

- Consul UI at http://localhost:8500
- DNS interface at port 8600/UDP
- 3-node server cluster for HA
- Service registration via API
- KV store for configuration

## Troubleshooting

- Consul leader election failed: Check server logs for port conflicts and ensure all three servers can communicate on port 8300
- Service registration returns 500 error: Verify consul-agent is connected to cluster using 'consul members' command
- DNS queries to port 8600 timeout: Confirm CONSUL_BIND_INTERFACE=eth0 environment variable and UDP port exposure
- Prometheus cannot scrape Consul metrics: Add 'prometheus_retention_time' to Consul config and verify /v1/agent/metrics endpoint accessibility
- Grafana dashboard shows no data: Check Prometheus data source configuration points to prometheus:9090 internal hostname
- Consul UI shows failed health checks: Review service definition health check intervals and ensure target services respond correctly

