# Prometheus HA Stack

> High-availability Prometheus setup with Thanos for long-term storage.

- Category: Monitoring & Observability
- Difficulty: advanced
- Components: prometheus, thanos-sidecar, thanos-query, thanos-store, minio, alertmanager, grafana
- Tags: prometheus, thanos, monitoring, ha, metrics
- Source: https://docker.recipes/monitoring/prometheus-stack-ha

## Overview

Prometheus is an open-source monitoring and alerting toolkit that revolutionized the way organizations monitor their infrastructure and applications. Originally developed at SoundCloud in 2012, Prometheus pioneered the pull-based metrics collection model and introduced PromQL, a powerful query language for time-series data. Its dimensional data model using labels allows for flexible aggregation and filtering of metrics across complex, dynamic environments like Kubernetes clusters and microservices architectures.

This high-availability stack combines Prometheus with Thanos to solve the fundamental limitations of standalone Prometheus deployments: limited storage capacity, lack of global querying across instances, and single points of failure. Thanos sidecars run alongside each Prometheus instance, uploading data blocks to MinIO object storage while providing a unified query interface through Thanos Query. The store gateway enables querying of historical data directly from object storage, while Alertmanager handles notification routing and Grafana provides rich visualization capabilities across the entire monitoring ecosystem.

This configuration is essential for organizations running multiple Prometheus instances in production environments who need unlimited retention, global queries across data centers, and true high availability. Platform engineering teams, SRE organizations, and enterprises with compliance requirements for long-term metrics retention will find this stack indispensable for maintaining observability at scale while avoiding the operational complexity and cost of vendor-managed solutions.

## Features

- Dual Prometheus instances with Thanos sidecars for true high availability and zero data loss
- Unlimited metrics retention through Thanos store gateway and MinIO object storage
- Global querying across all Prometheus instances via Thanos Query with deduplication
- S3-compatible object storage with MinIO for cost-effective long-term data persistence
- Automatic block compaction and downsampling for efficient storage utilization
- Replica label deduplication to handle identical metrics from multiple Prometheus instances
- Integrated Alertmanager for centralized alert routing and notification management
- Grafana integration with Thanos Query for unified dashboards across all data sources

## Use cases

- Multi-region Kubernetes monitoring with global metrics visibility across clusters
- Enterprise infrastructure monitoring requiring years of metrics retention for compliance
- Platform engineering teams managing dozens of microservices across multiple environments
- Organizations migrating from vendor solutions like DataDog to reduce monitoring costs
- High-traffic SaaS applications needing reliable monitoring with zero data loss tolerance
- Financial services requiring long-term metrics storage for regulatory compliance audits
- Research institutions analyzing historical performance trends over extended periods

## Prerequisites

- Docker host with minimum 6GB RAM to support all components (Prometheus instances need 1GB+ each)
- At least 20GB available storage for MinIO object storage and local Prometheus data
- Network access to ports 3000, 9000-9001, 9090-9093, and 10902 for service interfaces
- Understanding of PromQL query language and Prometheus configuration syntax
- Familiarity with time-series data concepts and metrics cardinality considerations
- Basic knowledge of object storage concepts and S3-compatible APIs for troubleshooting

## docker-compose.yml

```yaml
services:
  prometheus1:
    image: prom/prometheus:latest
    container_name: prometheus1
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --storage.tsdb.retention.time=2h
      - --storage.tsdb.min-block-duration=2h
      - --storage.tsdb.max-block-duration=2h
      - --web.enable-lifecycle
      - --web.enable-admin-api
    volumes:
      - ./prometheus1.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus1-data:/prometheus
    ports:
      - "9090:9090"
    networks:
      - prometheus-network
    restart: unless-stopped

  thanos-sidecar1:
    image: quay.io/thanos/thanos:latest
    container_name: thanos-sidecar1
    command:
      - sidecar
      - --tsdb.path=/prometheus
      - --prometheus.url=http://prometheus1:9090
      - --grpc-address=0.0.0.0:10901
      - --http-address=0.0.0.0:10902
      - --objstore.config-file=/etc/thanos/bucket.yml
    volumes:
      - prometheus1-data:/prometheus:ro
      - ./bucket.yml:/etc/thanos/bucket.yml:ro
    depends_on:
      - prometheus1
      - minio
    networks:
      - prometheus-network
    restart: unless-stopped

  prometheus2:
    image: prom/prometheus:latest
    container_name: prometheus2
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --storage.tsdb.retention.time=2h
      - --storage.tsdb.min-block-duration=2h
      - --storage.tsdb.max-block-duration=2h
      - --web.enable-lifecycle
      - --web.enable-admin-api
    volumes:
      - ./prometheus2.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus2-data:/prometheus
    ports:
      - "9091:9090"
    networks:
      - prometheus-network
    restart: unless-stopped

  thanos-sidecar2:
    image: quay.io/thanos/thanos:latest
    container_name: thanos-sidecar2
    command:
      - sidecar
      - --tsdb.path=/prometheus
      - --prometheus.url=http://prometheus2:9090
      - --grpc-address=0.0.0.0:10901
      - --http-address=0.0.0.0:10902
      - --objstore.config-file=/etc/thanos/bucket.yml
    volumes:
      - prometheus2-data:/prometheus:ro
      - ./bucket.yml:/etc/thanos/bucket.yml:ro
    depends_on:
      - prometheus2
      - minio
    networks:
      - prometheus-network
    restart: unless-stopped

  thanos-query:
    image: quay.io/thanos/thanos:latest
    container_name: thanos-query
    command:
      - query
      - --http-address=0.0.0.0:9090
      - --grpc-address=0.0.0.0:10901
      - --store=thanos-sidecar1:10901
      - --store=thanos-sidecar2:10901
      - --store=thanos-store:10901
      - --query.replica-label=replica
    ports:
      - "10902:9090"
    networks:
      - prometheus-network
    restart: unless-stopped

  thanos-store:
    image: quay.io/thanos/thanos:latest
    container_name: thanos-store
    command:
      - store
      - --grpc-address=0.0.0.0:10901
      - --http-address=0.0.0.0:10902
      - --data-dir=/data
      - --objstore.config-file=/etc/thanos/bucket.yml
    volumes:
      - thanos-store-data:/data
      - ./bucket.yml:/etc/thanos/bucket.yml:ro
    depends_on:
      - minio
    networks:
      - prometheus-network
    restart: unless-stopped

  minio:
    image: minio/minio:latest
    container_name: thanos-minio
    command: server /data --console-address ":9001"
    environment:
      - MINIO_ROOT_USER=${MINIO_ACCESS_KEY}
      - MINIO_ROOT_PASSWORD=${MINIO_SECRET_KEY}
    volumes:
      - minio-data:/data
    ports:
      - "9000:9000"
      - "9001:9001"
    networks:
      - prometheus-network
    restart: unless-stopped

  alertmanager:
    image: prom/alertmanager:latest
    container_name: alertmanager
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
      - alertmanager-data:/alertmanager
    ports:
      - "9093:9093"
    networks:
      - prometheus-network
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
    volumes:
      - grafana-data:/var/lib/grafana
    ports:
      - "3000:3000"
    networks:
      - prometheus-network
    restart: unless-stopped

volumes:
  prometheus1-data:
  prometheus2-data:
  thanos-store-data:
  minio-data:
  alertmanager-data:
  grafana-data:

networks:
  prometheus-network:
    driver: bridge
```

## .env

```bash
# Prometheus HA with Thanos
GRAFANA_PASSWORD=secure_grafana_password
MINIO_ACCESS_KEY=thanos
MINIO_SECRET_KEY=thanos_secret

# Create bucket.yml:
# type: S3
# config:
#   bucket: thanos
#   endpoint: minio:9000
#   access_key: thanos
#   secret_key: thanos_secret
#   insecure: true
```

## Usage notes

- Thanos Query at http://localhost:10902
- Grafana at http://localhost:3000
- Alertmanager at http://localhost:9093
- Long-term storage in MinIO
- Global view across Prometheus instances

## Troubleshooting

- Thanos sidecar 'connection refused' to MinIO: Verify MINIO_ACCESS_KEY and MINIO_SECRET_KEY environment variables match bucket.yml credentials
- Prometheus 'context deadline exceeded' errors: Increase storage.tsdb.retention.time if local disk space is insufficient for 2-hour blocks
- Thanos Query returns no data: Check that all sidecar and store --grpc-address endpoints are correctly specified in query configuration
- MinIO bucket access denied: Ensure the bucket specified in bucket.yml exists and credentials have read/write permissions
- Grafana cannot connect to Thanos Query: Verify Thanos Query is accessible at http://thanos-query:9090 from Grafana container
- High memory usage in Prometheus: Reduce scrape intervals or implement metric relabeling to decrease cardinality of stored time series

