# Thanos

> Highly available Prometheus setup with long-term storage capabilities.

- Category: Monitoring & Observability
- Difficulty: advanced
- Components: thanos, prometheus, minio
- Tags: thanos, prometheus, ha, long-term-storage, metrics
- Source: https://docker.recipes/monitoring/thanos

## Overview

Thanos is a set of components that can be composed into a highly available metric system with unlimited storage capacity, which can be added on top of existing Prometheus deployments. Originally developed at Improbable, Thanos addresses Prometheus's fundamental limitations around high availability, long-term retention, and global query capabilities by providing a distributed system architecture that leverages object storage for infinite retention and horizontal scalability.

This stack combines Prometheus for metrics collection with Thanos Sidecar and Query components to create a scalable monitoring solution. The Thanos Sidecar runs alongside Prometheus, uploading 2-hour TSDB blocks to object storage (configured for MinIO in the complete setup) while providing real-time access to local data. The Thanos Query component acts as a global query layer, federating data from multiple Prometheus instances and historical data from object storage, presenting a unified PromQL interface that spans across all time ranges and locations.

Platform engineers and SREs managing multi-cluster Kubernetes environments, large-scale microservices, or any infrastructure requiring long-term metrics retention should consider this stack. Unlike standalone Prometheus deployments limited by local storage and single-instance constraints, this Thanos configuration enables true horizontal scaling, cross-datacenter metric queries, and cost-effective long-term storage using object storage backends, making it ideal for organizations outgrowing single Prometheus instances.

## Features

- Global query interface via Thanos Query for federated metrics across multiple Prometheus instances
- Unlimited long-term storage retention using object storage backends like MinIO or cloud storage
- Horizontal scaling of Prometheus deployments without losing query capabilities across instances
- Automatic TSDB block upload to object storage with configurable 2-hour block duration
- PromQL compatibility maintained across historical and real-time data through unified query layer
- High availability setup supporting multiple Prometheus replicas with deduplication
- Cost-effective storage tiering with hot data in Prometheus and cold data in object storage
- Cross-cluster and multi-region metrics federation with single query endpoint

## Use cases

- Multi-cluster Kubernetes monitoring with centralized metrics querying across all clusters
- Long-term capacity planning and trend analysis requiring years of historical metrics data
- Highly available Prometheus setups with multiple replicas and automatic failover capabilities
- Cost optimization for metrics storage by moving historical data to cheaper object storage
- Cross-datacenter monitoring with unified dashboards spanning multiple geographic regions
- Compliance requirements demanding long-term metrics retention for auditing purposes
- Large-scale microservices monitoring where single Prometheus instances hit storage limits

## Prerequisites

- Minimum 2GB RAM available for Prometheus and Thanos components combined
- Object storage backend (MinIO, S3, or GCS) for long-term block storage configured separately
- Ports 9090 (Prometheus), 10901 (Thanos Sidecar), and 10902 (Thanos Query) available
- Understanding of PromQL and Prometheus architecture for effective troubleshooting
- Knowledge of TSDB block structure and compaction processes for storage optimization
- Sufficient disk space for local Prometheus storage before blocks are uploaded to object storage

## docker-compose.yml

```yaml
services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --storage.tsdb.max-block-duration=2h
      - --storage.tsdb.min-block-duration=2h
    volumes:
      - ./prometheus:/etc/prometheus
      - prometheus_data:/prometheus
    ports:
      - "9090:9090"
    networks:
      - thanos-network

  thanos-sidecar:
    image: quay.io/thanos/thanos:latest
    container_name: thanos-sidecar
    command:
      - sidecar
      - --prometheus.url=http://prometheus:9090
      - --tsdb.path=/prometheus
    volumes:
      - prometheus_data:/prometheus
    depends_on:
      - prometheus
    networks:
      - thanos-network

  thanos-query:
    image: quay.io/thanos/thanos:latest
    container_name: thanos-query
    command:
      - query
      - --endpoint=thanos-sidecar:10901
    ports:
      - "10902:10902"
    depends_on:
      - thanos-sidecar
    networks:
      - thanos-network

volumes:
  prometheus_data:

networks:
  thanos-network:
    driver: bridge
```

## .env

```bash
# Thanos configuration
```

## Usage notes

- Docs: https://thanos.io/tip/thanos/getting-started.md/
- Thanos Query UI at http://localhost:10902 - global view of metrics
- Prometheus at http://localhost:9090 - local data only
- Sidecar uploads blocks to object storage for long-term retention
- Add Store Gateway for querying historical data from object storage
- Supports multiple Prometheus instances for HA and federation

## Troubleshooting

- Thanos Sidecar fails to start: Ensure prometheus_data volume has correct permissions and Prometheus is running with TSDB block duration set to 2h
- Query returns incomplete data: Verify Thanos Sidecar is properly connected and blocks are being uploaded to object storage successfully
- High memory usage in Thanos Query: Increase memory limits and check for large query ranges that might be loading too much historical data
- Prometheus blocks not uploading: Check object storage configuration and network connectivity between Sidecar and storage backend
- Thanos Query endpoint timeouts: Verify Sidecar gRPC endpoint is accessible on port 10901 and not blocked by firewall rules
- Duplicate metrics in query results: Configure proper deduplication labels in Thanos Query and ensure Prometheus external labels are set correctly

