# Full Observability Stack

> Complete observability with Prometheus, Grafana, Loki, Tempo, and AlertManager

- Category: Monitoring & Observability
- Difficulty: advanced
- Components: prometheus, grafana, loki, tempo, alertmanager, promtail
- Tags: prometheus, grafana, loki, tempo, alertmanager, observability, metrics, logs, traces
- Source: https://docker.recipes/monitoring/full-observability-stack

## Overview

Prometheus is an open-source monitoring system developed originally by SoundCloud that has become the de facto standard for metrics collection and alerting in cloud-native environments. Built around a dimensional data model and powerful PromQL query language, Prometheus uses a pull-based approach to scrape metrics from targets at regular intervals. This makes it particularly well-suited for monitoring dynamic, containerized environments where services come and go frequently.

This comprehensive observability stack combines Prometheus for metrics collection and alerting with Grafana for visualization, Loki for log aggregation, Tempo for distributed tracing, and AlertManager for notification routing. Together, these components provide the three pillars of observability - metrics, logs, and traces - in a unified platform that shares common design principles and integrates natively. The stack includes essential exporters like Node Exporter for host metrics and cAdvisor for container metrics, creating a complete monitoring solution.

This configuration is ideal for DevOps teams implementing comprehensive observability for containerized applications, site reliability engineers building monitoring infrastructure, and organizations migrating from proprietary monitoring solutions to open-source alternatives. The stack provides enterprise-grade observability capabilities while maintaining the flexibility to customize and extend based on specific monitoring requirements, making it suitable for everything from small development teams to large-scale production deployments.

## Features

- Pull-based metrics collection with automatic service discovery and multi-dimensional labeling
- PromQL query language for complex time-series analysis and alerting rule creation
- LogQL integration between Loki and Grafana for correlated metrics and logs analysis
- Distributed tracing with Tempo supporting OTLP, Jaeger, and Zipkin protocols
- Multi-channel alert routing through AlertManager with grouping, inhibition, and silencing
- Native Grafana dashboards with templating, annotations, and explore mode for ad-hoc queries
- Label-based log indexing in Loki for cost-effective storage without full-text indexing
- Container and host metrics collection via cAdvisor and Node Exporter integration

## Use cases

- Kubernetes cluster monitoring with pod, service, and node-level visibility
- Microservices observability with distributed tracing and service dependency mapping
- Infrastructure monitoring for bare metal servers, VMs, and cloud instances
- Application performance monitoring with custom metrics and business KPI tracking
- Cost-effective log aggregation for containerized applications without Elasticsearch overhead
- SRE teams implementing SLI/SLO monitoring with automated alerting workflows
- Development teams requiring local observability stack for testing and debugging

## Prerequisites

- Docker host with minimum 4GB RAM (Prometheus 1GB+, Grafana 512MB+, Loki 1GB+)
- Available ports: 3000 (Grafana), 9090 (Prometheus), 3100 (Loki), 3200 (Tempo), 9093 (AlertManager)
- Basic understanding of PromQL for writing queries and creating custom dashboards
- Familiarity with YAML configuration for Prometheus scrape configs and Loki log parsing
- Knowledge of log parsing patterns and label extraction for Promtail configuration
- Understanding of OpenTelemetry standards for instrumenting applications with Tempo tracing

## docker-compose.yml

```yaml
services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    ports:
      - "${PROMETHEUS_PORT:-9090}:9090"
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.enable-lifecycle'
      - '--web.enable-remote-write-receiver'
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./prometheus/alerts:/etc/prometheus/alerts:ro
      - prometheus_data:/prometheus

  alertmanager:
    image: prom/alertmanager:latest
    container_name: alertmanager
    restart: unless-stopped
    ports:
      - "${ALERTMANAGER_PORT:-9093}:9093"
    volumes:
      - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
      - alertmanager_data:/alertmanager

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    restart: unless-stopped
    ports:
      - "${GRAFANA_PORT:-3000}:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD:-admin}
      - GF_USERS_ALLOW_SIGN_UP=false
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
    depends_on:
      - prometheus
      - loki
      - tempo

  loki:
    image: grafana/loki:latest
    container_name: loki
    restart: unless-stopped
    ports:
      - "${LOKI_PORT:-3100}:3100"
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - ./loki/loki-config.yaml:/etc/loki/local-config.yaml:ro
      - loki_data:/loki

  promtail:
    image: grafana/promtail:latest
    container_name: promtail
    restart: unless-stopped
    volumes:
      - ./promtail/promtail-config.yaml:/etc/promtail/config.yml:ro
      - /var/log:/var/log:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
    command: -config.file=/etc/promtail/config.yml
    depends_on:
      - loki

  tempo:
    image: grafana/tempo:latest
    container_name: tempo
    restart: unless-stopped
    ports:
      - "${TEMPO_PORT:-3200}:3200"
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP
    command: -config.file=/etc/tempo/tempo.yaml
    volumes:
      - ./tempo/tempo.yaml:/etc/tempo/tempo.yaml:ro
      - tempo_data:/var/tempo

  node-exporter:
    image: prom/node-exporter:latest
    container_name: node-exporter
    restart: unless-stopped
    ports:
      - "${NODE_EXPORTER_PORT:-9100}:9100"
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--path.rootfs=/rootfs'

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    restart: unless-stopped
    ports:
      - "${CADVISOR_PORT:-8080}:8080"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro

volumes:
  prometheus_data:
  alertmanager_data:
  grafana_data:
  loki_data:
  tempo_data:
```

## .env

```bash
# Full Observability Stack
PROMETHEUS_PORT=9090
ALERTMANAGER_PORT=9093
GRAFANA_PORT=3000
LOKI_PORT=3100
TEMPO_PORT=3200
NODE_EXPORTER_PORT=9100
CADVISOR_PORT=8080

# Grafana
GRAFANA_PASSWORD=admin
```

## Usage notes

- Grafana at http://localhost:3000 (admin/admin)
- Prometheus at http://localhost:9090
- Loki for logs, Tempo for traces
- Create prometheus.yml, loki-config.yaml, tempo.yaml configs
- Node Exporter collects host metrics
- cAdvisor collects container metrics

## Troubleshooting

- Prometheus 'context deadline exceeded' errors: Increase scrape timeout and interval in prometheus.yml configuration
- Loki 'entry out of order' errors: Enable accept_out_of_order_writes in Loki config or fix Promtail timestamp parsing
- Grafana 'Bad Gateway' when querying Prometheus: Check Prometheus data source URL uses container name, not localhost
- High memory usage in Prometheus: Reduce metric retention period or increase storage.tsdb.retention.time parameter
- Tempo traces not appearing: Verify OTLP endpoints 4317/4318 are accessible and applications use correct trace headers
- AlertManager notifications not sending: Check alertmanager.yml routing rules and ensure webhook URLs are accessible from container

