Full Observability Stack

advanced

Complete observability with Prometheus, Grafana, Loki, Tempo, and AlertManager

Overview

Prometheus is an open-source monitoring system developed originally by SoundCloud that has become the de facto standard for metrics collection and alerting in cloud-native environments. Built around a dimensional data model and powerful PromQL query language, Prometheus uses a pull-based approach to scrape metrics from targets at regular intervals. This makes it particularly well-suited for monitoring dynamic, containerized environments where services come and go frequently. This comprehensive observability stack combines Prometheus for metrics collection and alerting with Grafana for visualization, Loki for log aggregation, Tempo for distributed tracing, and AlertManager for notification routing. Together, these components provide the three pillars of observability - metrics, logs, and traces - in a unified platform that shares common design principles and integrates natively. The stack includes essential exporters like Node Exporter for host metrics and cAdvisor for container metrics, creating a complete monitoring solution. This configuration is ideal for DevOps teams implementing comprehensive observability for containerized applications, site reliability engineers building monitoring infrastructure, and organizations migrating from proprietary monitoring solutions to open-source alternatives. The stack provides enterprise-grade observability capabilities while maintaining the flexibility to customize and extend based on specific monitoring requirements, making it suitable for everything from small development teams to large-scale production deployments.

Key Features

Pull-based metrics collection with automatic service discovery and multi-dimensional labeling
PromQL query language for complex time-series analysis and alerting rule creation
LogQL integration between Loki and Grafana for correlated metrics and logs analysis
Distributed tracing with Tempo supporting OTLP, Jaeger, and Zipkin protocols
Multi-channel alert routing through AlertManager with grouping, inhibition, and silencing
Native Grafana dashboards with templating, annotations, and explore mode for ad-hoc queries
Label-based log indexing in Loki for cost-effective storage without full-text indexing
Container and host metrics collection via cAdvisor and Node Exporter integration

Common Use Cases

1Kubernetes cluster monitoring with pod, service, and node-level visibility
2Microservices observability with distributed tracing and service dependency mapping
3Infrastructure monitoring for bare metal servers, VMs, and cloud instances
4Application performance monitoring with custom metrics and business KPI tracking
5Cost-effective log aggregation for containerized applications without Elasticsearch overhead
6SRE teams implementing SLI/SLO monitoring with automated alerting workflows
7Development teams requiring local observability stack for testing and debugging

Prerequisites

Docker host with minimum 4GB RAM (Prometheus 1GB+, Grafana 512MB+, Loki 1GB+)
Available ports: 3000 (Grafana), 9090 (Prometheus), 3100 (Loki), 3200 (Tempo), 9093 (AlertManager)
Basic understanding of PromQL for writing queries and creating custom dashboards
Familiarity with YAML configuration for Prometheus scrape configs and Loki log parsing
Knowledge of log parsing patterns and label extraction for Promtail configuration
Understanding of OpenTelemetry standards for instrumenting applications with Tempo tracing

For development & testing. Review security settings, change default credentials, and test thoroughly before production use. See Terms

docker-compose.yml

1services: 
2  prometheus: 
3    image: prom/prometheus:latest
4    container_name: prometheus
5    restart: unless-stopped
6    ports: 
7      - "${PROMETHEUS_PORT:-9090}:9090"
8    command: 
9      - '--config.file=/etc/prometheus/prometheus.yml'
10      - '--storage.tsdb.path=/prometheus'
11      - '--web.enable-lifecycle'
12      - '--web.enable-remote-write-receiver'
13    volumes: 
14      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
15      - ./prometheus/alerts:/etc/prometheus/alerts:ro
16      - prometheus_data:/prometheus
17
18  alertmanager: 
19    image: prom/alertmanager:latest
20    container_name: alertmanager
21    restart: unless-stopped
22    ports: 
23      - "${ALERTMANAGER_PORT:-9093}:9093"
24    volumes: 
25      - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
26      - alertmanager_data:/alertmanager
27
28  grafana: 
29    image: grafana/grafana:latest
30    container_name: grafana
31    restart: unless-stopped
32    ports: 
33      - "${GRAFANA_PORT:-3000}:3000"
34    environment: 
35      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD:-admin}
36      - GF_USERS_ALLOW_SIGN_UP=false
37    volumes: 
38      - grafana_data:/var/lib/grafana
39      - ./grafana/provisioning:/etc/grafana/provisioning:ro
40    depends_on: 
41      - prometheus
42      - loki
43      - tempo
44
45  loki: 
46    image: grafana/loki:latest
47    container_name: loki
48    restart: unless-stopped
49    ports: 
50      - "${LOKI_PORT:-3100}:3100"
51    command: -config.file=/etc/loki/local-config.yaml
52    volumes: 
53      - ./loki/loki-config.yaml:/etc/loki/local-config.yaml:ro
54      - loki_data:/loki
55
56  promtail: 
57    image: grafana/promtail:latest
58    container_name: promtail
59    restart: unless-stopped
60    volumes: 
61      - ./promtail/promtail-config.yaml:/etc/promtail/config.yml:ro
62      - /var/log:/var/log:ro
63      - /var/lib/docker/containers:/var/lib/docker/containers:ro
64    command: -config.file=/etc/promtail/config.yml
65    depends_on: 
66      - loki
67
68  tempo: 
69    image: grafana/tempo:latest
70    container_name: tempo
71    restart: unless-stopped
72    ports: 
73      - "${TEMPO_PORT:-3200}:3200"
74      - "4317:4317"   # OTLP gRPC
75      - "4318:4318"   # OTLP HTTP
76    command: -config.file=/etc/tempo/tempo.yaml
77    volumes: 
78      - ./tempo/tempo.yaml:/etc/tempo/tempo.yaml:ro
79      - tempo_data:/var/tempo
80
81  node-exporter: 
82    image: prom/node-exporter:latest
83    container_name: node-exporter
84    restart: unless-stopped
85    ports: 
86      - "${NODE_EXPORTER_PORT:-9100}:9100"
87    volumes: 
88      - /proc:/host/proc:ro
89      - /sys:/host/sys:ro
90      - /:/rootfs:ro
91    command: 
92      - '--path.procfs=/host/proc'
93      - '--path.sysfs=/host/sys'
94      - '--path.rootfs=/rootfs'
95
96  cadvisor: 
97    image: gcr.io/cadvisor/cadvisor:latest
98    container_name: cadvisor
99    restart: unless-stopped
100    ports: 
101      - "${CADVISOR_PORT:-8080}:8080"
102    volumes: 
103      - /:/rootfs:ro
104      - /var/run:/var/run:ro
105      - /sys:/sys:ro
106      - /var/lib/docker/:/var/lib/docker:ro
107
108volumes: 
109  prometheus_data: 
110  alertmanager_data: 
111  grafana_data: 
112  loki_data: 
113  tempo_data:

.env Template

.env

1# Full Observability Stack
2PROMETHEUS_PORT=9090
3ALERTMANAGER_PORT=9093
4GRAFANA_PORT=3000
5LOKI_PORT=3100
6TEMPO_PORT=3200
7NODE_EXPORTER_PORT=9100
8CADVISOR_PORT=8080
9
10# Grafana
11GRAFANA_PASSWORD=admin

Usage Notes

1Grafana at http://localhost:3000 (admin/admin)
2Prometheus at http://localhost:9090
3Loki for logs, Tempo for traces
4Create prometheus.yml, loki-config.yaml, tempo.yaml configs
5Node Exporter collects host metrics
6cAdvisor collects container metrics

Individual Services(8 services)

Copy individual services to mix and match with your existing compose files.

prometheus

prometheus:
  image: prom/prometheus:latest
  container_name: prometheus
  restart: unless-stopped
  ports:
    - ${PROMETHEUS_PORT:-9090}:9090
  command:
    - "--config.file=/etc/prometheus/prometheus.yml"
    - "--storage.tsdb.path=/prometheus"
    - "--web.enable-lifecycle"
    - "--web.enable-remote-write-receiver"
  volumes:
    - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
    - ./prometheus/alerts:/etc/prometheus/alerts:ro
    - prometheus_data:/prometheus

alertmanager

alertmanager:
  image: prom/alertmanager:latest
  container_name: alertmanager
  restart: unless-stopped
  ports:
    - ${ALERTMANAGER_PORT:-9093}:9093
  volumes:
    - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
    - alertmanager_data:/alertmanager

grafana

grafana:
  image: grafana/grafana:latest
  container_name: grafana
  restart: unless-stopped
  ports:
    - ${GRAFANA_PORT:-3000}:3000
  environment:
    - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD:-admin}
    - GF_USERS_ALLOW_SIGN_UP=false
  volumes:
    - grafana_data:/var/lib/grafana
    - ./grafana/provisioning:/etc/grafana/provisioning:ro
  depends_on:
    - prometheus
    - loki
    - tempo

loki

loki:
  image: grafana/loki:latest
  container_name: loki
  restart: unless-stopped
  ports:
    - ${LOKI_PORT:-3100}:3100
  command: "-config.file=/etc/loki/local-config.yaml"
  volumes:
    - ./loki/loki-config.yaml:/etc/loki/local-config.yaml:ro
    - loki_data:/loki

promtail

promtail:
  image: grafana/promtail:latest
  container_name: promtail
  restart: unless-stopped
  volumes:
    - ./promtail/promtail-config.yaml:/etc/promtail/config.yml:ro
    - /var/log:/var/log:ro
    - /var/lib/docker/containers:/var/lib/docker/containers:ro
  command: "-config.file=/etc/promtail/config.yml"
  depends_on:
    - loki

tempo

tempo:
  image: grafana/tempo:latest
  container_name: tempo
  restart: unless-stopped
  ports:
    - ${TEMPO_PORT:-3200}:3200
    - "4317:4317"
    - "4318:4318"
  command: "-config.file=/etc/tempo/tempo.yaml"
  volumes:
    - ./tempo/tempo.yaml:/etc/tempo/tempo.yaml:ro
    - tempo_data:/var/tempo

node-exporter

node-exporter:
  image: prom/node-exporter:latest
  container_name: node-exporter
  restart: unless-stopped
  ports:
    - ${NODE_EXPORTER_PORT:-9100}:9100
  volumes:
    - /proc:/host/proc:ro
    - /sys:/host/sys:ro
    - /:/rootfs:ro
  command:
    - "--path.procfs=/host/proc"
    - "--path.sysfs=/host/sys"
    - "--path.rootfs=/rootfs"

cadvisor

cadvisor:
  image: gcr.io/cadvisor/cadvisor:latest
  container_name: cadvisor
  restart: unless-stopped
  ports:
    - ${CADVISOR_PORT:-8080}:8080
  volumes:
    - /:/rootfs:ro
    - /var/run:/var/run:ro
    - /sys:/sys:ro
    - /var/lib/docker/:/var/lib/docker:ro

Quick Start

terminal

1# 1. Create the compose file
2cat > docker-compose.yml << 'EOF'
3services:
4  prometheus:
5    image: prom/prometheus:latest
6    container_name: prometheus
7    restart: unless-stopped
8    ports:
9      - "${PROMETHEUS_PORT:-9090}:9090"
10    command:
11      - '--config.file=/etc/prometheus/prometheus.yml'
12      - '--storage.tsdb.path=/prometheus'
13      - '--web.enable-lifecycle'
14      - '--web.enable-remote-write-receiver'
15    volumes:
16      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
17      - ./prometheus/alerts:/etc/prometheus/alerts:ro
18      - prometheus_data:/prometheus
19
20  alertmanager:
21    image: prom/alertmanager:latest
22    container_name: alertmanager
23    restart: unless-stopped
24    ports:
25      - "${ALERTMANAGER_PORT:-9093}:9093"
26    volumes:
27      - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
28      - alertmanager_data:/alertmanager
29
30  grafana:
31    image: grafana/grafana:latest
32    container_name: grafana
33    restart: unless-stopped
34    ports:
35      - "${GRAFANA_PORT:-3000}:3000"
36    environment:
37      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD:-admin}
38      - GF_USERS_ALLOW_SIGN_UP=false
39    volumes:
40      - grafana_data:/var/lib/grafana
41      - ./grafana/provisioning:/etc/grafana/provisioning:ro
42    depends_on:
43      - prometheus
44      - loki
45      - tempo
46
47  loki:
48    image: grafana/loki:latest
49    container_name: loki
50    restart: unless-stopped
51    ports:
52      - "${LOKI_PORT:-3100}:3100"
53    command: -config.file=/etc/loki/local-config.yaml
54    volumes:
55      - ./loki/loki-config.yaml:/etc/loki/local-config.yaml:ro
56      - loki_data:/loki
57
58  promtail:
59    image: grafana/promtail:latest
60    container_name: promtail
61    restart: unless-stopped
62    volumes:
63      - ./promtail/promtail-config.yaml:/etc/promtail/config.yml:ro
64      - /var/log:/var/log:ro
65      - /var/lib/docker/containers:/var/lib/docker/containers:ro
66    command: -config.file=/etc/promtail/config.yml
67    depends_on:
68      - loki
69
70  tempo:
71    image: grafana/tempo:latest
72    container_name: tempo
73    restart: unless-stopped
74    ports:
75      - "${TEMPO_PORT:-3200}:3200"
76      - "4317:4317"   # OTLP gRPC
77      - "4318:4318"   # OTLP HTTP
78    command: -config.file=/etc/tempo/tempo.yaml
79    volumes:
80      - ./tempo/tempo.yaml:/etc/tempo/tempo.yaml:ro
81      - tempo_data:/var/tempo
82
83  node-exporter:
84    image: prom/node-exporter:latest
85    container_name: node-exporter
86    restart: unless-stopped
87    ports:
88      - "${NODE_EXPORTER_PORT:-9100}:9100"
89    volumes:
90      - /proc:/host/proc:ro
91      - /sys:/host/sys:ro
92      - /:/rootfs:ro
93    command:
94      - '--path.procfs=/host/proc'
95      - '--path.sysfs=/host/sys'
96      - '--path.rootfs=/rootfs'
97
98  cadvisor:
99    image: gcr.io/cadvisor/cadvisor:latest
100    container_name: cadvisor
101    restart: unless-stopped
102    ports:
103      - "${CADVISOR_PORT:-8080}:8080"
104    volumes:
105      - /:/rootfs:ro
106      - /var/run:/var/run:ro
107      - /sys:/sys:ro
108      - /var/lib/docker/:/var/lib/docker:ro
109
110volumes:
111  prometheus_data:
112  alertmanager_data:
113  grafana_data:
114  loki_data:
115  tempo_data:
116EOF
117
118# 2. Create the .env file
119cat > .env << 'EOF'
120# Full Observability Stack
121PROMETHEUS_PORT=9090
122ALERTMANAGER_PORT=9093
123GRAFANA_PORT=3000
124LOKI_PORT=3100
125TEMPO_PORT=3200
126NODE_EXPORTER_PORT=9100
127CADVISOR_PORT=8080
128
129# Grafana
130GRAFANA_PASSWORD=admin
131EOF
132
133# 3. Start the services
134docker compose up -d
135
136# 4. View logs
137docker compose logs -f

One-Liner

Run this command to download and set up the recipe in one step:

terminal

1curl -fsSL https://docker.recipes/api/recipes/full-observability-stack/run | bash

Troubleshooting

Prometheus 'context deadline exceeded' errors: Increase scrape timeout and interval in prometheus.yml configuration
Loki 'entry out of order' errors: Enable accept_out_of_order_writes in Loki config or fix Promtail timestamp parsing
Grafana 'Bad Gateway' when querying Prometheus: Check Prometheus data source URL uses container name, not localhost
High memory usage in Prometheus: Reduce metric retention period or increase storage.tsdb.retention.time parameter
Tempo traces not appearing: Verify OTLP endpoints 4317/4318 are accessible and applications use correct trace headers
AlertManager notifications not sending: Check alertmanager.yml routing rules and ensure webhook URLs are accessible from container

Community Notes

Loading notes...

Download Recipe Kit

Get all files in a ready-to-deploy package

Includes docker-compose.yml, .env template, README, and license

Components

prometheusgrafanalokitempoalertmanagerpromtail

Related Recipes

Ad Space

Shortcuts: C CopyF FavoriteD Download