docker.recipes

Grafana Loki Logging Stack

intermediate

Log aggregation with Loki, Promtail collector, Grafana visualization, and alerting.

Overview

Grafana Loki is a horizontally scalable log aggregation system inspired by Prometheus that revolutionizes log management through label-based indexing rather than full-text indexing. Unlike traditional solutions like Elasticsearch, Loki only indexes metadata labels, making it extremely cost-effective for storing large volumes of logs while maintaining fast query performance through its LogQL query language. This approach makes Loki particularly attractive for organizations already using Prometheus, as it extends the same labeling philosophy to log data. This deployment creates a complete logging pipeline with four specialized services: Loki serves as the core log storage engine, Promtail acts as the log collection agent that scrapes logs from the host system and Docker containers, Grafana provides the visualization interface with native Loki integration for log exploration and dashboard creation, and Alertmanager handles alert notifications based on log-derived metrics. The stack is pre-configured with Promtail automatically discovering and shipping logs from /var/log and Docker container logs to Loki, while Grafana comes provisioned with Loki as a datasource. This configuration is ideal for DevOps teams transitioning from expensive commercial logging solutions, organizations seeking unified observability alongside Prometheus metrics, and infrastructure teams needing cost-effective log retention at scale. The combination provides enterprise-grade log aggregation capabilities with the operational simplicity of the Grafana ecosystem, making it perfect for Kubernetes environments, application debugging, security monitoring, and compliance logging requirements.

Key Features

  • Label-based log indexing inspired by Prometheus for efficient storage and querying
  • LogQL query language with grep-like syntax and metric extraction capabilities
  • Promtail agent with automatic service discovery for Docker containers and system logs
  • Native Grafana integration with Explore mode for ad-hoc log investigation
  • Multi-tenant log isolation and retention policies per stream
  • Alertmanager integration for log-based alerting with multiple notification channels
  • Cost-effective storage compared to full-text indexing solutions
  • Horizontal scaling support with configurable storage backends

Common Use Cases

  • 1Kubernetes cluster log aggregation and troubleshooting across multiple namespaces
  • 2Application debugging and error tracking with structured log correlation
  • 3Infrastructure monitoring for system logs, authentication events, and security auditing
  • 4Microservices observability combining metrics and logs in unified Grafana dashboards
  • 5Cost-effective log retention for compliance and regulatory requirements
  • 6DevOps pipeline monitoring for CI/CD build logs and deployment tracking
  • 7Container orchestration platforms needing centralized logging without vendor lock-in

Prerequisites

  • Minimum 2GB RAM (1GB for Loki, 512MB for Grafana, remainder for other services)
  • Docker and Docker Compose with access to Docker socket for container log collection
  • Available ports 3000 (Grafana), 3100 (Loki), and 9093 (Alertmanager)
  • GRAFANA_PASSWORD environment variable set for admin authentication
  • Configuration files: loki-config.yml, promtail-config.yml, grafana-datasources.yml, alertmanager.yml
  • Understanding of LogQL syntax for effective log querying and dashboard creation

For development & testing. Review security settings, change default credentials, and test thoroughly before production use. See Terms

docker-compose.yml

docker-compose.yml
1services: 
2  loki: 
3    image: grafana/loki:latest
4    ports: 
5      - "3100:3100"
6    volumes: 
7      - ./loki-config.yml:/etc/loki/local-config.yaml:ro
8      - loki_data:/loki
9    command: -config.file=/etc/loki/local-config.yaml
10    networks: 
11      - loki-net
12    restart: unless-stopped
13
14  promtail: 
15    image: grafana/promtail:latest
16    volumes: 
17      - ./promtail-config.yml:/etc/promtail/config.yml:ro
18      - /var/log:/var/log:ro
19      - /var/lib/docker/containers:/var/lib/docker/containers:ro
20    command: -config.file=/etc/promtail/config.yml
21    depends_on: 
22      - loki
23    networks: 
24      - loki-net
25    restart: unless-stopped
26
27  grafana: 
28    image: grafana/grafana:latest
29    ports: 
30      - "3000:3000"
31    environment: 
32      GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_PASSWORD}
33      GF_EXPLORE_ENABLED: "true"
34    volumes: 
35      - grafana_data:/var/lib/grafana
36      - ./grafana-datasources.yml:/etc/grafana/provisioning/datasources/ds.yml:ro
37    depends_on: 
38      - loki
39    networks: 
40      - loki-net
41    restart: unless-stopped
42
43  alertmanager: 
44    image: prom/alertmanager:latest
45    ports: 
46      - "9093:9093"
47    volumes: 
48      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
49      - alertmanager_data:/alertmanager
50    networks: 
51      - loki-net
52    restart: unless-stopped
53
54volumes: 
55  loki_data: 
56  grafana_data: 
57  alertmanager_data: 
58
59networks: 
60  loki-net: 
61    driver: bridge

.env Template

.env
1# Grafana
2GRAFANA_PASSWORD=secure_grafana_password
3
4# Loki Retention
5LOKI_RETENTION_PERIOD=168h

Usage Notes

  1. 1Grafana at http://localhost:3000
  2. 2Loki API at http://localhost:3100
  3. 3Use LogQL for querying logs
  4. 4Configure Promtail for log sources

Individual Services(4 services)

Copy individual services to mix and match with your existing compose files.

loki
loki:
  image: grafana/loki:latest
  ports:
    - "3100:3100"
  volumes:
    - ./loki-config.yml:/etc/loki/local-config.yaml:ro
    - loki_data:/loki
  command: "-config.file=/etc/loki/local-config.yaml"
  networks:
    - loki-net
  restart: unless-stopped
promtail
promtail:
  image: grafana/promtail:latest
  volumes:
    - ./promtail-config.yml:/etc/promtail/config.yml:ro
    - /var/log:/var/log:ro
    - /var/lib/docker/containers:/var/lib/docker/containers:ro
  command: "-config.file=/etc/promtail/config.yml"
  depends_on:
    - loki
  networks:
    - loki-net
  restart: unless-stopped
grafana
grafana:
  image: grafana/grafana:latest
  ports:
    - "3000:3000"
  environment:
    GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_PASSWORD}
    GF_EXPLORE_ENABLED: "true"
  volumes:
    - grafana_data:/var/lib/grafana
    - ./grafana-datasources.yml:/etc/grafana/provisioning/datasources/ds.yml:ro
  depends_on:
    - loki
  networks:
    - loki-net
  restart: unless-stopped
alertmanager
alertmanager:
  image: prom/alertmanager:latest
  ports:
    - "9093:9093"
  volumes:
    - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
    - alertmanager_data:/alertmanager
  networks:
    - loki-net
  restart: unless-stopped

Quick Start

terminal
1# 1. Create the compose file
2cat > docker-compose.yml << 'EOF'
3services:
4  loki:
5    image: grafana/loki:latest
6    ports:
7      - "3100:3100"
8    volumes:
9      - ./loki-config.yml:/etc/loki/local-config.yaml:ro
10      - loki_data:/loki
11    command: -config.file=/etc/loki/local-config.yaml
12    networks:
13      - loki-net
14    restart: unless-stopped
15
16  promtail:
17    image: grafana/promtail:latest
18    volumes:
19      - ./promtail-config.yml:/etc/promtail/config.yml:ro
20      - /var/log:/var/log:ro
21      - /var/lib/docker/containers:/var/lib/docker/containers:ro
22    command: -config.file=/etc/promtail/config.yml
23    depends_on:
24      - loki
25    networks:
26      - loki-net
27    restart: unless-stopped
28
29  grafana:
30    image: grafana/grafana:latest
31    ports:
32      - "3000:3000"
33    environment:
34      GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_PASSWORD}
35      GF_EXPLORE_ENABLED: "true"
36    volumes:
37      - grafana_data:/var/lib/grafana
38      - ./grafana-datasources.yml:/etc/grafana/provisioning/datasources/ds.yml:ro
39    depends_on:
40      - loki
41    networks:
42      - loki-net
43    restart: unless-stopped
44
45  alertmanager:
46    image: prom/alertmanager:latest
47    ports:
48      - "9093:9093"
49    volumes:
50      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
51      - alertmanager_data:/alertmanager
52    networks:
53      - loki-net
54    restart: unless-stopped
55
56volumes:
57  loki_data:
58  grafana_data:
59  alertmanager_data:
60
61networks:
62  loki-net:
63    driver: bridge
64EOF
65
66# 2. Create the .env file
67cat > .env << 'EOF'
68# Grafana
69GRAFANA_PASSWORD=secure_grafana_password
70
71# Loki Retention
72LOKI_RETENTION_PERIOD=168h
73EOF
74
75# 3. Start the services
76docker compose up -d
77
78# 4. View logs
79docker compose logs -f

One-Liner

Run this command to download and set up the recipe in one step:

terminal
1curl -fsSL https://docker.recipes/api/recipes/loki-stack-full/run | bash

Troubleshooting

  • Promtail 'permission denied' errors: Ensure Docker containers run with proper permissions to access /var/log and /var/lib/docker/containers volumes
  • Loki startup failure 'failed to create storage': Verify loki-config.yml has correct storage configuration and loki_data volume is writable
  • Grafana shows 'Data source connected but no labels received': Check Promtail is successfully scraping logs and verify Loki datasource URL is http://loki:3100
  • Missing Docker container logs in queries: Confirm Promtail container has access to Docker socket and containers directory with proper mount permissions
  • Alertmanager notifications not sending: Validate alertmanager.yml configuration syntax and ensure network connectivity to notification endpoints
  • LogQL queries timing out or returning no data: Verify log labels match query selectors and check time range covers period when logs were ingested

Community Notes

Loading...
Loading notes...

Download Recipe Kit

Get all files in a ready-to-deploy package

Includes docker-compose.yml, .env template, README, and license

Components

lokipromtailgrafanaalertmanager

Tags

#loki#logging#grafana#promtail#observability

Category

Monitoring & Observability

Related Recipes

Ad Space