# Grafana Loki Logging Stack

> Log aggregation with Loki, Promtail collector, Grafana visualization, and alerting.

- Category: Monitoring & Observability
- Difficulty: intermediate
- Components: loki, promtail, grafana, alertmanager
- Tags: loki, logging, grafana, promtail, observability
- Source: https://docker.recipes/monitoring/loki-stack-full

## Overview

Grafana Loki is a horizontally scalable log aggregation system inspired by Prometheus that revolutionizes log management through label-based indexing rather than full-text indexing. Unlike traditional solutions like Elasticsearch, Loki only indexes metadata labels, making it extremely cost-effective for storing large volumes of logs while maintaining fast query performance through its LogQL query language. This approach makes Loki particularly attractive for organizations already using Prometheus, as it extends the same labeling philosophy to log data.

This deployment creates a complete logging pipeline with four specialized services: Loki serves as the core log storage engine, Promtail acts as the log collection agent that scrapes logs from the host system and Docker containers, Grafana provides the visualization interface with native Loki integration for log exploration and dashboard creation, and Alertmanager handles alert notifications based on log-derived metrics. The stack is pre-configured with Promtail automatically discovering and shipping logs from /var/log and Docker container logs to Loki, while Grafana comes provisioned with Loki as a datasource.

This configuration is ideal for DevOps teams transitioning from expensive commercial logging solutions, organizations seeking unified observability alongside Prometheus metrics, and infrastructure teams needing cost-effective log retention at scale. The combination provides enterprise-grade log aggregation capabilities with the operational simplicity of the Grafana ecosystem, making it perfect for Kubernetes environments, application debugging, security monitoring, and compliance logging requirements.

## Features

- Label-based log indexing inspired by Prometheus for efficient storage and querying
- LogQL query language with grep-like syntax and metric extraction capabilities
- Promtail agent with automatic service discovery for Docker containers and system logs
- Native Grafana integration with Explore mode for ad-hoc log investigation
- Multi-tenant log isolation and retention policies per stream
- Alertmanager integration for log-based alerting with multiple notification channels
- Cost-effective storage compared to full-text indexing solutions
- Horizontal scaling support with configurable storage backends

## Use cases

- Kubernetes cluster log aggregation and troubleshooting across multiple namespaces
- Application debugging and error tracking with structured log correlation
- Infrastructure monitoring for system logs, authentication events, and security auditing
- Microservices observability combining metrics and logs in unified Grafana dashboards
- Cost-effective log retention for compliance and regulatory requirements
- DevOps pipeline monitoring for CI/CD build logs and deployment tracking
- Container orchestration platforms needing centralized logging without vendor lock-in

## Prerequisites

- Minimum 2GB RAM (1GB for Loki, 512MB for Grafana, remainder for other services)
- Docker and Docker Compose with access to Docker socket for container log collection
- Available ports 3000 (Grafana), 3100 (Loki), and 9093 (Alertmanager)
- GRAFANA_PASSWORD environment variable set for admin authentication
- Configuration files: loki-config.yml, promtail-config.yml, grafana-datasources.yml, alertmanager.yml
- Understanding of LogQL syntax for effective log querying and dashboard creation

## docker-compose.yml

```yaml
services:
  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yml:/etc/loki/local-config.yaml:ro
      - loki_data:/loki
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - loki-net
    restart: unless-stopped

  promtail:
    image: grafana/promtail:latest
    volumes:
      - ./promtail-config.yml:/etc/promtail/config.yml:ro
      - /var/log:/var/log:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
    command: -config.file=/etc/promtail/config.yml
    depends_on:
      - loki
    networks:
      - loki-net
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_PASSWORD}
      GF_EXPLORE_ENABLED: "true"
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana-datasources.yml:/etc/grafana/provisioning/datasources/ds.yml:ro
    depends_on:
      - loki
    networks:
      - loki-net
    restart: unless-stopped

  alertmanager:
    image: prom/alertmanager:latest
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
      - alertmanager_data:/alertmanager
    networks:
      - loki-net
    restart: unless-stopped

volumes:
  loki_data:
  grafana_data:
  alertmanager_data:

networks:
  loki-net:
    driver: bridge
```

## .env

```bash
# Grafana
GRAFANA_PASSWORD=secure_grafana_password

# Loki Retention
LOKI_RETENTION_PERIOD=168h
```

## Usage notes

- Grafana at http://localhost:3000
- Loki API at http://localhost:3100
- Use LogQL for querying logs
- Configure Promtail for log sources

## Troubleshooting

- Promtail 'permission denied' errors: Ensure Docker containers run with proper permissions to access /var/log and /var/lib/docker/containers volumes
- Loki startup failure 'failed to create storage': Verify loki-config.yml has correct storage configuration and loki_data volume is writable
- Grafana shows 'Data source connected but no labels received': Check Promtail is successfully scraping logs and verify Loki datasource URL is http://loki:3100
- Missing Docker container logs in queries: Confirm Promtail container has access to Docker socket and containers directory with proper mount permissions
- Alertmanager notifications not sending: Validate alertmanager.yml configuration syntax and ensure network connectivity to notification endpoints
- LogQL queries timing out or returning no data: Verify log labels match query selectors and check time range covers period when logs were ingested

