# Loki Logging Stack

> Grafana Loki for log aggregation with Promtail and Grafana.

- Category: Monitoring & Observability
- Difficulty: intermediate
- Components: loki, promtail, grafana
- Tags: loki, logging, grafana, promtail, prometheus
- Source: https://docker.recipes/monitoring/loki-logging-stack

## Overview

Grafana Loki is a horizontally scalable, highly available log aggregation system inspired by Prometheus that revolutionizes log management by indexing only metadata rather than full log content. Unlike traditional log aggregation solutions like Elasticsearch that index entire log messages, Loki uses label-based indexing similar to Prometheus metrics, making it significantly more cost-effective for long-term log storage while maintaining powerful query capabilities through its LogQL query language.

This logging stack combines Loki as the central log storage engine, Promtail as the log collection agent, and Grafana for visualization and exploration. Promtail acts as the log shipper that discovers log files on the host system, applies labels based on configured rules, and forwards structured log entries to Loki. Grafana provides the user interface for querying logs using LogQL, creating dashboards that combine both metrics and logs, and setting up alerting rules based on log patterns or volume.

This stack is ideal for teams already using Prometheus and Grafana who want to add centralized logging without the complexity and cost of Elasticsearch-based solutions. It's particularly valuable for Kubernetes environments, cost-conscious organizations that need long-term log retention, and teams that prefer the operational simplicity of the Prometheus ecosystem. The combination provides a unified observability platform where metrics, logs, and traces can be correlated in a single interface.

## Features

- LogQL query language for powerful log filtering and aggregation similar to PromQL
- Label-based indexing that dramatically reduces storage costs compared to full-text indexing
- Native Grafana integration allowing log and metric correlation in unified dashboards
- Promtail automatic service discovery for Docker containers and Kubernetes pods
- Multi-tenant architecture supporting multiple organizations with data isolation
- Horizontal scalability with support for object storage backends like S3, GCS, and Azure
- Real-time log streaming and alerting based on log patterns and volume thresholds
- Compression and chunk-based storage optimization for efficient long-term retention

## Use cases

- Kubernetes cluster log aggregation with automatic pod discovery and labeling
- Cost-effective alternative to Elasticsearch for organizations with high log volumes
- Unified observability platform combining Prometheus metrics with centralized logging
- Application debugging and troubleshooting with structured log queries
- Security monitoring and audit log analysis using label-based filtering
- DevOps teams needing correlation between application performance metrics and logs
- Multi-environment log management for development, staging, and production systems

## Prerequisites

- Minimum 1.5GB RAM available (256MB for Loki, 512MB for Grafana, 256MB for Promtail)
- Docker and Docker Compose v2.0 or higher installed
- Ports 3000 (Grafana) and 3100 (Loki) available on the host system
- Basic understanding of log aggregation concepts and Grafana dashboard creation
- Read access to system log directories (promtail-config.yml must be created)
- Sufficient disk space for log storage (Loki data volume grows with log retention)

## docker-compose.yml

```yaml
services:
  loki:
    image: grafana/loki:latest
    container_name: loki
    restart: unless-stopped
    ports:
      - "${LOKI_PORT:-3100}:3100"
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - loki_data:/loki
    networks:
      - loki-network

  promtail:
    image: grafana/promtail:latest
    container_name: promtail
    restart: unless-stopped
    volumes:
      - /var/log:/var/log:ro
      - ./promtail-config.yml:/etc/promtail/config.yml:ro
    command: -config.file=/etc/promtail/config.yml
    depends_on:
      - loki
    networks:
      - loki-network

  grafana:
    image: grafana/grafana:latest
    container_name: grafana-loki
    restart: unless-stopped
    ports:
      - "${GRAFANA_PORT:-3000}:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD:-admin}
    volumes:
      - grafana_data:/var/lib/grafana
    depends_on:
      - loki
    networks:
      - loki-network

volumes:
  loki_data:
  grafana_data:

networks:
  loki-network:
    driver: bridge
```

## .env

```bash
# Loki Stack
LOKI_PORT=3100
GRAFANA_PORT=3000
GRAFANA_PASSWORD=admin
```

## Usage notes

- Grafana at http://localhost:3000
- Loki at http://localhost:3100
- Add Loki datasource in Grafana
- Create promtail-config.yml

## Troubleshooting

- Promtail shows 'permission denied' errors: Ensure Docker has read access to /var/log or adjust volume mounts to accessible directories
- Loki container exits with 'failed to parse config': Verify the built-in local-config.yaml is not overridden by custom configurations
- Grafana shows 'Bad Gateway' when querying Loki: Check that Loki datasource URL is set to http://loki:3100 using the Docker network name
- No logs appearing in Grafana: Create and mount promtail-config.yml with proper scrape_configs pointing to log file paths
- Loki runs out of disk space: Configure log retention policies in Loki config or set up object storage backend for longer retention
- Promtail not discovering containers: Add Docker socket mount and configure docker_sd_config in promtail configuration

