# Grafana Loki Logging Stack

> Grafana Loki logging stack with Promtail for log aggregation and visualization.

- Category: Monitoring & Observability
- Difficulty: intermediate
- Components: loki, grafana, promtail
- Tags: loki, grafana, promtail, logging, observability
- Source: https://docker.recipes/monitoring/grafana-loki-promtail

## Overview

Grafana Loki is a horizontally scalable log aggregation system inspired by Prometheus, designed to efficiently store and query logs using label-based indexing rather than full-text search. Unlike traditional logging solutions that index entire log content, Loki indexes only metadata labels, making it significantly more cost-effective while providing powerful querying capabilities through its LogQL query language. This approach makes Loki particularly attractive for organizations already using Prometheus for metrics, as it follows similar labeling principles and integrates natively with Grafana.

This deployment creates a complete logging stack with three interconnected services: Loki as the central log storage engine, Promtail as the log collection agent, and Grafana for visualization and querying. Promtail automatically discovers and ships logs from the host system and Docker containers to Loki, which stores them efficiently using its unique indexing strategy. Grafana connects directly to Loki as a data source, providing rich dashboards and real-time log exploration capabilities through its Explore interface.

This stack is ideal for teams wanting to implement centralized logging without the complexity and cost of Elasticsearch-based solutions. It's particularly valuable for Kubernetes environments, development teams needing quick log analysis, and organizations building comprehensive observability platforms. The combination provides enterprise-grade log aggregation capabilities while maintaining simplicity in deployment and operation, making it accessible to both small teams and large-scale production environments.

## Features

- Label-based log indexing inspired by Prometheus for efficient storage and querying
- LogQL query language for powerful log filtering, parsing, and aggregation
- Promtail agent with automatic Docker container and system log discovery
- Native Grafana integration with built-in Loki data source support
- Multi-tenant architecture supporting isolated log streams per tenant or environment
- Cost-effective storage model that doesn't require full-text indexing
- Real-time log streaming and alerting capabilities through Grafana
- Horizontal scalability with support for object storage backends like S3

## Use cases

- Kubernetes cluster log aggregation and monitoring across multiple namespaces
- Application troubleshooting and debugging with correlated metrics and logs in Grafana
- Docker container log centralization for microservices architectures
- Development environment log analysis without expensive enterprise tools
- Infrastructure monitoring combining system logs with Prometheus metrics
- Security incident investigation using label-based log filtering and correlation
- Multi-environment log management with tenant isolation between dev, staging, and production

## Prerequisites

- Docker host with at least 2GB RAM (1GB+ for Loki, 512MB+ for Grafana, 256MB for Promtail)
- Available ports 3000 (Grafana) and 3100 (Loki) on the host system
- Promtail configuration file (promtail-config.yaml) defining log sources and Loki endpoint
- Environment variables GF_ADMIN_USER and GF_ADMIN_PASSWORD for Grafana authentication
- Basic understanding of LogQL syntax for effective log querying and dashboard creation
- Read access to /var/log and Docker container logs directory for Promtail collection

## docker-compose.yml

```yaml
services:
  loki:
    image: grafana/loki:2.9.0
    container_name: loki
    ports:
      - "3100:3100"
    volumes:
      - loki_data:/loki
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - loki-network

  promtail:
    image: grafana/promtail:2.9.0
    container_name: promtail
    volumes:
      - /var/log:/var/log:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - ./promtail-config.yaml:/etc/promtail/config.yml:ro
    command: -config.file=/etc/promtail/config.yml
    networks:
      - loki-network

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    environment:
      - GF_SECURITY_ADMIN_USER=${GF_ADMIN_USER}
      - GF_SECURITY_ADMIN_PASSWORD=${GF_ADMIN_PASSWORD}
    ports:
      - "3000:3000"
    volumes:
      - grafana_data:/var/lib/grafana
    depends_on:
      - loki
    networks:
      - loki-network

volumes:
  loki_data:
  grafana_data:

networks:
  loki-network:
    driver: bridge
```

## .env

```bash
# Grafana Loki Stack
GF_ADMIN_USER=admin
GF_ADMIN_PASSWORD=admin123
```

## Usage notes

- Grafana at http://localhost:3000
- Add Loki as datasource (http://loki:3100)
- Create promtail-config.yaml
- Label-based log querying
- LogQL for queries

## Troubleshooting

- Promtail 'no such file or directory' error: Ensure promtail-config.yaml exists and verify volume mount paths match your system
- Loki connection refused in Grafana: Check that Loki service is running and use http://loki:3100 as the data source URL
- No logs appearing in Grafana: Verify Promtail configuration targets correct log paths and Loki endpoint is reachable
- Grafana login fails: Check GF_ADMIN_USER and GF_ADMIN_PASSWORD environment variables are properly set
- High memory usage in Loki: Adjust retention policies and consider using object storage for older logs
- LogQL queries timing out: Use more specific label filters to reduce query scope and improve performance

