# Graylog Log Management

> Centralized log management and analysis platform.

- Category: Monitoring & Observability
- Difficulty: intermediate
- Components: graylog, mongodb, elasticsearch
- Tags: logging, graylog, log-management, siem, analysis
- Source: https://docker.recipes/monitoring/graylog-complete

## Overview

Graylog is an enterprise-grade open-source log management platform that transforms raw log data into actionable insights through centralized collection, indexing, and analysis. Originally developed by Lennart Koopmann in 2010, Graylog has evolved into a comprehensive Security Information and Event Management (SIEM) solution that competes with commercial offerings like Splunk and IBM QRadar. Built on a modern microservices architecture, Graylog excels at processing millions of log messages per day while providing real-time alerting, dashboard visualization, and powerful search capabilities that help organizations maintain security posture and operational visibility.

This stack combines Graylog's log processing engine with Elasticsearch's distributed search capabilities and MongoDB's flexible document storage to create a robust three-tier logging infrastructure. Elasticsearch serves as the primary search index for log data, enabling sub-second queries across terabytes of historical logs, while MongoDB stores Graylog's configuration metadata, user settings, and system state. The architecture separates concerns effectively: MongoDB handles transactional configuration data, Elasticsearch optimizes for search and analytics workloads, and Graylog orchestrates the entire pipeline from log ingestion through visualization.

System administrators managing multi-service architectures, security teams implementing SIEM capabilities, and DevOps engineers troubleshooting distributed applications will find this stack invaluable for consolidating disparate log sources into a unified analysis platform. Unlike simple log aggregation tools, this combination provides advanced features like stream processing, field extraction rules, and correlation capabilities that transform basic logging into comprehensive operational intelligence and security monitoring.

## Features

- Multi-protocol log ingestion supporting Syslog, GELF, Beats, and custom HTTP inputs with automatic parsing
- Real-time stream processing with configurable rules for field extraction, message routing, and data enrichment
- Advanced search interface with Lucene query syntax, saved searches, and collaborative dashboard sharing
- Flexible alerting system with email, Slack, and webhook notifications based on log patterns and thresholds
- Role-based access control with granular permissions for streams, dashboards, and administrative functions
- Elasticsearch integration optimized for log data with automated index rotation and retention policies
- MongoDB-backed configuration management ensuring consistent settings across Graylog cluster nodes
- Plugin ecosystem supporting custom inputs, outputs, and processing functions for specialized log formats

## Use cases

- Centralized logging infrastructure for microservices architectures with automatic service discovery and log correlation
- Security Operations Center (SOC) implementation with threat detection, incident response, and compliance reporting
- Application performance monitoring through structured logging analysis and error pattern detection
- Infrastructure monitoring combining system logs, network device logs, and container orchestration events
- Compliance audit trail maintenance for regulations requiring log retention and tamper-proof storage
- Development and staging environment debugging with real-time log streaming and search capabilities
- IoT device fleet monitoring with high-volume message processing and anomaly detection

## Prerequisites

- Minimum 6GB RAM allocation (2GB for Elasticsearch, 2GB for Graylog, 1GB for MongoDB, 1GB system overhead)
- Generated password secret (96+ character random string) and SHA-256 hashed root password for Graylog authentication
- Available ports 9000 (web UI), 1514 (Syslog), and 12201 (GELF) without conflicts from existing services
- Understanding of log formats (Syslog RFC, JSON, GELF) and network protocols for input configuration
- Basic knowledge of Elasticsearch concepts like indices, shards, and mapping for troubleshooting storage issues
- Familiarity with regular expressions and Grok patterns for custom log parsing and field extraction

## docker-compose.yml

```yaml
services:
  graylog:
    image: graylog/graylog:5.2
    container_name: graylog
    environment:
      - GRAYLOG_PASSWORD_SECRET=${PASSWORD_SECRET}
      - GRAYLOG_ROOT_PASSWORD_SHA2=${ROOT_PASSWORD_SHA2}
      - GRAYLOG_HTTP_EXTERNAL_URI=${EXTERNAL_URI}
      - GRAYLOG_ELASTICSEARCH_HOSTS=http://elasticsearch:9200
      - GRAYLOG_MONGODB_URI=mongodb://mongodb:27017/graylog
    volumes:
      - graylog-data:/usr/share/graylog/data
    ports:
      - "9000:9000"
      - "1514:1514"
      - "1514:1514/udp"
      - "12201:12201"
      - "12201:12201/udp"
    depends_on:
      - mongodb
      - elasticsearch
    networks:
      - graylog-network
    restart: unless-stopped

  mongodb:
    image: mongo:6
    container_name: graylog-mongodb
    volumes:
      - mongo-data:/data/db
    networks:
      - graylog-network
    restart: unless-stopped

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.15
    container_name: graylog-elasticsearch
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    volumes:
      - elasticsearch-data:/usr/share/elasticsearch/data
    networks:
      - graylog-network
    restart: unless-stopped

volumes:
  graylog-data:
  mongo-data:
  elasticsearch-data:

networks:
  graylog-network:
    driver: bridge
```

## .env

```bash
# Graylog
EXTERNAL_URI=http://localhost:9000/

# Generate with: pwgen -N 1 -s 96
PASSWORD_SECRET=your_password_secret_at_least_16_chars

# Generate with: echo -n "yourpassword" | sha256sum | cut -d" " -f1
ROOT_PASSWORD_SHA2=your_sha256_password_hash
```

## Usage notes

- Graylog UI at http://localhost:9000
- Login: admin / (your password)
- Syslog input on :1514
- GELF input on :12201
- Create inputs in System > Inputs

## Troubleshooting

- Graylog shows 'Elasticsearch cluster unavailable': Check Elasticsearch container health and network connectivity, verify ES_JAVA_OPTS memory allocation doesn't exceed container limits
- MongoDB connection failures during startup: Ensure MongoDB container starts before Graylog, check GRAYLOG_MONGODB_URI format and network resolution between containers
- Web UI returns 502 Bad Gateway: Verify GRAYLOG_HTTP_EXTERNAL_URI matches your access URL, check if Graylog process started successfully with proper environment variables
- No logs appearing in streams: Confirm input ports are properly exposed, check firewall rules, verify log format matches input type expectations (Syslog vs GELF)
- Elasticsearch yellow cluster status: Single-node clusters show yellow by default, add 'cluster.routing.allocation.disk.threshold_enabled: false' to elasticsearch environment if disk space warnings appear
- High memory usage and container crashes: Reduce ES_JAVA_OPTS heap size, implement log rotation policies, monitor Graylog buffer settings to prevent memory overflow

