# Fluentd

> Unified logging layer for collecting and forwarding logs.

- Category: Monitoring & Observability
- Difficulty: intermediate
- Components: fluentd
- Tags: fluentd, logs, collector, forwarding, cncf
- Source: https://docker.recipes/monitoring/fluentd

## Overview

Fluentd is an open-source data collector and log aggregation system developed by Treasure Data and now a graduated Cloud Native Computing Foundation (CNCF) project. As a unified logging layer, Fluentd excels at collecting, processing, and forwarding log data from multiple sources to various destinations, supporting over 500 plugins for different inputs, outputs, and filters. Its flexible architecture and lightweight footprint make it ideal for containerized environments where log centralization and processing are critical.

This Docker configuration establishes Fluentd as a centralized log collection hub, listening on the standard forward input port 24224 for both TCP and UDP traffic. The setup enables seamless integration with Docker's native logging driver, allowing containers across your infrastructure to send logs directly to Fluentd for processing, filtering, and forwarding to downstream systems like Elasticsearch, S3, or monitoring platforms.

This stack is particularly valuable for DevOps teams managing microservices architectures, platform engineers implementing observability solutions, and organizations requiring compliance logging with structured data processing. Fluentd's plugin ecosystem and configuration flexibility make it an excellent choice for environments needing sophisticated log routing, transformation, and multi-destination forwarding capabilities.

## Features

- Forward input protocol support on port 24224 for Docker logging driver integration
- Extensive plugin ecosystem with 500+ community-maintained input, output, and filter plugins
- Memory and file-based buffering with automatic retry mechanisms for reliable log delivery
- Built-in log parsing and transformation capabilities using regex, JSON, and custom formats
- Tag-based routing system for sophisticated log classification and destination mapping
- Zero-downtime configuration reloading via SIGHUP signal handling
- Multi-format output support including JSON, MessagePack, and custom serialization
- Automatic log rotation and compression for persistent storage management

## Use cases

- Centralized logging for Docker Swarm or Kubernetes clusters using native logging drivers
- Application log aggregation from multiple microservices with tag-based routing
- Compliance logging with structured data transformation and secure forwarding to SIEM systems
- Multi-cloud log forwarding from hybrid infrastructure to different storage backends
- Real-time log processing and alerting by filtering critical events before forwarding
- Development environment log debugging with local file output and structured formatting
- IoT device log collection and processing with lightweight footprint requirements

## Prerequisites

- Minimum 512MB RAM allocation for basic log processing workloads
- Available ports 24224 (TCP/UDP) for forward input protocol communication
- Pre-configured fluent.conf file in ./fluentd/ directory with input, output, and routing rules
- Understanding of Fluentd configuration syntax and plugin architecture
- Knowledge of target output destinations (Elasticsearch, S3, etc.) and their connection parameters
- Docker logging driver configuration for containers sending logs to Fluentd

## docker-compose.yml

```yaml
services:
  fluentd:
    image: fluent/fluentd:edge
    container_name: fluentd
    restart: unless-stopped
    volumes:
      - ./fluentd/fluent.conf:/fluentd/etc/fluent.conf:ro
      - fluentd_logs:/fluentd/log
    ports:
      - "24224:24224"
      - "24224:24224/udp"
    networks:
      - fluentd-network

volumes:
  fluentd_logs:

networks:
  fluentd-network:
    driver: bridge
```

## .env

```bash
# Fluentd configuration file required
```

## Usage notes

- Docs: https://docs.fluentd.org/
- Forward input on port 24224 (TCP/UDP)
- Create ./fluentd/fluent.conf configuration before starting
- Docker logging driver: --log-driver=fluentd --log-opt fluentd-address=localhost:24224
- Rich plugin ecosystem: 500+ input/output/filter plugins
- CNCF graduated project - production ready for enterprise

## Troubleshooting

- No route matched warning messages: Add catch-all match directive with relabel or null output plugin in fluent.conf
- Buffer overflow and dropped logs: Increase buffer_chunk_limit and buffer_queue_limit in output plugin configuration
- Permission denied writing to log files: Ensure fluentd container runs with appropriate user permissions or adjust volume mount ownership
- Plugin not found errors: Install required plugins using fluent-gem or create custom Docker image with pre-installed plugins
- High memory usage during log bursts: Configure file-based buffering instead of memory buffering in output sections
- Docker containers not sending logs: Verify logging driver configuration and ensure fluentd-address points to correct host and port

