Fluentd

intermediate

Unified logging layer for collecting and forwarding logs.

Overview

Fluentd is an open-source data collector and log aggregation system developed by Treasure Data and now a graduated Cloud Native Computing Foundation (CNCF) project. As a unified logging layer, Fluentd excels at collecting, processing, and forwarding log data from multiple sources to various destinations, supporting over 500 plugins for different inputs, outputs, and filters. Its flexible architecture and lightweight footprint make it ideal for containerized environments where log centralization and processing are critical. This Docker configuration establishes Fluentd as a centralized log collection hub, listening on the standard forward input port 24224 for both TCP and UDP traffic. The setup enables seamless integration with Docker's native logging driver, allowing containers across your infrastructure to send logs directly to Fluentd for processing, filtering, and forwarding to downstream systems like Elasticsearch, S3, or monitoring platforms. This stack is particularly valuable for DevOps teams managing microservices architectures, platform engineers implementing observability solutions, and organizations requiring compliance logging with structured data processing. Fluentd's plugin ecosystem and configuration flexibility make it an excellent choice for environments needing sophisticated log routing, transformation, and multi-destination forwarding capabilities.

Key Features

Forward input protocol support on port 24224 for Docker logging driver integration
Extensive plugin ecosystem with 500+ community-maintained input, output, and filter plugins
Memory and file-based buffering with automatic retry mechanisms for reliable log delivery
Built-in log parsing and transformation capabilities using regex, JSON, and custom formats
Tag-based routing system for sophisticated log classification and destination mapping
Zero-downtime configuration reloading via SIGHUP signal handling
Multi-format output support including JSON, MessagePack, and custom serialization
Automatic log rotation and compression for persistent storage management

Common Use Cases

1Centralized logging for Docker Swarm or Kubernetes clusters using native logging drivers
2Application log aggregation from multiple microservices with tag-based routing
3Compliance logging with structured data transformation and secure forwarding to SIEM systems
4Multi-cloud log forwarding from hybrid infrastructure to different storage backends
5Real-time log processing and alerting by filtering critical events before forwarding
6Development environment log debugging with local file output and structured formatting
7IoT device log collection and processing with lightweight footprint requirements

Prerequisites

Minimum 512MB RAM allocation for basic log processing workloads
Available ports 24224 (TCP/UDP) for forward input protocol communication
Pre-configured fluent.conf file in ./fluentd/ directory with input, output, and routing rules
Understanding of Fluentd configuration syntax and plugin architecture
Knowledge of target output destinations (Elasticsearch, S3, etc.) and their connection parameters
Docker logging driver configuration for containers sending logs to Fluentd

For development & testing. Review security settings, change default credentials, and test thoroughly before production use. See Terms

docker-compose.yml

1services: 
2  fluentd: 
3    image: fluent/fluentd:edge
4    container_name: fluentd
5    restart: unless-stopped
6    volumes: 
7      - ./fluentd/fluent.conf:/fluentd/etc/fluent.conf:ro
8      - fluentd_logs:/fluentd/log
9    ports: 
10      - "24224:24224"
11      - "24224:24224/udp"
12    networks: 
13      - fluentd-network
14
15volumes: 
16  fluentd_logs: 
17
18networks: 
19  fluentd-network: 
20    driver: bridge

.env Template

.env

1# Fluentd configuration file required

Usage Notes

1Docs: https://docs.fluentd.org/
2Forward input on port 24224 (TCP/UDP)
3Create ./fluentd/fluent.conf configuration before starting
4Docker logging driver: --log-driver=fluentd --log-opt fluentd-address=localhost:24224
5Rich plugin ecosystem: 500+ input/output/filter plugins
6CNCF graduated project - production ready for enterprise

Quick Start

terminal

1# 1. Create the compose file
2cat > docker-compose.yml << 'EOF'
3services:
4  fluentd:
5    image: fluent/fluentd:edge
6    container_name: fluentd
7    restart: unless-stopped
8    volumes:
9      - ./fluentd/fluent.conf:/fluentd/etc/fluent.conf:ro
10      - fluentd_logs:/fluentd/log
11    ports:
12      - "24224:24224"
13      - "24224:24224/udp"
14    networks:
15      - fluentd-network
16
17volumes:
18  fluentd_logs:
19
20networks:
21  fluentd-network:
22    driver: bridge
23EOF
24
25# 2. Create the .env file
26cat > .env << 'EOF'
27# Fluentd configuration file required
28EOF
29
30# 3. Start the services
31docker compose up -d
32
33# 4. View logs
34docker compose logs -f

One-Liner

Run this command to download and set up the recipe in one step:

terminal

1curl -fsSL https://docker.recipes/api/recipes/fluentd/run | bash

Troubleshooting

No route matched warning messages: Add catch-all match directive with relabel or null output plugin in fluent.conf
Buffer overflow and dropped logs: Increase buffer_chunk_limit and buffer_queue_limit in output plugin configuration
Permission denied writing to log files: Ensure fluentd container runs with appropriate user permissions or adjust volume mount ownership
Plugin not found errors: Install required plugins using fluent-gem or create custom Docker image with pre-installed plugins
High memory usage during log bursts: Configure file-based buffering instead of memory buffering in output sections
Docker containers not sending logs: Verify logging driver configuration and ensure fluentd-address points to correct host and port

Community Notes

Loading notes...

Download Recipe Kit

Get all files in a ready-to-deploy package

Includes docker-compose.yml, .env template, README, and license

Components

fluentd

Related Recipes

Ad Space

Shortcuts: C CopyF FavoriteD Download