Fluent Bit

beginner

Lightweight log processor and forwarder optimized for containers.

Overview

Fluent Bit is an open-source log processor and forwarder developed by Treasure Data that specializes in collecting, parsing, and routing logs from various sources to multiple destinations. Originally designed as a lightweight alternative to Fluentd, Fluent Bit consumes minimal system resources while maintaining high performance, making it particularly well-suited for containerized environments, edge computing, and resource-constrained systems. The project emerged from the need for a sub-megabyte log processor that could handle high-throughput log processing without the memory overhead of traditional logging solutions. This Docker configuration creates a centralized log collection point that can receive logs from multiple sources including Docker containers, system logs, and application logs, then forward them to various destinations such as Elasticsearch, InfluxDB, or cloud storage services. Fluent Bit's plugin architecture enables extensive customization for parsing different log formats and routing to specific outputs based on tags or content. DevOps teams managing containerized applications will find this setup invaluable for implementing unified logging strategies across distributed systems. The configuration is particularly beneficial for organizations running microservices architectures, edge computing deployments, or any environment where log aggregation needs to operate with minimal resource consumption while maintaining reliability and performance.

Key Features

Sub-megabyte memory footprint with C-based architecture optimized for performance
Built-in HTTP metrics server on port 2020 providing real-time processing statistics
Fluentd protocol compatibility on port 24224 for drop-in replacement scenarios
Stream processing capabilities with real-time data transformation and filtering
Multi-format log parsing including JSON, regex, and structured data extraction
Plugin ecosystem supporting 50+ input sources and output destinations
Built-in buffering and backpressure handling for reliable log delivery
Native Kubernetes integration with automatic pod metadata enrichment

Common Use Cases

1Kubernetes cluster log aggregation using DaemonSet deployment across all nodes
2IoT and edge device log collection where bandwidth and storage are limited
3Docker container log centralization for microservices monitoring and debugging
4System log forwarding from bare metal servers to cloud logging platforms
5Real-time log filtering and routing based on application tags or severity levels
6Multi-cloud log forwarding to different destinations based on compliance requirements
7Development environment log debugging with local Elasticsearch or file output

Prerequisites

Docker Engine 20.10+ and Docker Compose V2 for container orchestration
Minimum 128MB available RAM for Fluent Bit container operation
Network ports 2020 and 24224 available on the Docker host
Write access to ./fluent-bit directory for configuration file creation
Basic understanding of log routing concepts and output destination configuration
Read access to /var/log directory if collecting system logs from the host

For development & testing. Review security settings, change default credentials, and test thoroughly before production use. See Terms

docker-compose.yml

1services: 
2  fluent-bit: 
3    image: fluent/fluent-bit:latest
4    container_name: fluent-bit
5    restart: unless-stopped
6    volumes: 
7      - ./fluent-bit/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro
8      - /var/log:/var/log:ro
9    ports: 
10      - "2020:2020"
11      - "24224:24224"
12    networks: 
13      - fluentbit-network
14
15networks: 
16  fluentbit-network: 
17    driver: bridge

.env Template

.env

1# Fluent Bit configuration file required

Usage Notes

1Docs: https://docs.fluentbit.io/
2Metrics/health endpoint at http://localhost:2020
3Forward input on port 24224 - compatible with Fluentd
4Create ./fluent-bit/fluent-bit.conf configuration before starting
5Sub-MB memory footprint - ideal for edge/IoT deployments
6Kubernetes DaemonSet recommended for cluster-wide log collection

Quick Start

terminal

1# 1. Create the compose file
2cat > docker-compose.yml << 'EOF'
3services:
4  fluent-bit:
5    image: fluent/fluent-bit:latest
6    container_name: fluent-bit
7    restart: unless-stopped
8    volumes:
9      - ./fluent-bit/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro
10      - /var/log:/var/log:ro
11    ports:
12      - "2020:2020"
13      - "24224:24224"
14    networks:
15      - fluentbit-network
16
17networks:
18  fluentbit-network:
19    driver: bridge
20EOF
21
22# 2. Create the .env file
23cat > .env << 'EOF'
24# Fluent Bit configuration file required
25EOF
26
27# 3. Start the services
28docker compose up -d
29
30# 4. View logs
31docker compose logs -f

One-Liner

Run this command to download and set up the recipe in one step:

terminal

1curl -fsSL https://docker.recipes/api/recipes/fluentbit/run | bash

Troubleshooting

Config validation failed: Check fluent-bit.conf syntax using fluent-bit --dry-run command
Permission denied reading /var/log: Add user: root to the fluent-bit service configuration
High memory usage: Reduce Mem_Buf_Limit in input plugins and increase flush intervals
Logs not forwarding: Verify output plugin credentials and network connectivity to destinations
Container restart loops: Check configuration file exists and has correct formatting in ./fluent-bit/fluent-bit.conf
Metrics endpoint unreachable: Ensure HTTP_Server On is configured in SERVICE section of fluent-bit.conf

Community Notes

Loading notes...

Download Recipe Kit

Get all files in a ready-to-deploy package

Includes docker-compose.yml, .env template, README, and license

Components

fluent-bit

Related Recipes

Ad Space

Shortcuts: C CopyF FavoriteD Download