# Apache Flink Streaming Cluster

> Apache Flink for real-time stream processing with job manager and task managers.

- Category: Database Stacks
- Difficulty: advanced
- Components: flink-jobmanager, flink-taskmanager
- Tags: flink, streaming, real-time, big-data, analytics
- Source: https://docker.recipes/databases/apache-flink-cluster

## Overview

Apache Flink is a distributed stream processing framework designed for high-throughput, low-latency real-time data processing. Originally developed at the Technical University of Berlin and later donated to the Apache Software Foundation, Flink has become a cornerstone technology for organizations requiring millisecond-level processing of continuous data streams. The flink-jobmanager serves as the cluster coordinator, managing job scheduling, checkpointing, and resource allocation, while flink-taskmanager instances handle the actual data processing workload across distributed computing resources.

This streaming cluster architecture separates concerns between job orchestration and task execution, enabling horizontal scalability and fault tolerance. The jobmanager maintains the master state, coordinates checkpoints for exactly-once processing guarantees, and manages the directed acyclic graph (DAG) of streaming operations. Multiple taskmanager instances provide the computational muscle, each offering configurable task slots that can execute parallel operations on data streams, automatically handling backpressure and state management.

Data engineers and platform teams building real-time analytics pipelines will find this configuration invaluable for processing high-velocity data streams from sources like Apache Kafka, Amazon Kinesis, or custom message brokers. The multi-taskmanager setup provides immediate horizontal scaling capabilities, while the centralized jobmanager ensures consistent job lifecycle management and monitoring through Flink's comprehensive web dashboard and metrics system.

## Features

- JobManager cluster coordination with automatic leader election and high availability
- Configurable task slots per TaskManager for fine-tuned parallelism control
- Built-in exactly-once processing semantics through distributed checkpointing
- Event-time processing with watermark generation for out-of-order data handling
- Integrated backpressure handling across the entire processing pipeline
- Rich windowing operations including tumbling, sliding, and session windows
- SQL interface support for stream processing with dynamic table concepts
- Savepoint mechanism for job versioning and cluster migration

## Use cases

- Real-time fraud detection for financial transactions with sub-second alerting
- IoT sensor data aggregation and anomaly detection for manufacturing systems
- Live recommendation engine updates based on user clickstream analysis
- Real-time ETL pipelines for data lake ingestion with schema evolution
- Complex event processing for supply chain monitoring and optimization
- Live dashboard metrics calculation from application logs and events
- Stream-to-stream joins for enriching data with external reference information

## Prerequisites

- Minimum 4GB RAM per TaskManager instance plus 2GB for JobManager
- Port 8081 available for Flink Web Dashboard access
- Understanding of stream processing concepts and event-time semantics
- Familiarity with Flink job submission via JAR files or SQL queries
- Basic knowledge of parallelism configuration and task slot allocation
- Experience with checkpoint and savepoint management for production deployments

## docker-compose.yml

```yaml
services:
  jobmanager:
    image: flink:latest
    container_name: flink-jobmanager
    restart: unless-stopped
    ports:
      - "${FLINK_WEBUI_PORT:-8081}:8081"
    command: jobmanager
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager
        parallelism.default: 2
    volumes:
      - flink_data:/opt/flink/data
    networks:
      - flink-network

  taskmanager-1:
    image: flink:latest
    container_name: flink-taskmanager-1
    restart: unless-stopped
    depends_on:
      - jobmanager
    command: taskmanager
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager
        taskmanager.numberOfTaskSlots: ${TASK_SLOTS:-4}
    networks:
      - flink-network

  taskmanager-2:
    image: flink:latest
    container_name: flink-taskmanager-2
    restart: unless-stopped
    depends_on:
      - jobmanager
    command: taskmanager
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager
        taskmanager.numberOfTaskSlots: ${TASK_SLOTS:-4}
    networks:
      - flink-network

volumes:
  flink_data:

networks:
  flink-network:
    driver: bridge
```

## .env

```bash
# Apache Flink
FLINK_WEBUI_PORT=8081
TASK_SLOTS=4
```

## Usage notes

- Flink Dashboard at http://localhost:8081
- Submit jobs via web UI or CLI
- Scale task managers as needed
- Great for real-time analytics

## Troubleshooting

- TaskManager not connecting to JobManager: Verify network connectivity and ensure jobmanager.rpc.address matches the service name
- OutOfMemoryError in TaskManager: Increase container memory limits or reduce taskmanager.memory.process.size in configuration
- Jobs failing with checkpoint timeout: Increase execution.checkpointing.timeout and check storage backend performance
- High backpressure warnings: Scale up TaskManager instances or optimize downstream operators and sinks
- Web UI shows 'No TaskManagers available': Check TaskManager logs for startup errors and verify FLINK_PROPERTIES formatting
- Job deployment fails with ClassNotFoundException: Ensure all required dependencies are packaged in the job JAR file

