# Apache Airflow with Celery

> Apache Airflow workflow orchestration with Celery executor and Redis broker.

- Category: DevOps & CI/CD
- Difficulty: advanced
- Components: airflow-webserver, airflow-scheduler, airflow-worker, postgres, redis
- Tags: airflow, workflow, orchestration, celery, etl, scheduling
- Source: https://docker.recipes/devops/airflow-celery-redis

## Overview

Apache Airflow is a platform for programmatically authoring, scheduling, and monitoring workflows using Python DAG definitions. Originally developed at Airbnb and later open-sourced, Airflow has become the industry standard for orchestrating complex data pipelines and ETL processes. The Celery executor enables distributed task execution across multiple worker nodes, making it suitable for high-throughput production environments where tasks need to be processed in parallel across different machines or containers.

This stack combines Airflow's web server, scheduler, and Celery workers with Redis as the message broker and PostgreSQL as the metadata database. Redis handles task queuing and result caching between the scheduler and workers, while PostgreSQL stores DAG metadata, task history, and execution logs. The Celery executor allows tasks to be distributed across multiple worker processes, enabling horizontal scaling and fault tolerance that the default SequentialExecutor cannot provide.

Data engineers and DevOps teams running complex ETL pipelines, machine learning workflows, or batch processing jobs benefit from this configuration. The distributed architecture supports organizations processing hundreds of daily workflows with varying resource requirements. Companies migrating from cron jobs or legacy schedulers find this setup provides better visibility, dependency management, and failure recovery compared to traditional approaches.

## Features

- Python DAG definition with rich scheduling expressions and cron syntax
- Distributed task execution using Celery workers across multiple containers
- Web-based UI for monitoring workflow status, logs, and task dependencies
- Redis message broker for fast task queuing and result backend caching
- PostgreSQL metadata database with ACID compliance for workflow state
- Flower monitoring interface for real-time Celery worker and queue visibility
- Task retry logic with exponential backoff and SLA monitoring
- Dynamic pipeline generation and templating with Jinja2 support

## Use cases

- Daily ETL pipelines processing data warehouse loads from multiple sources
- Machine learning model training workflows with GPU resource scheduling
- Data quality monitoring pipelines with automated alerting and remediation
- Multi-cloud data synchronization between AWS S3, Google Cloud, and Azure
- Financial reporting automation with regulatory compliance requirements
- IoT sensor data processing with real-time aggregation and anomaly detection
- Marketing campaign automation with customer segmentation and email triggers

## Prerequisites

- 8GB+ RAM recommended for production workloads (4GB minimum for development)
- Docker and Docker Compose with container orchestration knowledge
- Python programming experience for writing DAGs and custom operators
- Understanding of Celery distributed task queues and worker management
- PostgreSQL administration skills for database maintenance and backups
- Network access to ports 8080 (Airflow UI) and 5555 (Flower monitoring)

## docker-compose.yml

```yaml
services:
  postgres:
    image: postgres:16-alpine
    container_name: airflow-db
    environment:
      - POSTGRES_USER=airflow
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      - POSTGRES_DB=airflow
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U airflow"]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - airflow-network

  redis:
    image: redis:7-alpine
    container_name: airflow-redis
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - airflow-network

  airflow-init:
    image: apache/airflow:2.8.0
    container_name: airflow-init
    entrypoint: /bin/bash
    command:
      - -c
      - |
        airflow db init
        airflow users create --username admin --password ${AIRFLOW_ADMIN_PASSWORD} --firstname Admin --lastname User --role Admin --email admin@example.com
    environment:
      - AIRFLOW__CORE__EXECUTOR=CeleryExecutor
      - AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:${POSTGRES_PASSWORD}@postgres/airflow
      - AIRFLOW__CELERY__BROKER_URL=redis://redis:6379/0
      - AIRFLOW__CELERY__RESULT_BACKEND=db+postgresql://airflow:${POSTGRES_PASSWORD}@postgres/airflow
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    networks:
      - airflow-network

  airflow-webserver:
    image: apache/airflow:2.8.0
    container_name: airflow-webserver
    command: webserver
    environment:
      - AIRFLOW__CORE__EXECUTOR=CeleryExecutor
      - AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:${POSTGRES_PASSWORD}@postgres/airflow
      - AIRFLOW__CELERY__BROKER_URL=redis://redis:6379/0
      - AIRFLOW__CELERY__RESULT_BACKEND=db+postgresql://airflow:${POSTGRES_PASSWORD}@postgres/airflow
      - AIRFLOW__WEBSERVER__SECRET_KEY=${AIRFLOW_SECRET_KEY}
    ports:
      - "8080:8080"
    volumes:
      - ./dags:/opt/airflow/dags
      - ./logs:/opt/airflow/logs
      - ./plugins:/opt/airflow/plugins
    depends_on:
      airflow-init:
        condition: service_completed_successfully
    networks:
      - airflow-network

  airflow-scheduler:
    image: apache/airflow:2.8.0
    container_name: airflow-scheduler
    command: scheduler
    environment:
      - AIRFLOW__CORE__EXECUTOR=CeleryExecutor
      - AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:${POSTGRES_PASSWORD}@postgres/airflow
      - AIRFLOW__CELERY__BROKER_URL=redis://redis:6379/0
      - AIRFLOW__CELERY__RESULT_BACKEND=db+postgresql://airflow:${POSTGRES_PASSWORD}@postgres/airflow
    volumes:
      - ./dags:/opt/airflow/dags
      - ./logs:/opt/airflow/logs
      - ./plugins:/opt/airflow/plugins
    depends_on:
      airflow-init:
        condition: service_completed_successfully
    networks:
      - airflow-network

  airflow-worker:
    image: apache/airflow:2.8.0
    container_name: airflow-worker
    command: celery worker
    environment:
      - AIRFLOW__CORE__EXECUTOR=CeleryExecutor
      - AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:${POSTGRES_PASSWORD}@postgres/airflow
      - AIRFLOW__CELERY__BROKER_URL=redis://redis:6379/0
      - AIRFLOW__CELERY__RESULT_BACKEND=db+postgresql://airflow:${POSTGRES_PASSWORD}@postgres/airflow
    volumes:
      - ./dags:/opt/airflow/dags
      - ./logs:/opt/airflow/logs
      - ./plugins:/opt/airflow/plugins
    depends_on:
      airflow-init:
        condition: service_completed_successfully
    networks:
      - airflow-network

  flower:
    image: apache/airflow:2.8.0
    container_name: airflow-flower
    command: celery flower
    environment:
      - AIRFLOW__CORE__EXECUTOR=CeleryExecutor
      - AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:${POSTGRES_PASSWORD}@postgres/airflow
      - AIRFLOW__CELERY__BROKER_URL=redis://redis:6379/0
    ports:
      - "5555:5555"
    depends_on:
      airflow-init:
        condition: service_completed_successfully
    networks:
      - airflow-network

volumes:
  postgres_data:

networks:
  airflow-network:
    driver: bridge
```

## .env

```bash
# Apache Airflow
POSTGRES_PASSWORD=airflow_db_password
AIRFLOW_ADMIN_PASSWORD=admin123
AIRFLOW_SECRET_KEY=your-secret-key-here
```

## Usage notes

- Airflow UI at http://localhost:8080
- Flower (Celery monitor) at http://localhost:5555
- Place DAGs in ./dags directory
- Scale workers with docker compose scale
- Create dags, logs, plugins directories first

## Troubleshooting

- ImportError when loading DAGs: Ensure custom Python packages are installed in all Airflow containers using requirements.txt or custom Docker images
- Tasks stuck in queued state: Check Redis connectivity and verify Celery workers are running with docker compose logs airflow-worker
- Scheduler not picking up new DAGs: Restart airflow-scheduler container and verify DAG files have correct Python syntax and are in /opt/airflow/dags
- Database connection errors: Verify PostgreSQL container is healthy and POSTGRES_PASSWORD environment variable matches across all services
- Worker memory issues: Increase Docker container memory limits or reduce AIRFLOW__CELERY__WORKER_CONCURRENCY to prevent OOM kills
- Web UI 403 Forbidden errors: Check AIRFLOW__WEBSERVER__SECRET_KEY is set and consistent across restarts, regenerate if corrupted

