# Apache Airflow Data Pipeline

> Apache Airflow with Celery executors, PostgreSQL, Redis, and Flower monitoring

- Category: DevOps & CI/CD
- Difficulty: advanced
- Components: airflow, postgres, redis, celery, flower
- Tags: airflow, data-pipeline, etl, workflow, scheduling, celery
- Source: https://docker.recipes/devops/airflow-data-pipeline

## Overview

Apache Airflow is an open-source platform for programmatically authoring, scheduling, and monitoring workflows, originally developed by Airbnb in 2014 to solve complex data pipeline orchestration challenges. It allows data engineers to define workflows as code using Python DAGs (Directed Acyclic Graphs), making it the go-to solution for ETL processes, machine learning pipelines, and task automation across thousands of organizations worldwide. The Celery executor enables horizontal scaling by distributing task execution across multiple worker nodes, while PostgreSQL serves as the robust metadata database storing DAG runs, task instances, and connection details. Redis acts as the message broker for Celery, queuing tasks and enabling real-time communication between the scheduler and workers, while Flower provides a web-based monitoring interface for tracking worker performance and task distribution. This production-grade stack solves the fundamental challenge of reliable, scalable workflow orchestration by combining Airflow's rich scheduling capabilities with Celery's distributed task execution, PostgreSQL's ACID compliance for critical metadata, and Redis's sub-millisecond message passing. Data engineering teams, DevOps professionals, and organizations running complex ETL pipelines will benefit from this configuration's ability to handle thousands of concurrent tasks while maintaining visibility into workflow execution, task dependencies, and system health through comprehensive monitoring and alerting capabilities.

## Features

- Python-based DAG definition with rich operator library for databases, cloud services, and APIs
- CeleryExecutor for horizontal scaling with dynamic worker allocation across multiple nodes
- PostgreSQL metadata database with ACID compliance for reliable DAG run and task state tracking
- Redis message broker enabling sub-millisecond task queuing and worker communication
- Flower monitoring dashboard for real-time Celery worker metrics and task distribution analytics
- Triggerer service for efficient deferrable operator handling without blocking worker slots
- Web UI with Gantt charts, task logs, and DAG visualization for comprehensive workflow monitoring
- Connection and variable management with encryption for secure credential storage

## Use cases

- ETL pipelines processing data from multiple sources into data warehouses with complex dependencies
- Machine learning workflows orchestrating model training, validation, and deployment across environments
- Data quality monitoring with automated checks, alerts, and remediation workflows
- Multi-cloud data synchronization between AWS S3, Google Cloud Storage, and Azure Blob Storage
- Financial reporting automation with regulatory compliance checks and audit trails
- IoT data processing pipelines handling sensor data ingestion, transformation, and analytics
- Marketing campaign automation with customer segmentation, email triggers, and performance tracking

## Prerequisites

- Docker Engine 20.10+ and Docker Compose 2.0+ with at least 6GB available RAM for all services
- Python knowledge for writing DAGs and understanding Airflow's operator patterns
- Basic PostgreSQL administration skills for database maintenance and query optimization
- Understanding of Celery distributed task concepts and worker scaling strategies
- Network ports 8080 (Airflow UI) and 5555 (Flower) available and not blocked by firewalls
- Environment variables configured: POSTGRES_PASSWORD, FERNET_KEY, and ADMIN_PASSWORD

## docker-compose.yml

```yaml
x-airflow-common: &airflow-common
  image: apache/airflow:2.8.0
  environment: &airflow-common-env
    AIRFLOW__CORE__EXECUTOR: CeleryExecutor
    AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:${POSTGRES_PASSWORD}@postgres/airflow
    AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:${POSTGRES_PASSWORD}@postgres/airflow
    AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
    AIRFLOW__CORE__FERNET_KEY: ${FERNET_KEY}
    AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
    AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
    AIRFLOW__API__AUTH_BACKENDS: 'airflow.api.auth.backend.basic_auth'
  volumes:
    - ./dags:/opt/airflow/dags
    - ./logs:/opt/airflow/logs
    - ./plugins:/opt/airflow/plugins
  depends_on:
    - redis
    - postgres

services:
  postgres:
    image: postgres:15-alpine
    container_name: airflow-postgres
    restart: unless-stopped
    environment:
      - POSTGRES_USER=airflow
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      - POSTGRES_DB=airflow
    volumes:
      - postgres_data:/var/lib/postgresql/data

  redis:
    image: redis:7-alpine
    container_name: airflow-redis
    restart: unless-stopped

  airflow-webserver:
    <<: *airflow-common
    container_name: airflow-webserver
    command: webserver
    ports:
      - "${AIRFLOW_PORT:-8080}:8080"
    restart: unless-stopped

  airflow-scheduler:
    <<: *airflow-common
    container_name: airflow-scheduler
    command: scheduler
    restart: unless-stopped

  airflow-worker:
    <<: *airflow-common
    container_name: airflow-worker
    command: celery worker
    restart: unless-stopped

  airflow-triggerer:
    <<: *airflow-common
    container_name: airflow-triggerer
    command: triggerer
    restart: unless-stopped

  airflow-init:
    <<: *airflow-common
    container_name: airflow-init
    entrypoint: /bin/bash
    command:
      - -c
      - |
        airflow db init
        airflow users create --username admin --firstname Admin --lastname User --role Admin --email admin@example.com --password ${ADMIN_PASSWORD}
    profiles:
      - init

  flower:
    <<: *airflow-common
    container_name: airflow-flower
    command: celery flower
    ports:
      - "${FLOWER_PORT:-5555}:5555"
    restart: unless-stopped

volumes:
  postgres_data:
```

## .env

```bash
# Apache Airflow Stack
AIRFLOW_PORT=8080
FLOWER_PORT=5555

# Database
POSTGRES_PASSWORD=airflow_password

# Fernet key (generate with: python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())")
FERNET_KEY=your-fernet-key-here

# Admin user
ADMIN_PASSWORD=admin
```

## Usage notes

- Initialize first: docker compose --profile init up airflow-init
- Airflow UI at http://localhost:8080 (admin/admin)
- Flower (worker monitoring) at http://localhost:5555
- Place DAG files in ./dags directory
- Scale workers: docker compose up -d --scale airflow-worker=3
- Check task logs in ./logs directory

## Troubleshooting

- ImportError in DAGs folder: Ensure Python dependencies are installed in Airflow image or mount virtual environment
- Celery workers not picking up tasks: Check Redis connectivity and verify AIRFLOW__CELERY__BROKER_URL environment variable
- Scheduler not creating task instances: Verify DAG file syntax and check scheduler logs for parsing errors
- Database connection timeout: Increase PostgreSQL max_connections and check AIRFLOW__DATABASE__SQL_ALCHEMY_CONN format
- Flower shows no workers: Ensure airflow-worker container is running and Redis broker URL matches across all services
- Tasks stuck in queued state: Scale up workers with --scale airflow-worker=N or check worker resource limits

