# Apache Airflow

> Platform to programmatically author and schedule workflows.

- Category: AI & Machine Learning
- Difficulty: advanced
- Components: airflow, postgres, redis
- Tags: airflow, workflow, scheduling, etl
- Source: https://docker.recipes/ai-ml/airflow

## Overview

Apache Airflow is an open-source platform for developing, scheduling, and monitoring batch-oriented workflows. Originally created by Airbnb in 2014, Airflow allows data engineers to define workflows as code using Python, providing programmatic control over complex data pipelines with rich dependency management, retry logic, and monitoring capabilities. The platform has become the industry standard for orchestrating ETL processes, ML pipelines, and data engineering workflows across organizations from startups to Fortune 500 companies.

This Docker stack combines Airflow with PostgreSQL as the metadata database and Redis as the message broker for Celery-based distributed task execution. PostgreSQL stores all workflow metadata, task states, and execution history, while Redis handles the message queuing between the scheduler and worker nodes. The CeleryExecutor configuration enables horizontal scaling of task execution across multiple worker containers, making this setup suitable for production workloads that require parallel processing and fault tolerance.

Data engineers, ML engineers, and DevOps teams building automated data pipelines will find this stack particularly valuable. The combination provides enterprise-grade workflow orchestration with the ability to handle complex dependencies, manage task failures gracefully, and scale execution capacity based on workload demands. Unlike simpler cron-based solutions, this Airflow deployment offers rich monitoring, alerting, and the ability to handle dynamic workflows that adapt based on data conditions or external triggers.

## Features

- Python DAG (Directed Acyclic Graph) definition for workflow authoring with rich dependency management
- CeleryExecutor for distributed task execution across multiple worker containers
- PostgreSQL metadata database storing workflow definitions, task states, and execution history
- Redis message broker enabling reliable task queuing and worker communication
- Web UI at port 8080 for workflow monitoring, task debugging, and manual trigger management
- Automatic DAG discovery from mounted ./dags directory with hot-reloading capabilities
- Comprehensive logging system with persistent log storage across container restarts
- Built-in retry logic, SLA monitoring, and email alerting for production workflow management

## Use cases

- ETL pipeline orchestration for data warehouses with complex data source dependencies
- ML model training workflows with feature engineering, training, and deployment stages
- Daily batch processing of business reports with multiple data transformation steps
- Data lake ingestion pipelines handling various file formats and validation rules
- API data synchronization workflows pulling from multiple external services
- Database maintenance tasks including backups, cleanup, and data archival
- Multi-cloud data replication workflows with failure handling and monitoring

## Prerequisites

- Minimum 4GB RAM (1GB for Airflow components, 1GB for PostgreSQL, 512MB for Redis)
- Port 8080 available for Airflow web interface access
- Docker Compose v3.8+ with volume and network support
- Python knowledge for DAG development and workflow definition
- Understanding of ETL concepts and data pipeline architecture
- PostgreSQL connection string format for database configuration

## docker-compose.yml

```yaml
services:
  airflow-webserver:
    image: apache/airflow:latest
    container_name: airflow-webserver
    command: webserver
    environment:
      AIRFLOW__CORE__EXECUTOR: CeleryExecutor
      AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://${DB_USER}:${DB_PASSWORD}@postgres/${DB_NAME}
      AIRFLOW__CELERY__BROKER_URL: redis://redis:6379/0
      AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://${DB_USER}:${DB_PASSWORD}@postgres/${DB_NAME}
    volumes:
      - ./dags:/opt/airflow/dags
      - airflow_logs:/opt/airflow/logs
    ports:
      - "8080:8080"
    depends_on:
      - postgres
      - redis
    networks:
      - airflow

  airflow-scheduler:
    image: apache/airflow:latest
    container_name: airflow-scheduler
    command: scheduler
    environment:
      AIRFLOW__CORE__EXECUTOR: CeleryExecutor
      AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://${DB_USER}:${DB_PASSWORD}@postgres/${DB_NAME}
      AIRFLOW__CELERY__BROKER_URL: redis://redis:6379/0
    volumes:
      - ./dags:/opt/airflow/dags
      - airflow_logs:/opt/airflow/logs
    depends_on:
      - postgres
      - redis
    networks:
      - airflow

  airflow-worker:
    image: apache/airflow:latest
    container_name: airflow-worker
    command: celery worker
    environment:
      AIRFLOW__CORE__EXECUTOR: CeleryExecutor
      AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://${DB_USER}:${DB_PASSWORD}@postgres/${DB_NAME}
      AIRFLOW__CELERY__BROKER_URL: redis://redis:6379/0
    volumes:
      - ./dags:/opt/airflow/dags
      - airflow_logs:/opt/airflow/logs
    depends_on:
      - postgres
      - redis
    networks:
      - airflow

  postgres:
    image: postgres:16-alpine
    container_name: airflow-postgres
    environment:
      POSTGRES_DB: ${DB_NAME}
      POSTGRES_USER: ${DB_USER}
      POSTGRES_PASSWORD: ${DB_PASSWORD}
    volumes:
      - postgres_data:/var/lib/postgresql/data
    networks:
      - airflow

  redis:
    image: redis:alpine
    container_name: airflow-redis
    networks:
      - airflow

volumes:
  airflow_logs:
  postgres_data:

networks:
  airflow:
    driver: bridge
```

## .env

```bash
DB_NAME=airflow
DB_USER=airflow
DB_PASSWORD=changeme
```

## Usage notes

- Docs: https://airflow.apache.org/docs/
- Access at http://localhost:8080 - default login: airflow / airflow
- Initialize DB first: docker exec airflow-webserver airflow db init
- Place Python DAG files in ./dags folder - auto-detected
- Create admin: airflow users create --role Admin --username admin
- CeleryExecutor scales workers horizontally with replicas

## Troubleshooting

- Import errors in DAGs: Check Python dependencies in ./dags directory and ensure proper syntax
- Scheduler not picking up new DAGs: Verify ./dags volume mount and check DAG parsing errors in logs
- Tasks stuck in queued state: Check Redis connectivity and restart airflow-worker container
- Database connection failures: Verify PostgreSQL container health and environment variables match
- Web UI shows 'Airflow not ready': Run 'docker exec airflow-webserver airflow db init' to initialize database
- Worker tasks timing out: Increase Celery task timeout settings or scale worker replicas

