# Prefect

> Modern workflow orchestration for data pipelines.

- Category: AI & Machine Learning
- Difficulty: intermediate
- Components: prefect, postgres
- Tags: prefect, workflow, orchestration, pipelines
- Source: https://docker.recipes/ai-ml/prefect

## Overview

Prefect is a modern workflow orchestration platform built for Python-first data pipelines, offering a developer-friendly alternative to traditional tools like Apache Airflow. Created to address the complexity and rigidity of existing orchestration solutions, Prefect enables data engineers and data scientists to build dynamic, observable workflows using native Python code with decorators and intuitive APIs. The platform emphasizes ease of development, automatic retries, and real-time monitoring while supporting everything from simple ETL jobs to complex machine learning pipelines.

This stack pairs Prefect's orchestration engine with PostgreSQL as the backend database, creating a production-ready workflow management system. PostgreSQL stores Prefect's flow run metadata, task states, deployment configurations, and execution history, while providing ACID compliance and robust querying capabilities for workflow analytics. The combination offers persistent state management, detailed audit trails, and the ability to scale from single-user development environments to enterprise-grade orchestration platforms.

Data teams building Python-centric pipelines will find this stack particularly valuable, especially those transitioning from notebook-based workflows or seeking alternatives to more complex orchestration tools. Organizations processing data at scale, managing ML model training pipelines, or coordinating multi-step ETL processes benefit from Prefect's dynamic task generation and PostgreSQL's reliability. The setup is ideal for teams that value development velocity and observability while maintaining production-grade data persistence and workflow state management.

## Features

- Pythonic workflow definition using decorators and native Python functions
- Dynamic task generation and conditional workflow execution based on runtime data
- Built-in automatic retry mechanisms with exponential backoff and custom retry conditions
- Real-time flow run monitoring with detailed execution graphs and task state visualization
- Work pools and agent-based execution for distributed processing across multiple environments
- PostgreSQL-backed persistent storage for flow metadata, run history, and deployment configurations
- Caching and result persistence to optimize expensive computations and enable workflow resumption
- Block-based configuration system for managing secrets, storage connections, and third-party integrations

## Use cases

- Data engineering teams building ETL pipelines with complex dependencies and conditional logic
- ML engineers orchestrating model training, evaluation, and deployment workflows
- Financial institutions running daily risk calculations and regulatory reporting pipelines
- E-commerce companies managing inventory updates, recommendation model retraining, and analytics aggregation
- Research organizations coordinating multi-stage data processing for scientific computing
- DevOps teams automating infrastructure provisioning and application deployment workflows
- Startups transitioning from cron jobs and manual scripts to observable, maintainable pipeline automation

## Prerequisites

- Minimum 1.5GB RAM (512MB for Prefect server, 256MB for PostgreSQL, plus workflow execution overhead)
- Port 4200 available for Prefect UI and API access
- Basic Python knowledge for creating flows and understanding decorator-based workflow definitions
- PostgreSQL familiarity for database maintenance and query optimization
- Understanding of workflow orchestration concepts like task dependencies and state management
- Docker Compose environment with network connectivity between services

## docker-compose.yml

```yaml
services:
  prefect-server:
    image: prefecthq/prefect:2-python3.11
    container_name: prefect-server
    restart: unless-stopped
    command: prefect server start --host 0.0.0.0
    environment:
      PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://${DB_USER}:${DB_PASSWORD}@postgres:5432/${DB_NAME}
    ports:
      - "4200:4200"
    depends_on:
      - postgres
    networks:
      - prefect

  postgres:
    image: postgres:16-alpine
    container_name: prefect-postgres
    environment:
      POSTGRES_DB: ${DB_NAME}
      POSTGRES_USER: ${DB_USER}
      POSTGRES_PASSWORD: ${DB_PASSWORD}
    volumes:
      - postgres_data:/var/lib/postgresql/data
    networks:
      - prefect

volumes:
  postgres_data:

networks:
  prefect:
    driver: bridge
```

## .env

```bash
DB_NAME=prefect
DB_USER=prefect
DB_PASSWORD=changeme
```

## Usage notes

- Docs: https://docs.prefect.io/
- UI at http://localhost:4200 - flow runs, deployments, work pools
- Configure client: prefect config set PREFECT_API_URL=http://localhost:4200/api
- Create flow: @flow decorator, run locally or deploy to server
- Start worker: prefect worker start -p default-agent-pool
- Blocks for secrets, storage, and integrations in UI

## Troubleshooting

- Connection refused when accessing UI: Verify prefect-server container is running and port 4200 is properly mapped
- Database connection errors on startup: Check PostgreSQL container health and ensure PREFECT_API_DATABASE_CONNECTION_URL matches postgres credentials
- Flows not appearing in UI: Run 'prefect config set PREFECT_API_URL=http://localhost:4200/api' to configure client connection
- Worker not picking up flow runs: Start a worker with 'prefect worker start -p default-agent-pool' and ensure work pool exists
- PostgreSQL data loss after container restart: Verify postgres_data volume is properly mounted and persistent
- Flow runs stuck in pending state: Check that worker pools are configured and workers are actively polling for work

