Prefect

Key Features

• Pythonic workflow definition using decorators and native Python functions
• Dynamic task generation and conditional workflow execution based on runtime data
• Built-in automatic retry mechanisms with exponential backoff and custom retry conditions
• Real-time flow run monitoring with detailed execution graphs and task state visualization
• Work pools and agent-based execution for distributed processing across multiple environments
• PostgreSQL-backed persistent storage for flow metadata, run history, and deployment configurations
• Caching and result persistence to optimize expensive computations and enable workflow resumption
• Block-based configuration system for managing secrets, storage connections, and third-party integrations

Common Use Cases

• 1Data engineering teams building ETL pipelines with complex dependencies and conditional logic
• 2ML engineers orchestrating model training, evaluation, and deployment workflows
• 3Financial institutions running daily risk calculations and regulatory reporting pipelines
• 4E-commerce companies managing inventory updates, recommendation model retraining, and analytics aggregation
• 5Research organizations coordinating multi-stage data processing for scientific computing
• 6DevOps teams automating infrastructure provisioning and application deployment workflows
• 7Startups transitioning from cron jobs and manual scripts to observable, maintainable pipeline automation

Prerequisites

• Minimum 1.5GB RAM (512MB for Prefect server, 256MB for PostgreSQL, plus workflow execution overhead)
• Port 4200 available for Prefect UI and API access
• Basic Python knowledge for creating flows and understanding decorator-based workflow definitions
• PostgreSQL familiarity for database maintenance and query optimization
• Understanding of workflow orchestration concepts like task dependencies and state management
• Docker Compose environment with network connectivity between services

docker-compose.yml

1services: 
2  prefect-server: 
3    image: prefecthq/prefect:2-python3.11
4    container_name: prefect-server
5    restart: unless-stopped
6    command: prefect server start --host 0.0.0.0
7    environment: 
8      PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://${DB_USER}:${DB_PASSWORD}@postgres:5432/${DB_NAME}
9    ports: 
10      - "4200:4200"
11    depends_on: 
12      - postgres
13    networks: 
14      - prefect
15
16  postgres: 
17    image: postgres:16-alpine
18    container_name: prefect-postgres
19    environment: 
20      POSTGRES_DB: ${DB_NAME}
21      POSTGRES_USER: ${DB_USER}
22      POSTGRES_PASSWORD: ${DB_PASSWORD}
23    volumes: 
24      - postgres_data:/var/lib/postgresql/data
25    networks: 
26      - prefect
27
28volumes: 
29  postgres_data: 
30
31networks: 
32  prefect: 
33    driver: bridge

.env Template

1DB_NAME=prefect
2DB_USER=prefect
3DB_PASSWORD=changeme

Usage Notes

Individual Services(2 services)

Copy individual services to mix and match with your existing compose files.

prefect-server:
  image: prefecthq/prefect:2-python3.11
  container_name: prefect-server
  restart: unless-stopped
  command: prefect server start --host 0.0.0.0
  environment:
    PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://${DB_USER}:${DB_PASSWORD}@postgres:5432/${DB_NAME}
  ports:
    - "4200:4200"
  depends_on:
    - postgres
  networks:
    - prefect

postgres:
  image: postgres:16-alpine
  container_name: prefect-postgres
  environment:
    POSTGRES_DB: ${DB_NAME}
    POSTGRES_USER: ${DB_USER}
    POSTGRES_PASSWORD: ${DB_PASSWORD}
  volumes:
    - postgres_data:/var/lib/postgresql/data
  networks:
    - prefect

Quick Start

1# 1. Create the compose file
2cat > docker-compose.yml << 'EOF'
3services:
4  prefect-server:
5    image: prefecthq/prefect:2-python3.11
6    container_name: prefect-server
7    restart: unless-stopped
8    command: prefect server start --host 0.0.0.0
9    environment:
10      PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://${DB_USER}:${DB_PASSWORD}@postgres:5432/${DB_NAME}
11    ports:
12      - "4200:4200"
13    depends_on:
14      - postgres
15    networks:
16      - prefect
17
18  postgres:
19    image: postgres:16-alpine
20    container_name: prefect-postgres
21    environment:
22      POSTGRES_DB: ${DB_NAME}
23      POSTGRES_USER: ${DB_USER}
24      POSTGRES_PASSWORD: ${DB_PASSWORD}
25    volumes:
26      - postgres_data:/var/lib/postgresql/data
27    networks:
28      - prefect
29
30volumes:
31  postgres_data:
32
33networks:
34  prefect:
35    driver: bridge
36EOF
37
38# 2. Create the .env file
39cat > .env << 'EOF'
40DB_NAME=prefect
41DB_USER=prefect
42DB_PASSWORD=changeme
43EOF
44
45# 3. Start the services
46docker compose up -d
47
48# 4. View logs
49docker compose logs -f

One-Liner

Run this command to download and set up the recipe in one step:

1curl -fsSL https://docker.recipes/api/recipes/prefect/run | bash

Troubleshooting