docker.recipes

Prefect Workflow Orchestration

intermediate

Prefect for workflow orchestration with server, agent, and PostgreSQL.

Overview

Prefect is a modern Python-native workflow orchestration platform that emerged as a next-generation alternative to Apache Airflow, designed to eliminate the pain points of traditional workflow management. Unlike rigid DAG-based systems, Prefect allows developers to build dynamic, data-dependent workflows using pure Python code with built-in observability, automatic retries, and intelligent caching. The platform consists of a server component that provides the API, UI, and orchestration engine, while agents act as lightweight workers that poll for and execute flow runs. This stack combines the Prefect server with PostgreSQL as the backend database and a Prefect agent for flow execution, creating a complete orchestration environment. PostgreSQL serves as the metadata store, tracking flow runs, task states, logs, and scheduling information, while the agent enables distributed execution of workflows across different environments. The server exposes a modern web UI for monitoring pipeline performance, debugging failed runs, and managing deployments. Data engineers, ML engineers, and Python developers building complex ETL pipelines, machine learning workflows, or data processing jobs will find this stack particularly valuable. Unlike Airflow's template-based approach, Prefect allows for truly dynamic workflows where tasks can be generated at runtime based on data or external conditions. This makes it ideal for organizations that need flexible, maintainable data pipelines with modern observability features and minimal operational overhead.

Key Features

  • Pythonic workflow definition with native Python functions as tasks
  • Dynamic task generation and conditional workflow logic at runtime
  • Real-time flow monitoring with detailed execution graphs and logs
  • Automatic retry mechanisms with exponential backoff and custom retry policies
  • Result caching and data persistence between task runs
  • Work pools and agents for distributed execution across environments
  • PostgreSQL metadata store with full ACID compliance and concurrent access
  • Built-in notifications and alerting for flow state changes

Common Use Cases

  • 1ETL pipelines processing daily batch data with dynamic source discovery
  • 2Machine learning model training workflows with hyperparameter optimization
  • 3Data validation and quality monitoring pipelines with conditional alerting
  • 4Multi-tenant data processing with isolated execution environments
  • 5Financial data processing requiring strict audit trails and compliance
  • 6IoT data ingestion pipelines with real-time processing and aggregation
  • 7Development and staging environments for testing complex data workflows

Prerequisites

  • Docker Engine 20.10+ and Docker Compose V2 for container orchestration
  • Minimum 2GB RAM (1GB for PostgreSQL, 1GB for Prefect components)
  • Port 4200 available for Prefect UI access
  • Python knowledge for creating and deploying Prefect flows
  • Basic understanding of workflow orchestration concepts and task dependencies
  • PostgreSQL familiarity for database maintenance and troubleshooting

For development & testing. Review security settings, change default credentials, and test thoroughly before production use. See Terms

docker-compose.yml

docker-compose.yml
1services: 
2  postgres: 
3    image: postgres:16-alpine
4    container_name: prefect-postgres
5    restart: unless-stopped
6    environment: 
7      POSTGRES_USER: ${POSTGRES_USER:-prefect}
8      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-prefect}
9      POSTGRES_DB: ${POSTGRES_DB:-prefect}
10    volumes: 
11      - postgres_data:/var/lib/postgresql/data
12    networks: 
13      - prefect-network
14
15  prefect-server: 
16    image: prefecthq/prefect:2-python3.11
17    container_name: prefect-server
18    restart: unless-stopped
19    command: prefect server start
20    ports: 
21      - "${PREFECT_PORT:-4200}:4200"
22    environment: 
23      PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://${POSTGRES_USER:-prefect}:${POSTGRES_PASSWORD:-prefect}@postgres:5432/${POSTGRES_DB:-prefect}
24      PREFECT_SERVER_API_HOST: 0.0.0.0
25    depends_on: 
26      - postgres
27    networks: 
28      - prefect-network
29
30  prefect-agent: 
31    image: prefecthq/prefect:2-python3.11
32    container_name: prefect-agent
33    restart: unless-stopped
34    command: prefect agent start -q default
35    environment: 
36      PREFECT_API_URL: http://prefect-server:4200/api
37    depends_on: 
38      - prefect-server
39    networks: 
40      - prefect-network
41
42volumes: 
43  postgres_data: 
44
45networks: 
46  prefect-network: 
47    driver: bridge

.env Template

.env
1# Prefect
2PREFECT_PORT=4200
3POSTGRES_USER=prefect
4POSTGRES_PASSWORD=prefect
5POSTGRES_DB=prefect

Usage Notes

  1. 1Prefect UI at http://localhost:4200
  2. 2Define flows in Python
  3. 3Agent executes flow runs
  4. 4Modern alternative to Airflow

Individual Services(3 services)

Copy individual services to mix and match with your existing compose files.

postgres
postgres:
  image: postgres:16-alpine
  container_name: prefect-postgres
  restart: unless-stopped
  environment:
    POSTGRES_USER: ${POSTGRES_USER:-prefect}
    POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-prefect}
    POSTGRES_DB: ${POSTGRES_DB:-prefect}
  volumes:
    - postgres_data:/var/lib/postgresql/data
  networks:
    - prefect-network
prefect-server
prefect-server:
  image: prefecthq/prefect:2-python3.11
  container_name: prefect-server
  restart: unless-stopped
  command: prefect server start
  ports:
    - ${PREFECT_PORT:-4200}:4200
  environment:
    PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://${POSTGRES_USER:-prefect}:${POSTGRES_PASSWORD:-prefect}@postgres:5432/${POSTGRES_DB:-prefect}
    PREFECT_SERVER_API_HOST: 0.0.0.0
  depends_on:
    - postgres
  networks:
    - prefect-network
prefect-agent
prefect-agent:
  image: prefecthq/prefect:2-python3.11
  container_name: prefect-agent
  restart: unless-stopped
  command: prefect agent start -q default
  environment:
    PREFECT_API_URL: http://prefect-server:4200/api
  depends_on:
    - prefect-server
  networks:
    - prefect-network

Quick Start

terminal
1# 1. Create the compose file
2cat > docker-compose.yml << 'EOF'
3services:
4  postgres:
5    image: postgres:16-alpine
6    container_name: prefect-postgres
7    restart: unless-stopped
8    environment:
9      POSTGRES_USER: ${POSTGRES_USER:-prefect}
10      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-prefect}
11      POSTGRES_DB: ${POSTGRES_DB:-prefect}
12    volumes:
13      - postgres_data:/var/lib/postgresql/data
14    networks:
15      - prefect-network
16
17  prefect-server:
18    image: prefecthq/prefect:2-python3.11
19    container_name: prefect-server
20    restart: unless-stopped
21    command: prefect server start
22    ports:
23      - "${PREFECT_PORT:-4200}:4200"
24    environment:
25      PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://${POSTGRES_USER:-prefect}:${POSTGRES_PASSWORD:-prefect}@postgres:5432/${POSTGRES_DB:-prefect}
26      PREFECT_SERVER_API_HOST: 0.0.0.0
27    depends_on:
28      - postgres
29    networks:
30      - prefect-network
31
32  prefect-agent:
33    image: prefecthq/prefect:2-python3.11
34    container_name: prefect-agent
35    restart: unless-stopped
36    command: prefect agent start -q default
37    environment:
38      PREFECT_API_URL: http://prefect-server:4200/api
39    depends_on:
40      - prefect-server
41    networks:
42      - prefect-network
43
44volumes:
45  postgres_data:
46
47networks:
48  prefect-network:
49    driver: bridge
50EOF
51
52# 2. Create the .env file
53cat > .env << 'EOF'
54# Prefect
55PREFECT_PORT=4200
56POSTGRES_USER=prefect
57POSTGRES_PASSWORD=prefect
58POSTGRES_DB=prefect
59EOF
60
61# 3. Start the services
62docker compose up -d
63
64# 4. View logs
65docker compose logs -f

One-Liner

Run this command to download and set up the recipe in one step:

terminal
1curl -fsSL https://docker.recipes/api/recipes/prefect-orchestration/run | bash

Troubleshooting

  • prefect-server container fails with database connection errors: Verify PostgreSQL container is healthy and POSTGRES_PASSWORD matches PREFECT_API_DATABASE_CONNECTION_URL
  • Agent shows 'No work queues found' in logs: Create a work queue in the UI or use 'prefect work-queue create default' command
  • Flow runs stuck in 'Pending' state: Check agent logs for connection issues and ensure PREFECT_API_URL points to correct server address
  • PostgreSQL container crashes with 'data directory not empty': Remove postgres_data volume or check file permissions on mounted directories
  • Prefect UI loads but shows API connection errors: Verify prefect-server container is running and accessible on port 4200
  • High memory usage in PostgreSQL: Tune shared_buffers and work_mem settings, or implement log retention policies for large deployments

Community Notes

Loading...
Loading notes...

Download Recipe Kit

Get all files in a ready-to-deploy package

Includes docker-compose.yml, .env template, README, and license

Components

prefect-serverprefect-agentpostgresql

Tags

#prefect#orchestration#workflow#etl#python

Category

Database Stacks

Related Recipes

Ad Space