docker.recipes

MLflow + MinIO + PostgreSQL

advanced

Complete MLflow platform with S3-compatible artifact storage and PostgreSQL backend.

Overview

MLflow is an open-source platform designed to manage the complete machine learning lifecycle, from experimentation and tracking to model deployment and monitoring. Originally developed by Databricks in 2018, MLflow addresses the complexity of ML workflows by providing standardized interfaces for experiment tracking, model packaging, and deployment across different frameworks like TensorFlow, PyTorch, and scikit-learn. This comprehensive platform helps data scientists and ML engineers organize their work, reproduce experiments, and collaborate effectively on machine learning projects. This stack combines MLflow with PostgreSQL as the backend database for metadata storage and MinIO as S3-compatible object storage for artifacts. PostgreSQL provides ACID-compliant transaction support for experiment metadata, parameters, and metrics, while MinIO handles large ML artifacts like datasets, model files, and plots with high-performance S3-compatible APIs. This architecture separates concerns between structured metadata and blob storage, enabling better scalability and performance compared to file-based backends. Data science teams and ML engineers working on collaborative projects will benefit from this stack's enterprise-grade storage capabilities and self-hosted deployment model. Organizations requiring data sovereignty, cost control over cloud storage fees, or integration with existing on-premises infrastructure will find this combination particularly valuable. The stack supports multiple concurrent users, experiment comparison, model versioning, and provides a complete audit trail for compliance requirements in regulated industries.

Key Features

  • Experiment tracking with parameter, metric, and artifact logging across ML frameworks
  • Model registry with versioning, staging, and production deployment workflows
  • S3-compatible artifact storage with MinIO's high-performance object storage
  • PostgreSQL backend for ACID-compliant metadata storage and complex query support
  • Multi-user collaboration with experiment sharing and comparison capabilities
  • RESTful APIs for programmatic access to experiments and model registry
  • Automatic bucket creation and configuration for MLflow artifact storage
  • Web-based UI for experiment visualization and model management

Common Use Cases

  • 1Data science teams tracking experiments across multiple ML projects and frameworks
  • 2ML engineers implementing model versioning and deployment pipelines
  • 3Organizations requiring on-premises ML experiment tracking for data compliance
  • 4Startups building cost-effective ML infrastructure without cloud storage fees
  • 5Research institutions managing large-scale ML experiments with artifact storage
  • 6Companies integrating ML tracking with existing PostgreSQL infrastructure
  • 7MLOps workflows requiring audit trails and experiment reproducibility

Prerequisites

  • Minimum 2GB RAM (1GB for MLflow, 512MB for PostgreSQL, 512MB for MinIO)
  • Docker and Docker Compose installed with network access between containers
  • Available ports 5000 (MLflow), 9000 (MinIO API), and 9001 (MinIO Console)
  • Understanding of ML experiment tracking concepts and S3-compatible storage
  • Basic knowledge of environment variable configuration for database credentials
  • Sufficient disk space for PostgreSQL data and MinIO object storage volumes

For development & testing. Review security settings, change default credentials, and test thoroughly before production use. See Terms

docker-compose.yml

docker-compose.yml
1services: 
2  mlflow: 
3    image: ghcr.io/mlflow/mlflow:latest
4    command: >
5      mlflow server
6      --backend-store-uri postgresql: //${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/mlflow
7      --default-artifact-root s3: //mlflow/
8      --host 0.0.0.0
9      --port 5000
10    environment: 
11      - MLFLOW_S3_ENDPOINT_URL=http://minio:9000
12      - AWS_ACCESS_KEY_ID=${MINIO_ACCESS_KEY}
13      - AWS_SECRET_ACCESS_KEY=${MINIO_SECRET_KEY}
14    ports: 
15      - "5000:5000"
16    depends_on: 
17      - postgres
18      - minio
19    networks: 
20      - mlflow-network
21    restart: unless-stopped
22
23  postgres: 
24    image: postgres:15
25    environment: 
26      - POSTGRES_USER=${POSTGRES_USER}
27      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
28      - POSTGRES_DB=mlflow
29    volumes: 
30      - postgres-data:/var/lib/postgresql/data
31    networks: 
32      - mlflow-network
33    restart: unless-stopped
34
35  minio: 
36    image: minio/minio:latest
37    command: server /data --console-address ":9001"
38    environment: 
39      - MINIO_ROOT_USER=${MINIO_ACCESS_KEY}
40      - MINIO_ROOT_PASSWORD=${MINIO_SECRET_KEY}
41    volumes: 
42      - minio-data:/data
43    ports: 
44      - "9000:9000"
45      - "9001:9001"
46    networks: 
47      - mlflow-network
48    restart: unless-stopped
49
50  minio-init: 
51    image: minio/mc:latest
52    entrypoint: >
53      /bin/sh -c "
54      sleep 5;
55      mc alias set myminio http: //minio:9000 ${MINIO_ACCESS_KEY} ${MINIO_SECRET_KEY};
56      mc mb myminio/mlflow --ignore-existing;
57      exit 0;
58      "
59    depends_on: 
60      - minio
61    networks: 
62      - mlflow-network
63
64volumes: 
65  postgres-data: 
66  minio-data: 
67
68networks: 
69  mlflow-network: 
70    driver: bridge

.env Template

.env
1# MLflow
2POSTGRES_USER=mlflow
3POSTGRES_PASSWORD=secure_postgres_password
4
5# MinIO (S3-compatible storage)
6MINIO_ACCESS_KEY=minioadmin
7MINIO_SECRET_KEY=secure_minio_password

Usage Notes

  1. 1MLflow UI at http://localhost:5000
  2. 2MinIO console at http://localhost:9001
  3. 3Experiment tracking with PostgreSQL
  4. 4Artifact storage in MinIO
  5. 5Model registry included

Individual Services(4 services)

Copy individual services to mix and match with your existing compose files.

mlflow
mlflow:
  image: ghcr.io/mlflow/mlflow:latest
  command: |
    mlflow server --backend-store-uri postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/mlflow --default-artifact-root s3://mlflow/ --host 0.0.0.0 --port 5000
  environment:
    - MLFLOW_S3_ENDPOINT_URL=http://minio:9000
    - AWS_ACCESS_KEY_ID=${MINIO_ACCESS_KEY}
    - AWS_SECRET_ACCESS_KEY=${MINIO_SECRET_KEY}
  ports:
    - "5000:5000"
  depends_on:
    - postgres
    - minio
  networks:
    - mlflow-network
  restart: unless-stopped
postgres
postgres:
  image: postgres:15
  environment:
    - POSTGRES_USER=${POSTGRES_USER}
    - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
    - POSTGRES_DB=mlflow
  volumes:
    - postgres-data:/var/lib/postgresql/data
  networks:
    - mlflow-network
  restart: unless-stopped
minio
minio:
  image: minio/minio:latest
  command: server /data --console-address ":9001"
  environment:
    - MINIO_ROOT_USER=${MINIO_ACCESS_KEY}
    - MINIO_ROOT_PASSWORD=${MINIO_SECRET_KEY}
  volumes:
    - minio-data:/data
  ports:
    - "9000:9000"
    - "9001:9001"
  networks:
    - mlflow-network
  restart: unless-stopped
minio-init
minio-init:
  image: minio/mc:latest
  entrypoint: |
    /bin/sh -c " sleep 5; mc alias set myminio http://minio:9000 ${MINIO_ACCESS_KEY} ${MINIO_SECRET_KEY}; mc mb myminio/mlflow --ignore-existing; exit 0; "
  depends_on:
    - minio
  networks:
    - mlflow-network

Quick Start

terminal
1# 1. Create the compose file
2cat > docker-compose.yml << 'EOF'
3services:
4  mlflow:
5    image: ghcr.io/mlflow/mlflow:latest
6    command: >
7      mlflow server
8      --backend-store-uri postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/mlflow
9      --default-artifact-root s3://mlflow/
10      --host 0.0.0.0
11      --port 5000
12    environment:
13      - MLFLOW_S3_ENDPOINT_URL=http://minio:9000
14      - AWS_ACCESS_KEY_ID=${MINIO_ACCESS_KEY}
15      - AWS_SECRET_ACCESS_KEY=${MINIO_SECRET_KEY}
16    ports:
17      - "5000:5000"
18    depends_on:
19      - postgres
20      - minio
21    networks:
22      - mlflow-network
23    restart: unless-stopped
24
25  postgres:
26    image: postgres:15
27    environment:
28      - POSTGRES_USER=${POSTGRES_USER}
29      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
30      - POSTGRES_DB=mlflow
31    volumes:
32      - postgres-data:/var/lib/postgresql/data
33    networks:
34      - mlflow-network
35    restart: unless-stopped
36
37  minio:
38    image: minio/minio:latest
39    command: server /data --console-address ":9001"
40    environment:
41      - MINIO_ROOT_USER=${MINIO_ACCESS_KEY}
42      - MINIO_ROOT_PASSWORD=${MINIO_SECRET_KEY}
43    volumes:
44      - minio-data:/data
45    ports:
46      - "9000:9000"
47      - "9001:9001"
48    networks:
49      - mlflow-network
50    restart: unless-stopped
51
52  minio-init:
53    image: minio/mc:latest
54    entrypoint: >
55      /bin/sh -c "
56      sleep 5;
57      mc alias set myminio http://minio:9000 ${MINIO_ACCESS_KEY} ${MINIO_SECRET_KEY};
58      mc mb myminio/mlflow --ignore-existing;
59      exit 0;
60      "
61    depends_on:
62      - minio
63    networks:
64      - mlflow-network
65
66volumes:
67  postgres-data:
68  minio-data:
69
70networks:
71  mlflow-network:
72    driver: bridge
73EOF
74
75# 2. Create the .env file
76cat > .env << 'EOF'
77# MLflow
78POSTGRES_USER=mlflow
79POSTGRES_PASSWORD=secure_postgres_password
80
81# MinIO (S3-compatible storage)
82MINIO_ACCESS_KEY=minioadmin
83MINIO_SECRET_KEY=secure_minio_password
84EOF
85
86# 3. Start the services
87docker compose up -d
88
89# 4. View logs
90docker compose logs -f

One-Liner

Run this command to download and set up the recipe in one step:

terminal
1curl -fsSL https://docker.recipes/api/recipes/mlflow-complete/run | bash

Troubleshooting

  • MLflow connection refused: Ensure PostgreSQL is fully initialized before MLflow starts, increase depends_on wait time
  • S3 endpoint not found errors: Verify MLFLOW_S3_ENDPOINT_URL points to minio:9000 and MinIO credentials match
  • Database connection failed: Check POSTGRES_USER and POSTGRES_PASSWORD environment variables match in both services
  • MinIO bucket creation failed: Ensure minio-init container runs after MinIO is ready and credentials are correct
  • Artifact upload timeout: Increase MinIO memory allocation or check network connectivity between MLflow and MinIO
  • Experiment metadata corruption: Verify PostgreSQL volume persistence and check database connection stability

Community Notes

Loading...
Loading notes...

Download Recipe Kit

Get all files in a ready-to-deploy package

Includes docker-compose.yml, .env template, README, and license

Components

mlflowpostgresqlminionginx

Tags

#mlflow#ml-ops#tracking#model-registry#experiments

Category

AI & Machine Learning

Related Recipes

Ad Space