docker.recipes

MLflow Tracking Server

intermediate

MLflow for ML experiment tracking with PostgreSQL and MinIO.

Overview

MLflow is an open-source platform developed by Databricks that addresses the complete machine learning lifecycle, from experimentation to production deployment. It provides essential MLOps capabilities including experiment tracking, model versioning, artifact storage, and model registry management. The platform has become increasingly popular since its 2018 release as organizations struggle to manage the complexity of ML workflows, reproducibility challenges, and collaboration across data science teams. This stack combines MLflow's tracking server with PostgreSQL as the backend store and MinIO for artifact storage, creating a production-grade ML experiment management system. PostgreSQL handles all metadata storage including experiment parameters, metrics, and run information with ACID compliance and robust querying capabilities. MinIO provides S3-compatible object storage for model artifacts, datasets, and other files, offering high performance and data protection through erasure coding. This architecture separates concerns effectively - PostgreSQL excels at structured metadata while MinIO handles large binary artifacts. Data science teams, ML engineers, and organizations implementing MLOps practices should consider this stack when they need centralized experiment tracking with proper data persistence. Unlike cloud-hosted solutions like Weights & Biases, this self-hosted approach provides complete control over sensitive ML data and eliminates recurring costs. The combination is particularly valuable for teams already familiar with PostgreSQL operations or those requiring compliance with data residency requirements, while MinIO's S3 compatibility ensures easy migration paths to cloud storage when needed.

Key Features

  • Centralized experiment tracking with parameter, metric, and artifact logging across multiple ML frameworks
  • Model registry with versioning, stage transitions, and lifecycle management for production deployments
  • S3-compatible artifact storage through MinIO with high-performance throughput and data protection
  • PostgreSQL backend providing ACID-compliant metadata storage with advanced querying capabilities
  • Multi-framework support including scikit-learn, TensorFlow, PyTorch, XGBoost, and custom models
  • Web UI for experiment comparison, visualization, and collaborative model review processes
  • RESTful API enabling programmatic access to experiments, runs, and model registry operations
  • Artifact versioning and lineage tracking for reproducible ML workflows and audit trails

Common Use Cases

  • 1Data science teams needing centralized experiment tracking across multiple ML projects and researchers
  • 2MLOps implementation for organizations transitioning from ad-hoc ML workflows to structured processes
  • 3Model comparison and hyperparameter optimization workflows with historical performance analysis
  • 4Regulatory compliance scenarios requiring on-premises ML artifact storage and audit trails
  • 5Multi-team ML organizations needing shared model registry with controlled deployment stages
  • 6Academic research institutions requiring cost-effective ML experiment management without cloud vendor lock-in
  • 7Enterprises with data residency requirements preventing use of cloud-hosted ML tracking platforms

Prerequisites

  • Minimum 2GB RAM available (1GB for MLflow/PostgreSQL, 512MB for MinIO operations)
  • Docker and Docker Compose installed with ports 5000, 9000, and 9001 available
  • Basic understanding of MLflow concepts including experiments, runs, and artifact storage
  • Python environment with mlflow package installed for client-side experiment logging
  • Network access between ML training environments and the MLflow tracking server
  • Understanding of S3-compatible storage concepts for artifact management configuration

For development & testing. Review security settings, change default credentials, and test thoroughly before production use. See Terms

docker-compose.yml

docker-compose.yml
1services: 
2  postgres: 
3    image: postgres:16-alpine
4    container_name: mlflow-postgres
5    restart: unless-stopped
6    environment: 
7      POSTGRES_USER: ${POSTGRES_USER:-mlflow}
8      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-mlflow}
9      POSTGRES_DB: ${POSTGRES_DB:-mlflow}
10    volumes: 
11      - postgres_data:/var/lib/postgresql/data
12    networks: 
13      - mlflow-network
14
15  minio: 
16    image: minio/minio:latest
17    container_name: mlflow-minio
18    restart: unless-stopped
19    ports: 
20      - "${MINIO_PORT:-9000}:9000"
21      - "${MINIO_CONSOLE_PORT:-9001}:9001"
22    environment: 
23      MINIO_ROOT_USER: ${MINIO_USER:-minioadmin}
24      MINIO_ROOT_PASSWORD: ${MINIO_PASSWORD:-minioadmin}
25    command: server /data --console-address ":9001"
26    volumes: 
27      - minio_data:/data
28    networks: 
29      - mlflow-network
30
31  mlflow: 
32    image: ghcr.io/mlflow/mlflow:latest
33    container_name: mlflow
34    restart: unless-stopped
35    ports: 
36      - "${MLFLOW_PORT:-5000}:5000"
37    environment: 
38      MLFLOW_S3_ENDPOINT_URL: http://minio:9000
39      AWS_ACCESS_KEY_ID: ${MINIO_USER:-minioadmin}
40      AWS_SECRET_ACCESS_KEY: ${MINIO_PASSWORD:-minioadmin}
41    command: >
42      mlflow server
43      --backend-store-uri postgresql: //${POSTGRES_USER:-mlflow}:${POSTGRES_PASSWORD:-mlflow}@postgres:5432/${POSTGRES_DB:-mlflow}
44      --default-artifact-root s3: //mlflow/
45      --host 0.0.0.0
46    depends_on: 
47      - postgres
48      - minio
49    networks: 
50      - mlflow-network
51
52volumes: 
53  postgres_data: 
54  minio_data: 
55
56networks: 
57  mlflow-network: 
58    driver: bridge

.env Template

.env
1# MLflow
2MLFLOW_PORT=5000
3POSTGRES_USER=mlflow
4POSTGRES_PASSWORD=mlflow
5POSTGRES_DB=mlflow
6MINIO_PORT=9000
7MINIO_CONSOLE_PORT=9001
8MINIO_USER=minioadmin
9MINIO_PASSWORD=minioadmin

Usage Notes

  1. 1Docs: https://mlflow.org/docs/latest/
  2. 2MLflow UI at http://localhost:5000 - experiments, runs, model registry
  3. 3MinIO Console at http://localhost:9001 - create 'mlflow' bucket first
  4. 4Python: mlflow.set_tracking_uri('http://localhost:5000'), mlflow.log_*
  5. 5Set MLFLOW_S3_ENDPOINT_URL in client for artifact storage
  6. 6Model registry: mlflow.register_model(), stage transitions

Individual Services(3 services)

Copy individual services to mix and match with your existing compose files.

postgres
postgres:
  image: postgres:16-alpine
  container_name: mlflow-postgres
  restart: unless-stopped
  environment:
    POSTGRES_USER: ${POSTGRES_USER:-mlflow}
    POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-mlflow}
    POSTGRES_DB: ${POSTGRES_DB:-mlflow}
  volumes:
    - postgres_data:/var/lib/postgresql/data
  networks:
    - mlflow-network
minio
minio:
  image: minio/minio:latest
  container_name: mlflow-minio
  restart: unless-stopped
  ports:
    - ${MINIO_PORT:-9000}:9000
    - ${MINIO_CONSOLE_PORT:-9001}:9001
  environment:
    MINIO_ROOT_USER: ${MINIO_USER:-minioadmin}
    MINIO_ROOT_PASSWORD: ${MINIO_PASSWORD:-minioadmin}
  command: server /data --console-address ":9001"
  volumes:
    - minio_data:/data
  networks:
    - mlflow-network
mlflow
mlflow:
  image: ghcr.io/mlflow/mlflow:latest
  container_name: mlflow
  restart: unless-stopped
  ports:
    - ${MLFLOW_PORT:-5000}:5000
  environment:
    MLFLOW_S3_ENDPOINT_URL: http://minio:9000
    AWS_ACCESS_KEY_ID: ${MINIO_USER:-minioadmin}
    AWS_SECRET_ACCESS_KEY: ${MINIO_PASSWORD:-minioadmin}
  command: |
    mlflow server --backend-store-uri postgresql://${POSTGRES_USER:-mlflow}:${POSTGRES_PASSWORD:-mlflow}@postgres:5432/${POSTGRES_DB:-mlflow} --default-artifact-root s3://mlflow/ --host 0.0.0.0
  depends_on:
    - postgres
    - minio
  networks:
    - mlflow-network

Quick Start

terminal
1# 1. Create the compose file
2cat > docker-compose.yml << 'EOF'
3services:
4  postgres:
5    image: postgres:16-alpine
6    container_name: mlflow-postgres
7    restart: unless-stopped
8    environment:
9      POSTGRES_USER: ${POSTGRES_USER:-mlflow}
10      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-mlflow}
11      POSTGRES_DB: ${POSTGRES_DB:-mlflow}
12    volumes:
13      - postgres_data:/var/lib/postgresql/data
14    networks:
15      - mlflow-network
16
17  minio:
18    image: minio/minio:latest
19    container_name: mlflow-minio
20    restart: unless-stopped
21    ports:
22      - "${MINIO_PORT:-9000}:9000"
23      - "${MINIO_CONSOLE_PORT:-9001}:9001"
24    environment:
25      MINIO_ROOT_USER: ${MINIO_USER:-minioadmin}
26      MINIO_ROOT_PASSWORD: ${MINIO_PASSWORD:-minioadmin}
27    command: server /data --console-address ":9001"
28    volumes:
29      - minio_data:/data
30    networks:
31      - mlflow-network
32
33  mlflow:
34    image: ghcr.io/mlflow/mlflow:latest
35    container_name: mlflow
36    restart: unless-stopped
37    ports:
38      - "${MLFLOW_PORT:-5000}:5000"
39    environment:
40      MLFLOW_S3_ENDPOINT_URL: http://minio:9000
41      AWS_ACCESS_KEY_ID: ${MINIO_USER:-minioadmin}
42      AWS_SECRET_ACCESS_KEY: ${MINIO_PASSWORD:-minioadmin}
43    command: >
44      mlflow server
45      --backend-store-uri postgresql://${POSTGRES_USER:-mlflow}:${POSTGRES_PASSWORD:-mlflow}@postgres:5432/${POSTGRES_DB:-mlflow}
46      --default-artifact-root s3://mlflow/
47      --host 0.0.0.0
48    depends_on:
49      - postgres
50      - minio
51    networks:
52      - mlflow-network
53
54volumes:
55  postgres_data:
56  minio_data:
57
58networks:
59  mlflow-network:
60    driver: bridge
61EOF
62
63# 2. Create the .env file
64cat > .env << 'EOF'
65# MLflow
66MLFLOW_PORT=5000
67POSTGRES_USER=mlflow
68POSTGRES_PASSWORD=mlflow
69POSTGRES_DB=mlflow
70MINIO_PORT=9000
71MINIO_CONSOLE_PORT=9001
72MINIO_USER=minioadmin
73MINIO_PASSWORD=minioadmin
74EOF
75
76# 3. Start the services
77docker compose up -d
78
79# 4. View logs
80docker compose logs -f

One-Liner

Run this command to download and set up the recipe in one step:

terminal
1curl -fsSL https://docker.recipes/api/recipes/mlflow-tracking-server/run | bash

Troubleshooting

  • MLflow server fails to start with database connection error: Ensure PostgreSQL container is healthy and POSTGRES_PASSWORD matches between services
  • Artifacts not uploading with S3 connection timeout: Create 'mlflow' bucket in MinIO console at localhost:9001 and verify MINIO_USER credentials
  • Experiment logging fails with 'Connection refused': Set mlflow.set_tracking_uri('http://localhost:5000') and ensure MLflow server port is accessible
  • MinIO returns 'Access Denied' for artifact storage: Verify AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables match MinIO root credentials
  • PostgreSQL data loss after container restart: Check that postgres_data volume is properly mounted and has sufficient disk space
  • MLflow UI shows experiments but artifacts missing: Confirm MLFLOW_S3_ENDPOINT_URL is set to http://minio:9000 in client environment

Community Notes

Loading...
Loading notes...

Download Recipe Kit

Get all files in a ready-to-deploy package

Includes docker-compose.yml, .env template, README, and license

Components

mlflowpostgresqlminio

Tags

#mlflow#ml-ops#tracking#experiments#models

Category

AI & Machine Learning

Related Recipes

Ad Space