docker.recipes

ML Training & Serving Stack

advanced

Complete ML pipeline with MLflow tracking, MinIO artifact storage, and Seldon serving

Overview

MLflow is an open-source platform developed by Databricks that manages the complete machine learning lifecycle, from experimentation and tracking to model deployment and monitoring. Originally created to address the fragmentation in ML tooling, MLflow provides a unified interface for experiment tracking, model versioning, and deployment across multiple frameworks like TensorFlow, PyTorch, and scikit-learn. This stack combines MLflow's comprehensive ML lifecycle management with MinIO's S3-compatible object storage for artifacts and PostgreSQL's robust relational database for metadata storage. The integration creates a complete MLOps pipeline where MinIO handles large model files and datasets while PostgreSQL stores experiment parameters, metrics, and model registry information. MLflow orchestrates the entire workflow, providing a centralized hub for data scientists and ML engineers to track experiments, compare models, and deploy the best performers. This configuration is ideal for ML teams transitioning from notebook-driven development to production-ready MLOps workflows, startups building their first ML infrastructure, and organizations requiring complete control over their ML artifacts and metadata without cloud vendor lock-in.

Key Features

  • Experiment tracking with automatic parameter, metric, and artifact logging
  • Model registry for versioning, staging, and production model management
  • S3-compatible artifact storage via MinIO for models, datasets, and experiment outputs
  • PostgreSQL backend for reliable metadata storage and complex experiment querying
  • Multi-framework support including TensorFlow, PyTorch, scikit-learn, and XGBoost
  • Jupyter notebook integration with pre-configured MLflow tracking environment
  • Web-based UI for experiment comparison, model visualization, and artifact browsing
  • RESTful API for programmatic access to experiments, models, and deployment

Common Use Cases

  • 1ML research teams tracking hyperparameter tuning experiments across multiple algorithms
  • 2Data science departments comparing model performance across different feature sets
  • 3Startups building their first production ML pipeline without cloud dependencies
  • 4Organizations migrating from cloud-based ML platforms to self-hosted infrastructure
  • 5Educational institutions teaching MLOps and experiment management practices
  • 6Companies requiring audit trails and governance for regulatory compliance
  • 7Development teams implementing A/B testing for model performance evaluation

Prerequisites

  • Minimum 4GB RAM (MLflow: 1GB, PostgreSQL: 1GB, MinIO: 2GB recommended)
  • Ports 5000, 8888, 9000, and 9001 available on host system
  • Docker and Docker Compose installed with volume mounting capabilities
  • Basic understanding of Python ML libraries and experiment tracking concepts
  • Familiarity with S3-style object storage APIs for advanced artifact management

For development & testing. Review security settings, change default credentials, and test thoroughly before production use. See Terms

docker-compose.yml

docker-compose.yml
1services: 
2  mlflow: 
3    image: ghcr.io/mlflow/mlflow:latest
4    container_name: mlflow
5    restart: unless-stopped
6    ports: 
7      - "${MLFLOW_PORT:-5000}:5000"
8    command: mlflow server --backend-store-uri postgresql://${DB_USER}:${DB_PASSWORD}@mlflow-db:5432/mlflow --default-artifact-root s3://mlflow/ --host 0.0.0.0
9    environment: 
10      - MLFLOW_S3_ENDPOINT_URL=http://minio:9000
11      - AWS_ACCESS_KEY_ID=${MINIO_ACCESS_KEY}
12      - AWS_SECRET_ACCESS_KEY=${MINIO_SECRET_KEY}
13    depends_on: 
14      - mlflow-db
15      - minio
16
17  mlflow-db: 
18    image: postgres:15-alpine
19    container_name: mlflow-db
20    restart: unless-stopped
21    environment: 
22      - POSTGRES_USER=${DB_USER}
23      - POSTGRES_PASSWORD=${DB_PASSWORD}
24      - POSTGRES_DB=mlflow
25    volumes: 
26      - mlflow_db_data:/var/lib/postgresql/data
27
28  minio: 
29    image: minio/minio:latest
30    container_name: mlflow-minio
31    restart: unless-stopped
32    ports: 
33      - "${MINIO_PORT:-9000}:9000"
34      - "${MINIO_CONSOLE:-9001}:9001"
35    environment: 
36      - MINIO_ROOT_USER=${MINIO_ACCESS_KEY}
37      - MINIO_ROOT_PASSWORD=${MINIO_SECRET_KEY}
38    volumes: 
39      - minio_data:/data
40    command: server /data --console-address ":9001"
41
42  minio-init: 
43    image: minio/mc:latest
44    container_name: minio-init
45    depends_on: 
46      - minio
47    entrypoint: >
48      /bin/sh -c "
49      sleep 5;
50      mc alias set myminio http: //minio:9000 ${MINIO_ACCESS_KEY} ${MINIO_SECRET_KEY};
51      mc mb myminio/mlflow --ignore-existing;
52      exit 0;
53      "
54
55  jupyter: 
56    image: jupyter/scipy-notebook:latest
57    container_name: jupyter
58    restart: unless-stopped
59    ports: 
60      - "${JUPYTER_PORT:-8888}:8888"
61    environment: 
62      - MLFLOW_TRACKING_URI=http://mlflow:5000
63      - MLFLOW_S3_ENDPOINT_URL=http://minio:9000
64      - AWS_ACCESS_KEY_ID=${MINIO_ACCESS_KEY}
65      - AWS_SECRET_ACCESS_KEY=${MINIO_SECRET_KEY}
66    volumes: 
67      - ./notebooks:/home/jovyan/work
68    depends_on: 
69      - mlflow
70
71volumes: 
72  mlflow_db_data: 
73  minio_data:

.env Template

.env
1# ML Training & Serving Stack
2MLFLOW_PORT=5000
3MINIO_PORT=9000
4MINIO_CONSOLE=9001
5JUPYTER_PORT=8888
6
7# Database
8DB_USER=mlflow
9DB_PASSWORD=mlflow_password
10
11# MinIO
12MINIO_ACCESS_KEY=minioadmin
13MINIO_SECRET_KEY=minioadmin

Usage Notes

  1. 1MLflow UI at http://localhost:5000
  2. 2MinIO Console at http://localhost:9001
  3. 3Jupyter Lab at http://localhost:8888
  4. 4Check Jupyter logs for access token
  5. 5MLflow tracks experiments, MinIO stores artifacts
  6. 6Use mlflow.log_params(), mlflow.log_metrics() in code

Individual Services(5 services)

Copy individual services to mix and match with your existing compose files.

mlflow
mlflow:
  image: ghcr.io/mlflow/mlflow:latest
  container_name: mlflow
  restart: unless-stopped
  ports:
    - ${MLFLOW_PORT:-5000}:5000
  command: mlflow server --backend-store-uri postgresql://${DB_USER}:${DB_PASSWORD}@mlflow-db:5432/mlflow --default-artifact-root s3://mlflow/ --host 0.0.0.0
  environment:
    - MLFLOW_S3_ENDPOINT_URL=http://minio:9000
    - AWS_ACCESS_KEY_ID=${MINIO_ACCESS_KEY}
    - AWS_SECRET_ACCESS_KEY=${MINIO_SECRET_KEY}
  depends_on:
    - mlflow-db
    - minio
mlflow-db
mlflow-db:
  image: postgres:15-alpine
  container_name: mlflow-db
  restart: unless-stopped
  environment:
    - POSTGRES_USER=${DB_USER}
    - POSTGRES_PASSWORD=${DB_PASSWORD}
    - POSTGRES_DB=mlflow
  volumes:
    - mlflow_db_data:/var/lib/postgresql/data
minio
minio:
  image: minio/minio:latest
  container_name: mlflow-minio
  restart: unless-stopped
  ports:
    - ${MINIO_PORT:-9000}:9000
    - ${MINIO_CONSOLE:-9001}:9001
  environment:
    - MINIO_ROOT_USER=${MINIO_ACCESS_KEY}
    - MINIO_ROOT_PASSWORD=${MINIO_SECRET_KEY}
  volumes:
    - minio_data:/data
  command: server /data --console-address ":9001"
minio-init
minio-init:
  image: minio/mc:latest
  container_name: minio-init
  depends_on:
    - minio
  entrypoint: |
    /bin/sh -c " sleep 5; mc alias set myminio http://minio:9000 ${MINIO_ACCESS_KEY} ${MINIO_SECRET_KEY}; mc mb myminio/mlflow --ignore-existing; exit 0; "
jupyter
jupyter:
  image: jupyter/scipy-notebook:latest
  container_name: jupyter
  restart: unless-stopped
  ports:
    - ${JUPYTER_PORT:-8888}:8888
  environment:
    - MLFLOW_TRACKING_URI=http://mlflow:5000
    - MLFLOW_S3_ENDPOINT_URL=http://minio:9000
    - AWS_ACCESS_KEY_ID=${MINIO_ACCESS_KEY}
    - AWS_SECRET_ACCESS_KEY=${MINIO_SECRET_KEY}
  volumes:
    - ./notebooks:/home/jovyan/work
  depends_on:
    - mlflow

Quick Start

terminal
1# 1. Create the compose file
2cat > docker-compose.yml << 'EOF'
3services:
4  mlflow:
5    image: ghcr.io/mlflow/mlflow:latest
6    container_name: mlflow
7    restart: unless-stopped
8    ports:
9      - "${MLFLOW_PORT:-5000}:5000"
10    command: mlflow server --backend-store-uri postgresql://${DB_USER}:${DB_PASSWORD}@mlflow-db:5432/mlflow --default-artifact-root s3://mlflow/ --host 0.0.0.0
11    environment:
12      - MLFLOW_S3_ENDPOINT_URL=http://minio:9000
13      - AWS_ACCESS_KEY_ID=${MINIO_ACCESS_KEY}
14      - AWS_SECRET_ACCESS_KEY=${MINIO_SECRET_KEY}
15    depends_on:
16      - mlflow-db
17      - minio
18
19  mlflow-db:
20    image: postgres:15-alpine
21    container_name: mlflow-db
22    restart: unless-stopped
23    environment:
24      - POSTGRES_USER=${DB_USER}
25      - POSTGRES_PASSWORD=${DB_PASSWORD}
26      - POSTGRES_DB=mlflow
27    volumes:
28      - mlflow_db_data:/var/lib/postgresql/data
29
30  minio:
31    image: minio/minio:latest
32    container_name: mlflow-minio
33    restart: unless-stopped
34    ports:
35      - "${MINIO_PORT:-9000}:9000"
36      - "${MINIO_CONSOLE:-9001}:9001"
37    environment:
38      - MINIO_ROOT_USER=${MINIO_ACCESS_KEY}
39      - MINIO_ROOT_PASSWORD=${MINIO_SECRET_KEY}
40    volumes:
41      - minio_data:/data
42    command: server /data --console-address ":9001"
43
44  minio-init:
45    image: minio/mc:latest
46    container_name: minio-init
47    depends_on:
48      - minio
49    entrypoint: >
50      /bin/sh -c "
51      sleep 5;
52      mc alias set myminio http://minio:9000 ${MINIO_ACCESS_KEY} ${MINIO_SECRET_KEY};
53      mc mb myminio/mlflow --ignore-existing;
54      exit 0;
55      "
56
57  jupyter:
58    image: jupyter/scipy-notebook:latest
59    container_name: jupyter
60    restart: unless-stopped
61    ports:
62      - "${JUPYTER_PORT:-8888}:8888"
63    environment:
64      - MLFLOW_TRACKING_URI=http://mlflow:5000
65      - MLFLOW_S3_ENDPOINT_URL=http://minio:9000
66      - AWS_ACCESS_KEY_ID=${MINIO_ACCESS_KEY}
67      - AWS_SECRET_ACCESS_KEY=${MINIO_SECRET_KEY}
68    volumes:
69      - ./notebooks:/home/jovyan/work
70    depends_on:
71      - mlflow
72
73volumes:
74  mlflow_db_data:
75  minio_data:
76EOF
77
78# 2. Create the .env file
79cat > .env << 'EOF'
80# ML Training & Serving Stack
81MLFLOW_PORT=5000
82MINIO_PORT=9000
83MINIO_CONSOLE=9001
84JUPYTER_PORT=8888
85
86# Database
87DB_USER=mlflow
88DB_PASSWORD=mlflow_password
89
90# MinIO
91MINIO_ACCESS_KEY=minioadmin
92MINIO_SECRET_KEY=minioadmin
93EOF
94
95# 3. Start the services
96docker compose up -d
97
98# 4. View logs
99docker compose logs -f

One-Liner

Run this command to download and set up the recipe in one step:

terminal
1curl -fsSL https://docker.recipes/api/recipes/ml-training-serving-stack/run | bash

Troubleshooting

  • MLflow UI shows 'Cannot connect to backend store': Verify PostgreSQL container is running and DB_USER/DB_PASSWORD environment variables match
  • Artifact storage fails with S3 errors: Check MINIO_ACCESS_KEY and MINIO_SECRET_KEY are set correctly and minio-init container completed successfully
  • Jupyter notebook cannot connect to MLflow: Ensure MLFLOW_TRACKING_URI points to http://mlflow:5000 and containers are on the same Docker network
  • MinIO console login fails: Use MINIO_ACCESS_KEY as username and MINIO_SECRET_KEY as password, not root/admin defaults
  • Experiment logging slow or timing out: Increase PostgreSQL memory allocation and check disk space for both database and MinIO volumes
  • Model artifacts not appearing in MLflow UI: Verify MinIO bucket 'mlflow' exists and MLflow has correct S3 endpoint configuration

Community Notes

Loading...
Loading notes...

Download Recipe Kit

Get all files in a ready-to-deploy package

Includes docker-compose.yml, .env template, README, and license

Components

mlflowminiopostgresseldon-core

Tags

#mlflow#mlops#training#serving#minio#machine-learning

Category

AI & Machine Learning

Related Recipes

Ad Space