# ML Training & Serving Stack

> Complete ML pipeline with MLflow tracking, MinIO artifact storage, and Seldon serving

- Category: AI & Machine Learning
- Difficulty: advanced
- Components: mlflow, minio, postgres, seldon-core
- Tags: mlflow, mlops, training, serving, minio, machine-learning
- Source: https://docker.recipes/ai-ml/ml-training-serving-stack

## Overview

MLflow is an open-source platform developed by Databricks that manages the complete machine learning lifecycle, from experimentation and tracking to model deployment and monitoring. Originally created to address the fragmentation in ML tooling, MLflow provides a unified interface for experiment tracking, model versioning, and deployment across multiple frameworks like TensorFlow, PyTorch, and scikit-learn. This stack combines MLflow's comprehensive ML lifecycle management with MinIO's S3-compatible object storage for artifacts and PostgreSQL's robust relational database for metadata storage. The integration creates a complete MLOps pipeline where MinIO handles large model files and datasets while PostgreSQL stores experiment parameters, metrics, and model registry information. MLflow orchestrates the entire workflow, providing a centralized hub for data scientists and ML engineers to track experiments, compare models, and deploy the best performers. This configuration is ideal for ML teams transitioning from notebook-driven development to production-ready MLOps workflows, startups building their first ML infrastructure, and organizations requiring complete control over their ML artifacts and metadata without cloud vendor lock-in.

## Features

- Experiment tracking with automatic parameter, metric, and artifact logging
- Model registry for versioning, staging, and production model management
- S3-compatible artifact storage via MinIO for models, datasets, and experiment outputs
- PostgreSQL backend for reliable metadata storage and complex experiment querying
- Multi-framework support including TensorFlow, PyTorch, scikit-learn, and XGBoost
- Jupyter notebook integration with pre-configured MLflow tracking environment
- Web-based UI for experiment comparison, model visualization, and artifact browsing
- RESTful API for programmatic access to experiments, models, and deployment

## Use cases

- ML research teams tracking hyperparameter tuning experiments across multiple algorithms
- Data science departments comparing model performance across different feature sets
- Startups building their first production ML pipeline without cloud dependencies
- Organizations migrating from cloud-based ML platforms to self-hosted infrastructure
- Educational institutions teaching MLOps and experiment management practices
- Companies requiring audit trails and governance for regulatory compliance
- Development teams implementing A/B testing for model performance evaluation

## Prerequisites

- Minimum 4GB RAM (MLflow: 1GB, PostgreSQL: 1GB, MinIO: 2GB recommended)
- Ports 5000, 8888, 9000, and 9001 available on host system
- Docker and Docker Compose installed with volume mounting capabilities
- Basic understanding of Python ML libraries and experiment tracking concepts
- Familiarity with S3-style object storage APIs for advanced artifact management

## docker-compose.yml

```yaml
services:
  mlflow:
    image: ghcr.io/mlflow/mlflow:latest
    container_name: mlflow
    restart: unless-stopped
    ports:
      - "${MLFLOW_PORT:-5000}:5000"
    command: mlflow server --backend-store-uri postgresql://${DB_USER}:${DB_PASSWORD}@mlflow-db:5432/mlflow --default-artifact-root s3://mlflow/ --host 0.0.0.0
    environment:
      - MLFLOW_S3_ENDPOINT_URL=http://minio:9000
      - AWS_ACCESS_KEY_ID=${MINIO_ACCESS_KEY}
      - AWS_SECRET_ACCESS_KEY=${MINIO_SECRET_KEY}
    depends_on:
      - mlflow-db
      - minio

  mlflow-db:
    image: postgres:15-alpine
    container_name: mlflow-db
    restart: unless-stopped
    environment:
      - POSTGRES_USER=${DB_USER}
      - POSTGRES_PASSWORD=${DB_PASSWORD}
      - POSTGRES_DB=mlflow
    volumes:
      - mlflow_db_data:/var/lib/postgresql/data

  minio:
    image: minio/minio:latest
    container_name: mlflow-minio
    restart: unless-stopped
    ports:
      - "${MINIO_PORT:-9000}:9000"
      - "${MINIO_CONSOLE:-9001}:9001"
    environment:
      - MINIO_ROOT_USER=${MINIO_ACCESS_KEY}
      - MINIO_ROOT_PASSWORD=${MINIO_SECRET_KEY}
    volumes:
      - minio_data:/data
    command: server /data --console-address ":9001"

  minio-init:
    image: minio/mc:latest
    container_name: minio-init
    depends_on:
      - minio
    entrypoint: >
      /bin/sh -c "
      sleep 5;
      mc alias set myminio http://minio:9000 ${MINIO_ACCESS_KEY} ${MINIO_SECRET_KEY};
      mc mb myminio/mlflow --ignore-existing;
      exit 0;
      "

  jupyter:
    image: jupyter/scipy-notebook:latest
    container_name: jupyter
    restart: unless-stopped
    ports:
      - "${JUPYTER_PORT:-8888}:8888"
    environment:
      - MLFLOW_TRACKING_URI=http://mlflow:5000
      - MLFLOW_S3_ENDPOINT_URL=http://minio:9000
      - AWS_ACCESS_KEY_ID=${MINIO_ACCESS_KEY}
      - AWS_SECRET_ACCESS_KEY=${MINIO_SECRET_KEY}
    volumes:
      - ./notebooks:/home/jovyan/work
    depends_on:
      - mlflow

volumes:
  mlflow_db_data:
  minio_data:
```

## .env

```bash
# ML Training & Serving Stack
MLFLOW_PORT=5000
MINIO_PORT=9000
MINIO_CONSOLE=9001
JUPYTER_PORT=8888

# Database
DB_USER=mlflow
DB_PASSWORD=mlflow_password

# MinIO
MINIO_ACCESS_KEY=minioadmin
MINIO_SECRET_KEY=minioadmin
```

## Usage notes

- MLflow UI at http://localhost:5000
- MinIO Console at http://localhost:9001
- Jupyter Lab at http://localhost:8888
- Check Jupyter logs for access token
- MLflow tracks experiments, MinIO stores artifacts
- Use mlflow.log_params(), mlflow.log_metrics() in code

## Troubleshooting

- MLflow UI shows 'Cannot connect to backend store': Verify PostgreSQL container is running and DB_USER/DB_PASSWORD environment variables match
- Artifact storage fails with S3 errors: Check MINIO_ACCESS_KEY and MINIO_SECRET_KEY are set correctly and minio-init container completed successfully
- Jupyter notebook cannot connect to MLflow: Ensure MLFLOW_TRACKING_URI points to http://mlflow:5000 and containers are on the same Docker network
- MinIO console login fails: Use MINIO_ACCESS_KEY as username and MINIO_SECRET_KEY as password, not root/admin defaults
- Experiment logging slow or timing out: Increase PostgreSQL memory allocation and check disk space for both database and MinIO volumes
- Model artifacts not appearing in MLflow UI: Verify MinIO bucket 'mlflow' exists and MLflow has correct S3 endpoint configuration

