# MLflow Complete ML Platform

> End-to-end ML lifecycle platform with experiment tracking, model registry, model serving, and PostgreSQL backend.

- Category: AI & Machine Learning
- Difficulty: advanced
- Components: mlflow, postgresql, minio, nginx, prometheus
- Tags: mlflow, ml-ops, model-registry, experiment-tracking, production
- Source: https://docker.recipes/ai-ml/mlflow-full-stack

## Overview

MLflow is an open-source machine learning lifecycle management platform developed by Databricks that addresses the challenges of experiment tracking, model reproducibility, and deployment in ML workflows. Originally created to solve the fragmented nature of ML tooling, MLflow provides a unified interface for managing experiments, packaging ML models in a reusable format, and deploying models to various serving platforms. The platform has become a cornerstone in the MLOps ecosystem, enabling data scientists and ML engineers to track parameters, metrics, and artifacts across different ML frameworks like scikit-learn, TensorFlow, PyTorch, and XGBoost.

This deployment creates a production-grade MLflow environment with five interconnected services: the MLflow server itself, PostgreSQL as the metadata backend store, MinIO for S3-compatible artifact storage, an initialization container to set up MinIO buckets, and NGINX as a reverse proxy. The MLflow server connects to PostgreSQL to store experiment metadata, runs, and model registry information, while using MinIO to store large artifacts like model files, datasets, and plots. This architecture separates metadata from artifacts, providing better performance and scalability compared to file-based storage backends.

This stack is ideal for ML teams transitioning from local development to production environments, organizations requiring self-hosted ML infrastructure, and companies needing complete control over their ML artifacts and metadata. The combination provides enterprise-grade features like model versioning, experiment comparison, and centralized artifact storage while maintaining the flexibility to customize and scale components independently based on workload requirements.

## Features

- Comprehensive ML experiment tracking with parameter, metric, and artifact logging across multiple frameworks
- Centralized model registry with versioning, stage transitions, and model lineage tracking
- PostgreSQL backend for reliable metadata storage with ACID compliance and query performance
- S3-compatible artifact storage via MinIO for models, datasets, and experiment outputs
- Model serving capabilities with REST API endpoints for real-time inference
- Multi-user experiment sharing and collaboration with organized project workspaces
- Automatic model packaging and containerization for deployment portability
- Web-based UI for experiment comparison, model performance visualization, and registry management

## Use cases

- ML research teams tracking hundreds of experiments across different algorithms and hyperparameters
- Production ML pipelines requiring model version control and automated deployment workflows
- Data science organizations needing centralized artifact storage for large models and datasets
- MLOps teams implementing CI/CD for machine learning with automated model validation
- Financial services companies requiring on-premises ML infrastructure for regulatory compliance
- Healthcare organizations managing medical AI models with strict data governance requirements
- Startups building ML products who need professional experiment tracking without cloud vendor costs

## Prerequisites

- Minimum 2GB RAM for MLflow server and PostgreSQL database operations
- At least 4GB available disk space for PostgreSQL data and MinIO artifact storage
- Ports 80, 5000, 9000, and 9001 available for NGINX, MLflow UI, MinIO API, and MinIO console
- Basic understanding of ML experiment tracking concepts and model lifecycle management
- Docker Compose environment variables configured for database and MinIO credentials
- Custom nginx.conf file configured for production proxy requirements and security headers

## docker-compose.yml

```yaml
services:
  mlflow:
    image: ghcr.io/mlflow/mlflow:latest
    ports:
      - "5000:5000"
    environment:
      - MLFLOW_BACKEND_STORE_URI=postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB}
      - MLFLOW_DEFAULT_ARTIFACT_ROOT=s3://mlflow
      - AWS_ACCESS_KEY_ID=${MINIO_ACCESS_KEY}
      - AWS_SECRET_ACCESS_KEY=${MINIO_SECRET_KEY}
      - MLFLOW_S3_ENDPOINT_URL=http://minio:9000
    command: mlflow server --host 0.0.0.0 --port 5000
    depends_on:
      postgres:
        condition: service_healthy
      minio:
        condition: service_started
    networks:
      - mlflow-net
    restart: unless-stopped

  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: ${POSTGRES_DB}
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER} -d ${POSTGRES_DB}"]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - mlflow-net
    restart: unless-stopped

  minio:
    image: minio/minio:latest
    ports:
      - "9000:9000"
      - "9001:9001"
    environment:
      MINIO_ROOT_USER: ${MINIO_ACCESS_KEY}
      MINIO_ROOT_PASSWORD: ${MINIO_SECRET_KEY}
    volumes:
      - minio_data:/data
    command: server /data --console-address ":9001"
    networks:
      - mlflow-net
    restart: unless-stopped

  minio-init:
    image: minio/mc:latest
    depends_on:
      - minio
    entrypoint: >
      /bin/sh -c "
      sleep 5;
      mc alias set myminio http://minio:9000 ${MINIO_ACCESS_KEY} ${MINIO_SECRET_KEY};
      mc mb myminio/mlflow --ignore-existing;
      exit 0;
      "
    networks:
      - mlflow-net

  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - mlflow
    networks:
      - mlflow-net
    restart: unless-stopped

volumes:
  postgres_data:
  minio_data:

networks:
  mlflow-net:
    driver: bridge
```

## .env

```bash
# MLflow Configuration
POSTGRES_USER=mlflow
POSTGRES_PASSWORD=secure_mlflow_password
POSTGRES_DB=mlflow

# MinIO Configuration
MINIO_ACCESS_KEY=minioadmin
MINIO_SECRET_KEY=secure_minio_password
```

## Usage notes

- MLflow UI at http://localhost:5000
- MinIO console at http://localhost:9001
- Create nginx.conf for production proxy
- Model artifacts stored in MinIO S3-compatible storage

## Troubleshooting

- MLflow server fails to start with database connection error: Verify PostgreSQL health check passes and POSTGRES_* environment variables match between mlflow and postgres services
- Artifact logging fails with S3 endpoint errors: Ensure minio-init container completed successfully and created the mlflow bucket, check MINIO_ACCESS_KEY and MINIO_SECRET_KEY variables
- PostgreSQL container restarts repeatedly: Check available disk space for postgres_data volume and verify POSTGRES_PASSWORD is set in environment variables
- MinIO console inaccessible at port 9001: Confirm MINIO_ROOT_USER and MINIO_ROOT_PASSWORD are properly configured and container logs show successful startup
- NGINX proxy returns 502 Bad Gateway: Verify mlflow service is healthy and running on port 5000, check nginx.conf syntax and container dependency order
- Model artifacts not appearing in MLflow UI: Validate S3 endpoint URL configuration points to minio:9000 and AWS credentials match MinIO root credentials

