# MLflow + MinIO + PostgreSQL

> Complete MLflow platform with S3-compatible artifact storage and PostgreSQL backend.

- Category: AI & Machine Learning
- Difficulty: advanced
- Components: mlflow, postgresql, minio, nginx
- Tags: mlflow, ml-ops, tracking, model-registry, experiments
- Source: https://docker.recipes/ai-ml/mlflow-complete

## Overview

MLflow is an open-source platform designed to manage the complete machine learning lifecycle, from experimentation and tracking to model deployment and monitoring. Originally developed by Databricks in 2018, MLflow addresses the complexity of ML workflows by providing standardized interfaces for experiment tracking, model packaging, and deployment across different frameworks like TensorFlow, PyTorch, and scikit-learn. This comprehensive platform helps data scientists and ML engineers organize their work, reproduce experiments, and collaborate effectively on machine learning projects.

This stack combines MLflow with PostgreSQL as the backend database for metadata storage and MinIO as S3-compatible object storage for artifacts. PostgreSQL provides ACID-compliant transaction support for experiment metadata, parameters, and metrics, while MinIO handles large ML artifacts like datasets, model files, and plots with high-performance S3-compatible APIs. This architecture separates concerns between structured metadata and blob storage, enabling better scalability and performance compared to file-based backends.

Data science teams and ML engineers working on collaborative projects will benefit from this stack's enterprise-grade storage capabilities and self-hosted deployment model. Organizations requiring data sovereignty, cost control over cloud storage fees, or integration with existing on-premises infrastructure will find this combination particularly valuable. The stack supports multiple concurrent users, experiment comparison, model versioning, and provides a complete audit trail for compliance requirements in regulated industries.

## Features

- Experiment tracking with parameter, metric, and artifact logging across ML frameworks
- Model registry with versioning, staging, and production deployment workflows
- S3-compatible artifact storage with MinIO's high-performance object storage
- PostgreSQL backend for ACID-compliant metadata storage and complex query support
- Multi-user collaboration with experiment sharing and comparison capabilities
- RESTful APIs for programmatic access to experiments and model registry
- Automatic bucket creation and configuration for MLflow artifact storage
- Web-based UI for experiment visualization and model management

## Use cases

- Data science teams tracking experiments across multiple ML projects and frameworks
- ML engineers implementing model versioning and deployment pipelines
- Organizations requiring on-premises ML experiment tracking for data compliance
- Startups building cost-effective ML infrastructure without cloud storage fees
- Research institutions managing large-scale ML experiments with artifact storage
- Companies integrating ML tracking with existing PostgreSQL infrastructure
- MLOps workflows requiring audit trails and experiment reproducibility

## Prerequisites

- Minimum 2GB RAM (1GB for MLflow, 512MB for PostgreSQL, 512MB for MinIO)
- Docker and Docker Compose installed with network access between containers
- Available ports 5000 (MLflow), 9000 (MinIO API), and 9001 (MinIO Console)
- Understanding of ML experiment tracking concepts and S3-compatible storage
- Basic knowledge of environment variable configuration for database credentials
- Sufficient disk space for PostgreSQL data and MinIO object storage volumes

## docker-compose.yml

```yaml
services:
  mlflow:
    image: ghcr.io/mlflow/mlflow:latest
    command: >
      mlflow server
      --backend-store-uri postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/mlflow
      --default-artifact-root s3://mlflow/
      --host 0.0.0.0
      --port 5000
    environment:
      - MLFLOW_S3_ENDPOINT_URL=http://minio:9000
      - AWS_ACCESS_KEY_ID=${MINIO_ACCESS_KEY}
      - AWS_SECRET_ACCESS_KEY=${MINIO_SECRET_KEY}
    ports:
      - "5000:5000"
    depends_on:
      - postgres
      - minio
    networks:
      - mlflow-network
    restart: unless-stopped

  postgres:
    image: postgres:15
    environment:
      - POSTGRES_USER=${POSTGRES_USER}
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      - POSTGRES_DB=mlflow
    volumes:
      - postgres-data:/var/lib/postgresql/data
    networks:
      - mlflow-network
    restart: unless-stopped

  minio:
    image: minio/minio:latest
    command: server /data --console-address ":9001"
    environment:
      - MINIO_ROOT_USER=${MINIO_ACCESS_KEY}
      - MINIO_ROOT_PASSWORD=${MINIO_SECRET_KEY}
    volumes:
      - minio-data:/data
    ports:
      - "9000:9000"
      - "9001:9001"
    networks:
      - mlflow-network
    restart: unless-stopped

  minio-init:
    image: minio/mc:latest
    entrypoint: >
      /bin/sh -c "
      sleep 5;
      mc alias set myminio http://minio:9000 ${MINIO_ACCESS_KEY} ${MINIO_SECRET_KEY};
      mc mb myminio/mlflow --ignore-existing;
      exit 0;
      "
    depends_on:
      - minio
    networks:
      - mlflow-network

volumes:
  postgres-data:
  minio-data:

networks:
  mlflow-network:
    driver: bridge
```

## .env

```bash
# MLflow
POSTGRES_USER=mlflow
POSTGRES_PASSWORD=secure_postgres_password

# MinIO (S3-compatible storage)
MINIO_ACCESS_KEY=minioadmin
MINIO_SECRET_KEY=secure_minio_password
```

## Usage notes

- MLflow UI at http://localhost:5000
- MinIO console at http://localhost:9001
- Experiment tracking with PostgreSQL
- Artifact storage in MinIO
- Model registry included

## Troubleshooting

- MLflow connection refused: Ensure PostgreSQL is fully initialized before MLflow starts, increase depends_on wait time
- S3 endpoint not found errors: Verify MLFLOW_S3_ENDPOINT_URL points to minio:9000 and MinIO credentials match
- Database connection failed: Check POSTGRES_USER and POSTGRES_PASSWORD environment variables match in both services
- MinIO bucket creation failed: Ensure minio-init container runs after MinIO is ready and credentials are correct
- Artifact upload timeout: Increase MinIO memory allocation or check network connectivity between MLflow and MinIO
- Experiment metadata corruption: Verify PostgreSQL volume persistence and check database connection stability

