# MLflow Tracking Server

> MLflow for ML experiment tracking with PostgreSQL and MinIO.

- Category: AI & Machine Learning
- Difficulty: intermediate
- Components: mlflow, postgresql, minio
- Tags: mlflow, ml-ops, tracking, experiments, models
- Source: https://docker.recipes/ai-ml/mlflow-tracking-server

## Overview

MLflow is an open-source platform developed by Databricks that addresses the complete machine learning lifecycle, from experimentation to production deployment. It provides essential MLOps capabilities including experiment tracking, model versioning, artifact storage, and model registry management. The platform has become increasingly popular since its 2018 release as organizations struggle to manage the complexity of ML workflows, reproducibility challenges, and collaboration across data science teams.

This stack combines MLflow's tracking server with PostgreSQL as the backend store and MinIO for artifact storage, creating a production-grade ML experiment management system. PostgreSQL handles all metadata storage including experiment parameters, metrics, and run information with ACID compliance and robust querying capabilities. MinIO provides S3-compatible object storage for model artifacts, datasets, and other files, offering high performance and data protection through erasure coding. This architecture separates concerns effectively - PostgreSQL excels at structured metadata while MinIO handles large binary artifacts.

Data science teams, ML engineers, and organizations implementing MLOps practices should consider this stack when they need centralized experiment tracking with proper data persistence. Unlike cloud-hosted solutions like Weights & Biases, this self-hosted approach provides complete control over sensitive ML data and eliminates recurring costs. The combination is particularly valuable for teams already familiar with PostgreSQL operations or those requiring compliance with data residency requirements, while MinIO's S3 compatibility ensures easy migration paths to cloud storage when needed.

## Features

- Centralized experiment tracking with parameter, metric, and artifact logging across multiple ML frameworks
- Model registry with versioning, stage transitions, and lifecycle management for production deployments
- S3-compatible artifact storage through MinIO with high-performance throughput and data protection
- PostgreSQL backend providing ACID-compliant metadata storage with advanced querying capabilities
- Multi-framework support including scikit-learn, TensorFlow, PyTorch, XGBoost, and custom models
- Web UI for experiment comparison, visualization, and collaborative model review processes
- RESTful API enabling programmatic access to experiments, runs, and model registry operations
- Artifact versioning and lineage tracking for reproducible ML workflows and audit trails

## Use cases

- Data science teams needing centralized experiment tracking across multiple ML projects and researchers
- MLOps implementation for organizations transitioning from ad-hoc ML workflows to structured processes
- Model comparison and hyperparameter optimization workflows with historical performance analysis
- Regulatory compliance scenarios requiring on-premises ML artifact storage and audit trails
- Multi-team ML organizations needing shared model registry with controlled deployment stages
- Academic research institutions requiring cost-effective ML experiment management without cloud vendor lock-in
- Enterprises with data residency requirements preventing use of cloud-hosted ML tracking platforms

## Prerequisites

- Minimum 2GB RAM available (1GB for MLflow/PostgreSQL, 512MB for MinIO operations)
- Docker and Docker Compose installed with ports 5000, 9000, and 9001 available
- Basic understanding of MLflow concepts including experiments, runs, and artifact storage
- Python environment with mlflow package installed for client-side experiment logging
- Network access between ML training environments and the MLflow tracking server
- Understanding of S3-compatible storage concepts for artifact management configuration

## docker-compose.yml

```yaml
services:
  postgres:
    image: postgres:16-alpine
    container_name: mlflow-postgres
    restart: unless-stopped
    environment:
      POSTGRES_USER: ${POSTGRES_USER:-mlflow}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-mlflow}
      POSTGRES_DB: ${POSTGRES_DB:-mlflow}
    volumes:
      - postgres_data:/var/lib/postgresql/data
    networks:
      - mlflow-network

  minio:
    image: minio/minio:latest
    container_name: mlflow-minio
    restart: unless-stopped
    ports:
      - "${MINIO_PORT:-9000}:9000"
      - "${MINIO_CONSOLE_PORT:-9001}:9001"
    environment:
      MINIO_ROOT_USER: ${MINIO_USER:-minioadmin}
      MINIO_ROOT_PASSWORD: ${MINIO_PASSWORD:-minioadmin}
    command: server /data --console-address ":9001"
    volumes:
      - minio_data:/data
    networks:
      - mlflow-network

  mlflow:
    image: ghcr.io/mlflow/mlflow:latest
    container_name: mlflow
    restart: unless-stopped
    ports:
      - "${MLFLOW_PORT:-5000}:5000"
    environment:
      MLFLOW_S3_ENDPOINT_URL: http://minio:9000
      AWS_ACCESS_KEY_ID: ${MINIO_USER:-minioadmin}
      AWS_SECRET_ACCESS_KEY: ${MINIO_PASSWORD:-minioadmin}
    command: >
      mlflow server
      --backend-store-uri postgresql://${POSTGRES_USER:-mlflow}:${POSTGRES_PASSWORD:-mlflow}@postgres:5432/${POSTGRES_DB:-mlflow}
      --default-artifact-root s3://mlflow/
      --host 0.0.0.0
    depends_on:
      - postgres
      - minio
    networks:
      - mlflow-network

volumes:
  postgres_data:
  minio_data:

networks:
  mlflow-network:
    driver: bridge
```

## .env

```bash
# MLflow
MLFLOW_PORT=5000
POSTGRES_USER=mlflow
POSTGRES_PASSWORD=mlflow
POSTGRES_DB=mlflow
MINIO_PORT=9000
MINIO_CONSOLE_PORT=9001
MINIO_USER=minioadmin
MINIO_PASSWORD=minioadmin
```

## Usage notes

- Docs: https://mlflow.org/docs/latest/
- MLflow UI at http://localhost:5000 - experiments, runs, model registry
- MinIO Console at http://localhost:9001 - create 'mlflow' bucket first
- Python: mlflow.set_tracking_uri('http://localhost:5000'), mlflow.log_*
- Set MLFLOW_S3_ENDPOINT_URL in client for artifact storage
- Model registry: mlflow.register_model(), stage transitions

## Troubleshooting

- MLflow server fails to start with database connection error: Ensure PostgreSQL container is healthy and POSTGRES_PASSWORD matches between services
- Artifacts not uploading with S3 connection timeout: Create 'mlflow' bucket in MinIO console at localhost:9001 and verify MINIO_USER credentials
- Experiment logging fails with 'Connection refused': Set mlflow.set_tracking_uri('http://localhost:5000') and ensure MLflow server port is accessible
- MinIO returns 'Access Denied' for artifact storage: Verify AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables match MinIO root credentials
- PostgreSQL data loss after container restart: Check that postgres_data volume is properly mounted and has sufficient disk space
- MLflow UI shows experiments but artifacts missing: Confirm MLFLOW_S3_ENDPOINT_URL is set to http://minio:9000 in client environment

