# DVC + MLflow Data Science Stack

> Data Version Control with MLflow experiment tracking for reproducible ML pipelines.

- Category: AI & Machine Learning
- Difficulty: intermediate
- Components: mlflow, minio, postgresql, jupyter
- Tags: dvc, mlflow, data-versioning, experiment-tracking, reproducibility
- Source: https://docker.recipes/ai-ml/dvc-mlflow-stack

## Overview

MLflow is an open-source platform designed to manage the complete machine learning lifecycle, from experimentation to deployment. Developed by Databricks, MLflow addresses the complexity of ML workflows by providing experiment tracking, model versioning, and artifact management capabilities that help data scientists and ML engineers maintain reproducibility across their projects. The platform has become essential for teams needing to scale ML operations beyond individual notebooks and ad-hoc experiments.

This stack combines MLflow with Data Version Control (DVC) principles using MinIO for S3-compatible object storage and PostgreSQL for metadata persistence. MLflow handles experiment tracking and model registry functions, while MinIO stores large datasets, model artifacts, and training outputs. PostgreSQL maintains experiment metadata, parameters, and metrics in a structured format. Jupyter provides the interactive development environment where data scientists can run experiments that automatically log to MLflow while accessing versioned data from MinIO.

Data science teams working on collaborative ML projects will find this stack particularly valuable for establishing reproducible workflows. The combination addresses common challenges like data drift tracking, experiment comparison, model lineage, and artifact storage. Research teams, ML startups, and enterprise data science groups can use this setup to implement MLOps practices without vendor lock-in, maintaining full control over their ML infrastructure while benefiting from industry-standard tools for version control and experiment management.

## Features

- Experiment tracking with automatic parameter, metric, and artifact logging
- S3-compatible object storage for datasets, models, and large ML artifacts
- Model registry with versioning and stage transitions for production deployment
- PostgreSQL backend for scalable experiment metadata and query performance
- DVC integration for data versioning and pipeline reproducibility
- JupyterLab interface with pre-configured MLflow and MinIO connectivity
- Bucket lifecycle management for automated artifact retention policies
- Multi-framework ML support including scikit-learn, TensorFlow, and PyTorch

## Use cases

- ML research teams tracking multiple experiment runs with hyperparameter tuning
- Data science organizations implementing reproducible ML pipelines with version control
- Startups building ML products requiring model versioning and deployment tracking
- Enterprise teams collaborating on ML projects with shared experiment history
- Academic institutions teaching MLOps practices with self-hosted infrastructure
- Companies migrating from cloud ML platforms to on-premises solutions
- Teams requiring audit trails and compliance tracking for ML model development

## Prerequisites

- Docker Engine 20.10+ and Docker Compose V2 for container orchestration
- Minimum 4GB RAM (8GB+ recommended for concurrent ML training workloads)
- Available ports 5000, 8888, 9000, and 9001 for service access
- Basic familiarity with Python ML libraries and Jupyter notebook environments
- Understanding of S3 API concepts for MinIO bucket and object management
- Knowledge of MLflow tracking concepts including runs, experiments, and artifacts

## docker-compose.yml

```yaml
services:
  jupyter:
    image: jupyter/scipy-notebook:latest
    ports:
      - "8888:8888"
    environment:
      - MLFLOW_TRACKING_URI=http://mlflow:5000
      - AWS_ACCESS_KEY_ID=${MINIO_ACCESS_KEY}
      - AWS_SECRET_ACCESS_KEY=${MINIO_SECRET_KEY}
      - AWS_DEFAULT_REGION=us-east-1
    volumes:
      - notebooks:/home/jovyan/work
      - ./dvc-config:/home/jovyan/.dvc
    command: start-notebook.sh --NotebookApp.token=${JUPYTER_TOKEN}
    networks:
      - mlstack-net
    restart: unless-stopped

  mlflow:
    image: ghcr.io/mlflow/mlflow:latest
    ports:
      - "5000:5000"
    environment:
      - MLFLOW_BACKEND_STORE_URI=postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/mlflow
      - MLFLOW_DEFAULT_ARTIFACT_ROOT=s3://mlflow-artifacts
      - AWS_ACCESS_KEY_ID=${MINIO_ACCESS_KEY}
      - AWS_SECRET_ACCESS_KEY=${MINIO_SECRET_KEY}
      - MLFLOW_S3_ENDPOINT_URL=http://minio:9000
    command: mlflow server --host 0.0.0.0 --port 5000
    depends_on:
      postgres:
        condition: service_healthy
      minio:
        condition: service_started
    networks:
      - mlstack-net
    restart: unless-stopped

  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: mlflow
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER}"]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - mlstack-net
    restart: unless-stopped

  minio:
    image: minio/minio:latest
    ports:
      - "9000:9000"
      - "9001:9001"
    environment:
      MINIO_ROOT_USER: ${MINIO_ACCESS_KEY}
      MINIO_ROOT_PASSWORD: ${MINIO_SECRET_KEY}
    volumes:
      - minio_data:/data
    command: server /data --console-address ":9001"
    networks:
      - mlstack-net
    restart: unless-stopped

  minio-init:
    image: minio/mc:latest
    depends_on:
      - minio
    entrypoint: >
      /bin/sh -c "
      sleep 5;
      mc alias set myminio http://minio:9000 ${MINIO_ACCESS_KEY} ${MINIO_SECRET_KEY};
      mc mb myminio/mlflow-artifacts --ignore-existing;
      mc mb myminio/dvc-storage --ignore-existing;
      exit 0;
      "
    networks:
      - mlstack-net

volumes:
  notebooks:
  postgres_data:
  minio_data:

networks:
  mlstack-net:
    driver: bridge
```

## .env

```bash
# Jupyter Configuration
JUPYTER_TOKEN=secure_jupyter_token

# PostgreSQL Configuration
POSTGRES_USER=mlflow
POSTGRES_PASSWORD=secure_postgres_password

# MinIO Configuration
MINIO_ACCESS_KEY=minioadmin
MINIO_SECRET_KEY=secure_minio_password
```

## Usage notes

- Jupyter Lab at http://localhost:8888
- MLflow UI at http://localhost:5000
- MinIO Console at http://localhost:9001
- Configure DVC remote: dvc remote add -d minio s3://dvc-storage

## Troubleshooting

- MLflow UI shows 'Connection refused' error: Ensure PostgreSQL container is healthy before MLflow starts, check database connection string format
- Jupyter cannot connect to MLflow tracking server: Verify MLFLOW_TRACKING_URI environment variable points to http://mlflow:5000 within Docker network
- MinIO bucket creation fails during initialization: Check MINIO_ACCESS_KEY and MINIO_SECRET_KEY are properly set in environment file, restart minio-init container
- Large model artifacts fail to upload to MLflow: Increase MinIO upload limits and verify AWS credentials are configured correctly in Jupyter environment
- DVC remote configuration errors: Ensure MinIO buckets exist and DVC is configured with s3://dvc-storage endpoint pointing to MinIO service
- PostgreSQL connection pool exhaustion: Increase max_connections in PostgreSQL configuration or reduce concurrent MLflow experiment runs

