# DVC + Iterative Studio + MinIO

> Data version control with experiment tracking and S3-compatible storage.

- Category: AI & Machine Learning
- Difficulty: advanced
- Components: dvc, minio, gitea, iterative-studio
- Tags: dvc, data-versioning, experiments, ml-ops, git
- Source: https://docker.recipes/ai-ml/dvc-studio-complete

## Overview

DVC (Data Version Control) is an open-source version control system specifically designed for machine learning projects, treating datasets and models as first-class citizens alongside code. Originally developed by Dmitry Petrov in 2017, DVC extends Git's capabilities to handle large files and complex ML pipelines, enabling reproducible experiments and collaborative data science workflows. Unlike traditional version control systems that struggle with large binary files, DVC uses a content-addressable storage approach that tracks metadata in Git while storing actual data in remote storage backends. This stack combines DVC with Iterative Studio for experiment visualization and tracking, MinIO as S3-compatible object storage for data artifacts, and Gitea for Git repository hosting. Together, these components create a complete MLOps platform that handles code versioning, data versioning, experiment tracking, and artifact storage in a unified workflow. The integration allows data scientists to version datasets alongside code, track experiment metrics and parameters, visualize results through Iterative Studio's web interface, and store large model files and datasets efficiently in MinIO's high-performance object storage. This combination is particularly valuable for teams transitioning from ad-hoc ML workflows to structured MLOps practices, providing enterprise-grade capabilities without the complexity and cost of cloud-based solutions. Data science teams benefit from having full control over their infrastructure while maintaining compatibility with existing Git workflows and S3-based tools, making it an ideal choice for organizations with data sovereignty requirements or those seeking to avoid cloud vendor lock-in.

## Features

- Git-based pipeline definition with automatic dependency tracking and cache invalidation
- S3-compatible data storage with MinIO providing high-performance object storage for datasets and model artifacts
- Iterative Studio integration for experiment comparison, metric visualization, and model performance tracking
- DVC remote storage configuration using MinIO as backend with automatic bucket initialization
- Gitea-hosted repositories with built-in CI/CD actions for automated DVC pipeline execution
- Content-addressable storage deduplication reducing storage costs for similar datasets
- Pipeline reproducibility with locked dependencies and parameterized experiment configuration
- Multi-stage ML pipeline support with automatic artifact caching between pipeline stages

## Use cases

- Machine learning teams versioning datasets and models with full experiment reproducibility
- Data science organizations requiring on-premises MLOps infrastructure for regulatory compliance
- Research institutions managing large-scale dataset collections with collaborative access controls
- MLOps teams implementing continuous integration for machine learning model training pipelines
- Startups building ML products needing cost-effective alternative to cloud-based MLOps platforms
- Enterprise data science departments with data sovereignty requirements and air-gapped environments
- Academic research groups sharing reproducible ML experiments and dataset lineage tracking

## Prerequisites

- Minimum 4GB RAM (2GB+ for MinIO, 512MB+ for Gitea, 1GB+ for DVC operations and Postgres)
- Docker Engine 20.10+ and Docker Compose v2 for container orchestration
- Available ports 3000, 2222, 9000, and 9001 for Gitea, SSH, MinIO API, and MinIO console
- Basic understanding of Git workflows and machine learning experiment management
- Python environment with DVC client installed for local repository operations
- At least 10GB free disk space for initial data volumes and ML artifacts storage

## docker-compose.yml

```yaml
services:
  gitea:
    image: gitea/gitea:latest
    environment:
      - USER_UID=1000
      - USER_GID=1000
      - GITEA__database__DB_TYPE=postgres
      - GITEA__database__HOST=postgres:5432
      - GITEA__database__NAME=gitea
      - GITEA__database__USER=${POSTGRES_USER}
      - GITEA__database__PASSWD=${POSTGRES_PASSWORD}
    volumes:
      - gitea-data:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "3000:3000"
      - "2222:22"
    depends_on:
      - postgres
    networks:
      - dvc-network
    restart: unless-stopped

  postgres:
    image: postgres:15
    environment:
      - POSTGRES_USER=${POSTGRES_USER}
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      - POSTGRES_DB=gitea
    volumes:
      - postgres-data:/var/lib/postgresql/data
    networks:
      - dvc-network
    restart: unless-stopped

  minio:
    image: minio/minio:latest
    command: server /data --console-address ":9001"
    environment:
      - MINIO_ROOT_USER=${MINIO_ACCESS_KEY}
      - MINIO_ROOT_PASSWORD=${MINIO_SECRET_KEY}
    volumes:
      - minio-data:/data
    ports:
      - "9000:9000"
      - "9001:9001"
    networks:
      - dvc-network
    restart: unless-stopped

  minio-init:
    image: minio/mc:latest
    entrypoint: >
      /bin/sh -c "
      sleep 5;
      mc alias set myminio http://minio:9000 ${MINIO_ACCESS_KEY} ${MINIO_SECRET_KEY};
      mc mb myminio/dvc-storage --ignore-existing;
      exit 0;
      "
    depends_on:
      - minio
    networks:
      - dvc-network

volumes:
  gitea-data:
  postgres-data:
  minio-data:

networks:
  dvc-network:
    driver: bridge
```

## .env

```bash
# DVC Studio
POSTGRES_USER=gitea
POSTGRES_PASSWORD=secure_postgres_password

# MinIO for DVC storage
MINIO_ACCESS_KEY=dvcaccesskey
MINIO_SECRET_KEY=secure_minio_secret

# DVC remote config:
# dvc remote add -d myremote s3://dvc-storage
# dvc remote modify myremote endpointurl http://localhost:9000
```

## Usage notes

- Gitea at http://localhost:3000
- MinIO console at http://localhost:9001
- Configure DVC with S3 remote
- Push data with dvc push
- Track experiments with Git

## Troubleshooting

- DVC push fails with S3 credentials error: Verify MINIO_ACCESS_KEY and MINIO_SECRET_KEY environment variables match your DVC remote configuration
- Gitea repository clone fails over SSH: Check that port 2222 is accessible and SSH keys are properly configured in Gitea user settings
- MinIO bucket access denied during DVC operations: Ensure the minio-init container completed successfully and dvc-storage bucket was created
- Iterative Studio cannot connect to Git repository: Verify Gitea is accessible from Studio and webhook URLs are properly configured for experiment tracking
- DVC pipeline fails with dependency resolution errors: Check that all pipeline stages have correct dependencies defined and cached artifacts are not corrupted
- PostgreSQL connection refused during Gitea startup: Wait for postgres container to fully initialize before Gitea attempts database connection, increase depends_on wait time if necessary

