# ClearML MLOps Platform

> Open-source MLOps platform with experiment management, data versioning, and model deployment.

- Category: AI & Machine Learning
- Difficulty: advanced
- Components: clearml-server, clearml-agent, mongodb, elasticsearch, redis, minio
- Tags: clearml, ml-ops, experiment-tracking, data-versioning, orchestration
- Source: https://docker.recipes/ai-ml/clearml-platform

## Overview

ClearML is an open-source MLOps platform that provides comprehensive experiment management, model versioning, and automated pipeline orchestration for machine learning teams. Originally developed by Allegro AI, ClearML addresses the critical challenges of ML workflow management by offering unified tracking of experiments, datasets, and model artifacts throughout the entire machine learning lifecycle. The platform combines a powerful web interface for visualization and collaboration with robust backend services for scalable ML operations.

This Docker stack integrates ClearML's core services with a complete data infrastructure consisting of MongoDB for metadata storage, Elasticsearch for experiment search and analytics, Redis for caching and job queuing, and MinIO for S3-compatible artifact storage. The architecture separates ClearML into distinct services: an API server for backend operations, a web server for the UI interface, a file server for artifact management, and an agent service for task execution. This distributed approach enables horizontal scaling and provides flexibility for different deployment scenarios.

ML engineers, data scientists, and AI teams who need enterprise-grade experiment tracking without vendor lock-in will find this stack particularly valuable. Unlike cloud-based MLOps platforms, this self-hosted solution provides complete data sovereignty while offering features comparable to commercial alternatives like Weights & Biases or MLflow with additional UI polish. The inclusion of the ClearML agent enables automated model training and deployment pipelines, making it suitable for teams transitioning from manual ML workflows to fully automated MLOps practices.

## Features

- Comprehensive experiment tracking with automatic logging of hyperparameters, metrics, and model artifacts
- Visual pipeline builder with drag-and-drop interface for creating complex ML workflows and automation
- Dataset versioning and lineage tracking with automatic data drift detection capabilities
- Remote task execution through ClearML agents with support for distributed computing clusters
- Model registry with A/B testing framework and automated model deployment capabilities
- Real-time collaboration features including experiment comparison, shared dashboards, and team workspaces
- Elasticsearch-powered advanced search across experiments with aggregation and filtering capabilities
- MinIO-based artifact storage providing S3-compatible object storage for models, datasets, and outputs

## Use cases

- ML research teams tracking hundreds of experiments across multiple projects with complex hyperparameter optimization
- Data science departments implementing reproducible ML pipelines with automated retraining and deployment
- AI startups requiring enterprise-grade MLOps capabilities without the recurring costs of cloud-based platforms
- Financial institutions needing on-premises ML experiment management due to regulatory compliance requirements
- Academic institutions managing student research projects with centralized experiment tracking and collaboration
- MLOps teams building automated model validation and deployment pipelines with integrated A/B testing
- Organizations migrating from notebook-based ML workflows to production-ready automated pipeline systems

## Prerequisites

- Minimum 8GB RAM recommended for full stack operation (MongoDB 2GB, Elasticsearch 4GB, other services 2GB)
- Docker Engine 20.10+ and Docker Compose v2 with BuildKit support for multi-service orchestration
- Available ports 8008, 8080, 8081, 9000, and 9001 for ClearML services and MinIO console access
- Understanding of MLOps concepts including experiment tracking, model versioning, and pipeline orchestration
- Python environment with clearml package installed on client machines for experiment integration
- Basic knowledge of S3-compatible storage concepts for artifact management and MinIO configuration

## docker-compose.yml

```yaml
services:
  apiserver:
    image: allegroai/clearml:latest
    restart: unless-stopped
    volumes:
      - clearml_logs:/var/log/clearml
      - clearml_config:/opt/clearml/config
      - clearml_data:/opt/clearml/data
    depends_on:
      - redis
      - mongo
      - elasticsearch
      - minio
    environment:
      CLEARML_HOST_IP: ${CLEARML_HOST_IP}
      CLEARML_ELASTIC_SERVICE_HOST: elasticsearch
      CLEARML_MONGODB_SERVICE_HOST: mongo
      CLEARML_REDIS_SERVICE_HOST: redis
      CLEARML_FILESERVER_HOST: http://minio:9000
      CLEARML__apiserver__mongo__host: mongodb://mongo:27017/auth
    ports:
      - "8008:8008"
    networks:
      - clearml-net

  webserver:
    image: allegroai/clearml:latest
    restart: unless-stopped
    depends_on:
      - apiserver
    ports:
      - "8080:80"
    networks:
      - clearml-net

  fileserver:
    image: allegroai/clearml:latest
    restart: unless-stopped
    volumes:
      - clearml_data:/opt/clearml/data
    ports:
      - "8081:8081"
    networks:
      - clearml-net

  mongo:
    image: mongo:6
    volumes:
      - mongo_data:/data/db
    networks:
      - clearml-net
    restart: unless-stopped

  elasticsearch:
    image: elasticsearch:8.11.0
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - ES_JAVA_OPTS=-Xms512m -Xmx512m
    volumes:
      - es_data:/usr/share/elasticsearch/data
    networks:
      - clearml-net
    restart: unless-stopped

  redis:
    image: redis:7-alpine
    volumes:
      - redis_data:/data
    networks:
      - clearml-net
    restart: unless-stopped

  minio:
    image: minio/minio:latest
    environment:
      MINIO_ROOT_USER: ${MINIO_ACCESS_KEY}
      MINIO_ROOT_PASSWORD: ${MINIO_SECRET_KEY}
    volumes:
      - minio_data:/data
    command: server /data --console-address ":9001"
    ports:
      - "9000:9000"
      - "9001:9001"
    networks:
      - clearml-net
    restart: unless-stopped

  agent:
    image: allegroai/clearml-agent:latest
    environment:
      CLEARML_API_HOST: http://apiserver:8008
      CLEARML_WEB_HOST: http://webserver:80
      CLEARML_FILES_HOST: http://fileserver:8081
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    depends_on:
      - apiserver
    networks:
      - clearml-net
    restart: unless-stopped

volumes:
  clearml_logs:
  clearml_config:
  clearml_data:
  mongo_data:
  es_data:
  redis_data:
  minio_data:

networks:
  clearml-net:
    driver: bridge
```

## .env

```bash
# ClearML Configuration
CLEARML_HOST_IP=localhost

# MinIO Configuration
MINIO_ACCESS_KEY=minioadmin
MINIO_SECRET_KEY=secure_minio_password
```

## Usage notes

- ClearML Web UI at http://localhost:8080
- API server at http://localhost:8008
- File server at http://localhost:8081
- Configure clearml.conf on client machines with server URLs

## Troubleshooting

- Elasticsearch fails to start with 'max virtual memory areas vm.max_map_count [65530] too low': Run sudo sysctl -w vm.max_map_count=262144 on host system
- ClearML agent cannot execute tasks with Docker socket permission errors: Ensure Docker socket has proper permissions or run agent container with --privileged flag
- Web UI shows 'Server not responding' errors: Verify CLEARML_HOST_IP environment variable matches your server's accessible IP address
- MinIO console inaccessible at port 9001: Check MINIO_ACCESS_KEY and MINIO_SECRET_KEY are properly set in environment variables
- MongoDB connection refused during ClearML startup: Wait for MongoDB to fully initialize before starting dependent services using proper depends_on configuration
- Large model artifacts fail to upload with timeout errors: Increase MinIO timeout settings and verify sufficient disk space in minio_data volume

