docker.recipes

ClearML MLOps Platform

advanced

Open-source MLOps platform with experiment management, data versioning, and model deployment.

Overview

ClearML is an open-source MLOps platform that provides comprehensive experiment management, model versioning, and automated pipeline orchestration for machine learning teams. Originally developed by Allegro AI, ClearML addresses the critical challenges of ML workflow management by offering unified tracking of experiments, datasets, and model artifacts throughout the entire machine learning lifecycle. The platform combines a powerful web interface for visualization and collaboration with robust backend services for scalable ML operations. This Docker stack integrates ClearML's core services with a complete data infrastructure consisting of MongoDB for metadata storage, Elasticsearch for experiment search and analytics, Redis for caching and job queuing, and MinIO for S3-compatible artifact storage. The architecture separates ClearML into distinct services: an API server for backend operations, a web server for the UI interface, a file server for artifact management, and an agent service for task execution. This distributed approach enables horizontal scaling and provides flexibility for different deployment scenarios. ML engineers, data scientists, and AI teams who need enterprise-grade experiment tracking without vendor lock-in will find this stack particularly valuable. Unlike cloud-based MLOps platforms, this self-hosted solution provides complete data sovereignty while offering features comparable to commercial alternatives like Weights & Biases or MLflow with additional UI polish. The inclusion of the ClearML agent enables automated model training and deployment pipelines, making it suitable for teams transitioning from manual ML workflows to fully automated MLOps practices.

Key Features

  • Comprehensive experiment tracking with automatic logging of hyperparameters, metrics, and model artifacts
  • Visual pipeline builder with drag-and-drop interface for creating complex ML workflows and automation
  • Dataset versioning and lineage tracking with automatic data drift detection capabilities
  • Remote task execution through ClearML agents with support for distributed computing clusters
  • Model registry with A/B testing framework and automated model deployment capabilities
  • Real-time collaboration features including experiment comparison, shared dashboards, and team workspaces
  • Elasticsearch-powered advanced search across experiments with aggregation and filtering capabilities
  • MinIO-based artifact storage providing S3-compatible object storage for models, datasets, and outputs

Common Use Cases

  • 1ML research teams tracking hundreds of experiments across multiple projects with complex hyperparameter optimization
  • 2Data science departments implementing reproducible ML pipelines with automated retraining and deployment
  • 3AI startups requiring enterprise-grade MLOps capabilities without the recurring costs of cloud-based platforms
  • 4Financial institutions needing on-premises ML experiment management due to regulatory compliance requirements
  • 5Academic institutions managing student research projects with centralized experiment tracking and collaboration
  • 6MLOps teams building automated model validation and deployment pipelines with integrated A/B testing
  • 7Organizations migrating from notebook-based ML workflows to production-ready automated pipeline systems

Prerequisites

  • Minimum 8GB RAM recommended for full stack operation (MongoDB 2GB, Elasticsearch 4GB, other services 2GB)
  • Docker Engine 20.10+ and Docker Compose v2 with BuildKit support for multi-service orchestration
  • Available ports 8008, 8080, 8081, 9000, and 9001 for ClearML services and MinIO console access
  • Understanding of MLOps concepts including experiment tracking, model versioning, and pipeline orchestration
  • Python environment with clearml package installed on client machines for experiment integration
  • Basic knowledge of S3-compatible storage concepts for artifact management and MinIO configuration

For development & testing. Review security settings, change default credentials, and test thoroughly before production use. See Terms

docker-compose.yml

docker-compose.yml
1services: 
2  apiserver: 
3    image: allegroai/clearml:latest
4    restart: unless-stopped
5    volumes: 
6      - clearml_logs:/var/log/clearml
7      - clearml_config:/opt/clearml/config
8      - clearml_data:/opt/clearml/data
9    depends_on: 
10      - redis
11      - mongo
12      - elasticsearch
13      - minio
14    environment: 
15      CLEARML_HOST_IP: ${CLEARML_HOST_IP}
16      CLEARML_ELASTIC_SERVICE_HOST: elasticsearch
17      CLEARML_MONGODB_SERVICE_HOST: mongo
18      CLEARML_REDIS_SERVICE_HOST: redis
19      CLEARML_FILESERVER_HOST: http://minio:9000
20      CLEARML__apiserver__mongo__host: mongodb://mongo:27017/auth
21    ports: 
22      - "8008:8008"
23    networks: 
24      - clearml-net
25
26  webserver: 
27    image: allegroai/clearml:latest
28    restart: unless-stopped
29    depends_on: 
30      - apiserver
31    ports: 
32      - "8080:80"
33    networks: 
34      - clearml-net
35
36  fileserver: 
37    image: allegroai/clearml:latest
38    restart: unless-stopped
39    volumes: 
40      - clearml_data:/opt/clearml/data
41    ports: 
42      - "8081:8081"
43    networks: 
44      - clearml-net
45
46  mongo: 
47    image: mongo:6
48    volumes: 
49      - mongo_data:/data/db
50    networks: 
51      - clearml-net
52    restart: unless-stopped
53
54  elasticsearch: 
55    image: elasticsearch:8.11.0
56    environment: 
57      - discovery.type=single-node
58      - xpack.security.enabled=false
59      - ES_JAVA_OPTS=-Xms512m -Xmx512m
60    volumes: 
61      - es_data:/usr/share/elasticsearch/data
62    networks: 
63      - clearml-net
64    restart: unless-stopped
65
66  redis: 
67    image: redis:7-alpine
68    volumes: 
69      - redis_data:/data
70    networks: 
71      - clearml-net
72    restart: unless-stopped
73
74  minio: 
75    image: minio/minio:latest
76    environment: 
77      MINIO_ROOT_USER: ${MINIO_ACCESS_KEY}
78      MINIO_ROOT_PASSWORD: ${MINIO_SECRET_KEY}
79    volumes: 
80      - minio_data:/data
81    command: server /data --console-address ":9001"
82    ports: 
83      - "9000:9000"
84      - "9001:9001"
85    networks: 
86      - clearml-net
87    restart: unless-stopped
88
89  agent: 
90    image: allegroai/clearml-agent:latest
91    environment: 
92      CLEARML_API_HOST: http://apiserver:8008
93      CLEARML_WEB_HOST: http://webserver:80
94      CLEARML_FILES_HOST: http://fileserver:8081
95    volumes: 
96      - /var/run/docker.sock:/var/run/docker.sock
97    depends_on: 
98      - apiserver
99    networks: 
100      - clearml-net
101    restart: unless-stopped
102
103volumes: 
104  clearml_logs: 
105  clearml_config: 
106  clearml_data: 
107  mongo_data: 
108  es_data: 
109  redis_data: 
110  minio_data: 
111
112networks: 
113  clearml-net: 
114    driver: bridge

.env Template

.env
1# ClearML Configuration
2CLEARML_HOST_IP=localhost
3
4# MinIO Configuration
5MINIO_ACCESS_KEY=minioadmin
6MINIO_SECRET_KEY=secure_minio_password

Usage Notes

  1. 1ClearML Web UI at http://localhost:8080
  2. 2API server at http://localhost:8008
  3. 3File server at http://localhost:8081
  4. 4Configure clearml.conf on client machines with server URLs

Individual Services(8 services)

Copy individual services to mix and match with your existing compose files.

apiserver
apiserver:
  image: allegroai/clearml:latest
  restart: unless-stopped
  volumes:
    - clearml_logs:/var/log/clearml
    - clearml_config:/opt/clearml/config
    - clearml_data:/opt/clearml/data
  depends_on:
    - redis
    - mongo
    - elasticsearch
    - minio
  environment:
    CLEARML_HOST_IP: ${CLEARML_HOST_IP}
    CLEARML_ELASTIC_SERVICE_HOST: elasticsearch
    CLEARML_MONGODB_SERVICE_HOST: mongo
    CLEARML_REDIS_SERVICE_HOST: redis
    CLEARML_FILESERVER_HOST: http://minio:9000
    CLEARML__apiserver__mongo__host: mongodb://mongo:27017/auth
  ports:
    - "8008:8008"
  networks:
    - clearml-net
webserver
webserver:
  image: allegroai/clearml:latest
  restart: unless-stopped
  depends_on:
    - apiserver
  ports:
    - "8080:80"
  networks:
    - clearml-net
fileserver
fileserver:
  image: allegroai/clearml:latest
  restart: unless-stopped
  volumes:
    - clearml_data:/opt/clearml/data
  ports:
    - "8081:8081"
  networks:
    - clearml-net
mongo
mongo:
  image: mongo:6
  volumes:
    - mongo_data:/data/db
  networks:
    - clearml-net
  restart: unless-stopped
elasticsearch
elasticsearch:
  image: elasticsearch:8.11.0
  environment:
    - discovery.type=single-node
    - xpack.security.enabled=false
    - ES_JAVA_OPTS=-Xms512m -Xmx512m
  volumes:
    - es_data:/usr/share/elasticsearch/data
  networks:
    - clearml-net
  restart: unless-stopped
redis
redis:
  image: redis:7-alpine
  volumes:
    - redis_data:/data
  networks:
    - clearml-net
  restart: unless-stopped
minio
minio:
  image: minio/minio:latest
  environment:
    MINIO_ROOT_USER: ${MINIO_ACCESS_KEY}
    MINIO_ROOT_PASSWORD: ${MINIO_SECRET_KEY}
  volumes:
    - minio_data:/data
  command: server /data --console-address ":9001"
  ports:
    - "9000:9000"
    - "9001:9001"
  networks:
    - clearml-net
  restart: unless-stopped
agent
agent:
  image: allegroai/clearml-agent:latest
  environment:
    CLEARML_API_HOST: http://apiserver:8008
    CLEARML_WEB_HOST: http://webserver:80
    CLEARML_FILES_HOST: http://fileserver:8081
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock
  depends_on:
    - apiserver
  networks:
    - clearml-net
  restart: unless-stopped

Quick Start

terminal
1# 1. Create the compose file
2cat > docker-compose.yml << 'EOF'
3services:
4  apiserver:
5    image: allegroai/clearml:latest
6    restart: unless-stopped
7    volumes:
8      - clearml_logs:/var/log/clearml
9      - clearml_config:/opt/clearml/config
10      - clearml_data:/opt/clearml/data
11    depends_on:
12      - redis
13      - mongo
14      - elasticsearch
15      - minio
16    environment:
17      CLEARML_HOST_IP: ${CLEARML_HOST_IP}
18      CLEARML_ELASTIC_SERVICE_HOST: elasticsearch
19      CLEARML_MONGODB_SERVICE_HOST: mongo
20      CLEARML_REDIS_SERVICE_HOST: redis
21      CLEARML_FILESERVER_HOST: http://minio:9000
22      CLEARML__apiserver__mongo__host: mongodb://mongo:27017/auth
23    ports:
24      - "8008:8008"
25    networks:
26      - clearml-net
27
28  webserver:
29    image: allegroai/clearml:latest
30    restart: unless-stopped
31    depends_on:
32      - apiserver
33    ports:
34      - "8080:80"
35    networks:
36      - clearml-net
37
38  fileserver:
39    image: allegroai/clearml:latest
40    restart: unless-stopped
41    volumes:
42      - clearml_data:/opt/clearml/data
43    ports:
44      - "8081:8081"
45    networks:
46      - clearml-net
47
48  mongo:
49    image: mongo:6
50    volumes:
51      - mongo_data:/data/db
52    networks:
53      - clearml-net
54    restart: unless-stopped
55
56  elasticsearch:
57    image: elasticsearch:8.11.0
58    environment:
59      - discovery.type=single-node
60      - xpack.security.enabled=false
61      - ES_JAVA_OPTS=-Xms512m -Xmx512m
62    volumes:
63      - es_data:/usr/share/elasticsearch/data
64    networks:
65      - clearml-net
66    restart: unless-stopped
67
68  redis:
69    image: redis:7-alpine
70    volumes:
71      - redis_data:/data
72    networks:
73      - clearml-net
74    restart: unless-stopped
75
76  minio:
77    image: minio/minio:latest
78    environment:
79      MINIO_ROOT_USER: ${MINIO_ACCESS_KEY}
80      MINIO_ROOT_PASSWORD: ${MINIO_SECRET_KEY}
81    volumes:
82      - minio_data:/data
83    command: server /data --console-address ":9001"
84    ports:
85      - "9000:9000"
86      - "9001:9001"
87    networks:
88      - clearml-net
89    restart: unless-stopped
90
91  agent:
92    image: allegroai/clearml-agent:latest
93    environment:
94      CLEARML_API_HOST: http://apiserver:8008
95      CLEARML_WEB_HOST: http://webserver:80
96      CLEARML_FILES_HOST: http://fileserver:8081
97    volumes:
98      - /var/run/docker.sock:/var/run/docker.sock
99    depends_on:
100      - apiserver
101    networks:
102      - clearml-net
103    restart: unless-stopped
104
105volumes:
106  clearml_logs:
107  clearml_config:
108  clearml_data:
109  mongo_data:
110  es_data:
111  redis_data:
112  minio_data:
113
114networks:
115  clearml-net:
116    driver: bridge
117EOF
118
119# 2. Create the .env file
120cat > .env << 'EOF'
121# ClearML Configuration
122CLEARML_HOST_IP=localhost
123
124# MinIO Configuration
125MINIO_ACCESS_KEY=minioadmin
126MINIO_SECRET_KEY=secure_minio_password
127EOF
128
129# 3. Start the services
130docker compose up -d
131
132# 4. View logs
133docker compose logs -f

One-Liner

Run this command to download and set up the recipe in one step:

terminal
1curl -fsSL https://docker.recipes/api/recipes/clearml-platform/run | bash

Troubleshooting

  • Elasticsearch fails to start with 'max virtual memory areas vm.max_map_count [65530] too low': Run sudo sysctl -w vm.max_map_count=262144 on host system
  • ClearML agent cannot execute tasks with Docker socket permission errors: Ensure Docker socket has proper permissions or run agent container with --privileged flag
  • Web UI shows 'Server not responding' errors: Verify CLEARML_HOST_IP environment variable matches your server's accessible IP address
  • MinIO console inaccessible at port 9001: Check MINIO_ACCESS_KEY and MINIO_SECRET_KEY are properly set in environment variables
  • MongoDB connection refused during ClearML startup: Wait for MongoDB to fully initialize before starting dependent services using proper depends_on configuration
  • Large model artifacts fail to upload with timeout errors: Increase MinIO timeout settings and verify sufficient disk space in minio_data volume

Community Notes

Loading...
Loading notes...

Download Recipe Kit

Get all files in a ready-to-deploy package

Includes docker-compose.yml, .env template, README, and license

Components

clearml-serverclearml-agentmongodbelasticsearchredisminio

Tags

#clearml#ml-ops#experiment-tracking#data-versioning#orchestration

Category

AI & Machine Learning

Related Recipes

Ad Space