Polyaxon

advanced

ML platform for experiment tracking and automation.

Overview

Polyaxon is a comprehensive machine learning operations platform that provides experiment tracking, hyperparameter optimization, and ML workflow automation. Originally developed to address the challenges of managing complex ML experiments at scale, Polyaxon offers a unified interface for tracking experiments, managing datasets, and orchestrating training jobs across different compute environments. The platform supports both individual data scientists and enterprise teams looking to standardize their ML workflows. This Docker stack combines Polyaxon's API server and scheduler with PostgreSQL for persistent metadata storage, Redis for caching and session management, and RabbitMQ for reliable task queuing and job coordination. PostgreSQL handles experiment metadata, model artifacts information, and user data with ACID compliance, while Redis accelerates frequent queries and caches intermediate results. RabbitMQ manages the distributed task execution, ensuring reliable job scheduling and status updates across the platform. ML teams, data science departments, and AI startups benefit from this configuration when they need centralized experiment tracking without the complexity of Kubernetes deployment. This setup is particularly valuable for teams transitioning from notebook-based workflows to production ML pipelines, organizations implementing MLOps best practices, and research groups requiring reproducible experiment management with proper versioning and collaboration features.

Key Features

Comprehensive experiment tracking with automatic logging of hyperparameters, metrics, and artifacts
PostgreSQL-backed metadata store with full ACID compliance for experiment reproducibility and audit trails
Redis-powered caching layer for fast dashboard queries and real-time experiment monitoring
RabbitMQ message broker enabling distributed job scheduling and reliable task execution
RESTful API server supporting programmatic experiment management and integration with ML frameworks
Scheduler service for automated hyperparameter optimization and experiment queue management
Multi-framework support including TensorFlow, PyTorch, Scikit-learn, and XGBoost integration
Web-based dashboard for experiment comparison, visualization, and collaboration across ML teams

Common Use Cases

1ML research teams tracking hundreds of experiments with different model architectures and hyperparameters
2Data science departments implementing standardized MLOps workflows across multiple projects
3Startups building AI products requiring experiment versioning and model performance comparison
4Academic institutions managing student research projects with proper experiment documentation
5Enterprise ML teams migrating from scattered notebook experiments to centralized tracking systems
6Consulting firms demonstrating ML model development progress to clients with detailed experiment logs
7MLOps engineers setting up local development environments before Kubernetes production deployment

Prerequisites

Minimum 4GB RAM recommended for PostgreSQL experiment storage and Redis caching operations
Docker Engine 20.10+ and Docker Compose v2 for proper container orchestration support
Port 8000 available for Polyaxon API access and web interface connectivity
Python environment with pip for installing polyaxon CLI client and ML framework integrations
Basic understanding of machine learning workflows and experiment management concepts
At least 10GB available disk space for PostgreSQL data persistence and experiment artifact storage

For development & testing. Review security settings, change default credentials, and test thoroughly before production use. See Terms

docker-compose.yml

1services: 
2  polyaxon-api: 
3    image: polyaxon/polyaxon-api:latest
4    container_name: polyaxon-api
5    restart: unless-stopped
6    environment: 
7      POLYAXON_DB_HOST: postgres
8      POLYAXON_REDIS_HOST: redis
9      POLYAXON_RABBITMQ_HOST: rabbitmq
10    ports: 
11      - "8000:8000"
12    depends_on: 
13      - postgres
14      - redis
15      - rabbitmq
16    networks: 
17      - polyaxon
18
19  polyaxon-scheduler: 
20    image: polyaxon/polyaxon-scheduler:latest
21    container_name: polyaxon-scheduler
22    restart: unless-stopped
23    environment: 
24      POLYAXON_DB_HOST: postgres
25      POLYAXON_REDIS_HOST: redis
26    depends_on: 
27      - postgres
28      - redis
29    networks: 
30      - polyaxon
31
32  postgres: 
33    image: postgres:16-alpine
34    container_name: polyaxon-postgres
35    environment: 
36      POSTGRES_DB: polyaxon
37      POSTGRES_USER: polyaxon
38      POSTGRES_PASSWORD: polyaxon
39    volumes: 
40      - postgres_data:/var/lib/postgresql/data
41    networks: 
42      - polyaxon
43
44  redis: 
45    image: redis:alpine
46    container_name: polyaxon-redis
47    networks: 
48      - polyaxon
49
50  rabbitmq: 
51    image: rabbitmq:3-management-alpine
52    container_name: polyaxon-rabbitmq
53    networks: 
54      - polyaxon
55
56volumes: 
57  postgres_data: 
58
59networks: 
60  polyaxon: 
61    driver: bridge

.env Template

.env

1# Configure via polyaxon CLI

Usage Notes

1Docs: https://polyaxon.com/docs/
2API at http://localhost:8000 - configure via polyaxon CLI
3CLI: pip install polyaxon, then polyaxon config set --host=localhost:8000
4Submit jobs: polyaxon run -f polyaxonfile.yaml
5Experiment tracking with automatic artifact storage
6Full platform requires Kubernetes for distributed workloads

Individual Services(5 services)

Copy individual services to mix and match with your existing compose files.

polyaxon-api

polyaxon-api:
  image: polyaxon/polyaxon-api:latest
  container_name: polyaxon-api
  restart: unless-stopped
  environment:
    POLYAXON_DB_HOST: postgres
    POLYAXON_REDIS_HOST: redis
    POLYAXON_RABBITMQ_HOST: rabbitmq
  ports:
    - "8000:8000"
  depends_on:
    - postgres
    - redis
    - rabbitmq
  networks:
    - polyaxon

polyaxon-scheduler

polyaxon-scheduler:
  image: polyaxon/polyaxon-scheduler:latest
  container_name: polyaxon-scheduler
  restart: unless-stopped
  environment:
    POLYAXON_DB_HOST: postgres
    POLYAXON_REDIS_HOST: redis
  depends_on:
    - postgres
    - redis
  networks:
    - polyaxon

postgres

postgres:
  image: postgres:16-alpine
  container_name: polyaxon-postgres
  environment:
    POSTGRES_DB: polyaxon
    POSTGRES_USER: polyaxon
    POSTGRES_PASSWORD: polyaxon
  volumes:
    - postgres_data:/var/lib/postgresql/data
  networks:
    - polyaxon

redis

redis:
  image: redis:alpine
  container_name: polyaxon-redis
  networks:
    - polyaxon

rabbitmq

rabbitmq:
  image: rabbitmq:3-management-alpine
  container_name: polyaxon-rabbitmq
  networks:
    - polyaxon

Quick Start

terminal

1# 1. Create the compose file
2cat > docker-compose.yml << 'EOF'
3services:
4  polyaxon-api:
5    image: polyaxon/polyaxon-api:latest
6    container_name: polyaxon-api
7    restart: unless-stopped
8    environment:
9      POLYAXON_DB_HOST: postgres
10      POLYAXON_REDIS_HOST: redis
11      POLYAXON_RABBITMQ_HOST: rabbitmq
12    ports:
13      - "8000:8000"
14    depends_on:
15      - postgres
16      - redis
17      - rabbitmq
18    networks:
19      - polyaxon
20
21  polyaxon-scheduler:
22    image: polyaxon/polyaxon-scheduler:latest
23    container_name: polyaxon-scheduler
24    restart: unless-stopped
25    environment:
26      POLYAXON_DB_HOST: postgres
27      POLYAXON_REDIS_HOST: redis
28    depends_on:
29      - postgres
30      - redis
31    networks:
32      - polyaxon
33
34  postgres:
35    image: postgres:16-alpine
36    container_name: polyaxon-postgres
37    environment:
38      POSTGRES_DB: polyaxon
39      POSTGRES_USER: polyaxon
40      POSTGRES_PASSWORD: polyaxon
41    volumes:
42      - postgres_data:/var/lib/postgresql/data
43    networks:
44      - polyaxon
45
46  redis:
47    image: redis:alpine
48    container_name: polyaxon-redis
49    networks:
50      - polyaxon
51
52  rabbitmq:
53    image: rabbitmq:3-management-alpine
54    container_name: polyaxon-rabbitmq
55    networks:
56      - polyaxon
57
58volumes:
59  postgres_data:
60
61networks:
62  polyaxon:
63    driver: bridge
64EOF
65
66# 2. Create the .env file
67cat > .env << 'EOF'
68# Configure via polyaxon CLI
69EOF
70
71# 3. Start the services
72docker compose up -d
73
74# 4. View logs
75docker compose logs -f

One-Liner

Run this command to download and set up the recipe in one step:

terminal

1curl -fsSL https://docker.recipes/api/recipes/polyaxon/run | bash

Troubleshooting

API connection refused on port 8000: Verify postgres and redis containers are healthy before polyaxon-api starts
Polyaxon CLI cannot connect to localhost:8000: Run 'polyaxon config set --host=http://localhost:8000' with http protocol specified
Database connection errors in polyaxon-api logs: Check POSTGRES_PASSWORD matches between postgres and polyaxon-api environment variables
Scheduler not processing jobs: Ensure RabbitMQ container is running and accessible from polyaxon-scheduler service
High memory usage in PostgreSQL: Configure postgres shared_buffers and work_mem limits using additional environment variables
Redis connection timeouts during experiment logging: Increase Redis memory allocation or add redis.conf with appropriate timeout settings

Community Notes

Loading notes...

Download Recipe Kit

Get all files in a ready-to-deploy package

Includes docker-compose.yml, .env template, README, and license

Components

polyaxonpostgresredisrabbitmq

Related Recipes

Ad Space

Shortcuts: C CopyF FavoriteD Download