# Ray Distributed Computing

> Ray cluster for distributed ML and Python workloads.

- Category: AI & Machine Learning
- Difficulty: intermediate
- Components: ray-head, ray-worker
- Tags: ray, distributed, ml, python, cluster
- Source: https://docker.recipes/ai-ml/ray-cluster

## Overview

Ray is an open-source distributed computing framework originally developed at UC Berkeley's RISELab that transforms single-machine Python code into distributed applications without significant rewrites. The ray-head component serves as the cluster coordinator, managing resource allocation, task scheduling, and providing the global control store through Redis, while also hosting the web-based dashboard for monitoring cluster health and job execution. Ray workers connect to the head node to form a compute cluster that can scale Python workloads across multiple machines, supporting everything from simple parallel processing to complex machine learning pipelines with Ray's ecosystem libraries like Ray Tune for hyperparameter optimization and Ray Serve for model deployment.

This containerized Ray cluster architecture enables teams to build scalable distributed systems using familiar Python syntax, where functions decorated with @ray.remote automatically become distributable across the cluster. The head node acts as the single point of coordination, maintaining the global state and scheduling tasks to available workers, while workers execute the actual computation and report back results. This setup eliminates the complexity typically associated with distributed computing frameworks like Spark or Dask, making parallel processing accessible to Python developers without deep distributed systems expertise.

Data scientists and ML engineers working with computationally intensive workloads will find this stack particularly valuable, as Ray excels at hyperparameter tuning, distributed training, and reinforcement learning scenarios that require dynamic task graphs. The containerized deployment makes it ideal for teams needing reproducible distributed computing environments across development and production, while the shared memory configuration optimizes performance for data-heavy operations common in machine learning workflows.

## Features

- Built-in Redis-backed global control store for distributed state management and coordination
- Interactive Ray Dashboard with real-time cluster metrics, job monitoring, and resource utilization graphs
- Automatic task scheduling with work-stealing algorithms for optimal resource utilization across workers
- Native support for nested parallelism allowing @ray.remote functions to call other remote functions
- Object store with automatic serialization handling for NumPy arrays, Pandas DataFrames, and custom Python objects
- Dynamic actor model supporting stateful distributed classes that persist across multiple function calls
- Built-in support for GPU scheduling and allocation when CUDA-enabled Ray images are used
- Ray Client interface enabling remote cluster access from Jupyter notebooks and local Python scripts

## Use cases

- Hyperparameter optimization for machine learning models using Ray Tune across multiple parameter combinations
- Distributed training of deep learning models that don't fit on a single GPU or require data parallelism
- Monte Carlo simulations and financial modeling requiring thousands of parallel scenario calculations
- Reinforcement learning environments where multiple agents train simultaneously across distributed workers
- Large-scale data preprocessing and ETL pipelines that benefit from parallel processing of data chunks
- Computer vision workloads like batch image processing or video analysis across distributed compute resources
- Scientific computing applications requiring parallel execution of CPU-intensive numerical computations

## Prerequisites

- Minimum 4GB RAM per container with additional 2GB shared memory allocation for efficient object sharing
- Docker host with sufficient CPU cores to benefit from distributed processing (recommended 4+ cores)
- Available ports 8265 for Ray Dashboard access and 10001 for Ray Client connections
- Basic understanding of Python decorators and async programming concepts for effective Ray usage
- Familiarity with distributed computing concepts like task scheduling and shared state management
- Network connectivity between containers for Redis communication on port 6379 and worker registration

## docker-compose.yml

```yaml
services:
  ray-head:
    image: rayproject/ray:latest
    container_name: ray-head
    restart: unless-stopped
    ports:
      - "${RAY_DASHBOARD_PORT:-8265}:8265"
      - "${RAY_CLIENT_PORT:-10001}:10001"
    command: ray start --head --dashboard-host=0.0.0.0 --block
    shm_size: ${SHM_SIZE:-2gb}
    networks:
      - ray-network

  ray-worker-1:
    image: rayproject/ray:latest
    container_name: ray-worker-1
    restart: unless-stopped
    command: ray start --address=ray-head:6379 --block
    shm_size: ${SHM_SIZE:-2gb}
    depends_on:
      - ray-head
    networks:
      - ray-network

  ray-worker-2:
    image: rayproject/ray:latest
    container_name: ray-worker-2
    restart: unless-stopped
    command: ray start --address=ray-head:6379 --block
    shm_size: ${SHM_SIZE:-2gb}
    depends_on:
      - ray-head
    networks:
      - ray-network

networks:
  ray-network:
    driver: bridge
```

## .env

```bash
# Ray Cluster
RAY_DASHBOARD_PORT=8265
RAY_CLIENT_PORT=10001
SHM_SIZE=2gb
```

## Usage notes

- Docs: https://docs.ray.io/
- Ray Dashboard at http://localhost:8265 - jobs, actors, metrics
- Connect from Python: ray.init('ray://localhost:10001')
- Scale by adding more ray-worker-N services in compose file
- Use @ray.remote decorator for distributed functions
- Ray Tune, Ray Serve, and Ray Data for ML workflows

## Troubleshooting

- Workers failing to connect with 'redis.exceptions.ConnectionError': Ensure ray-head container is fully started before workers attempt connection, increase depends_on timeout
- Out of shared memory errors during large object operations: Increase SHM_SIZE environment variable beyond default 2GB based on your data sizes
- Ray Dashboard showing 'Cluster not found' or connection timeouts: Verify RAY_DASHBOARD_PORT is properly mapped and dashboard-host=0.0.0.0 is set in head node command
- Tasks hanging indefinitely without completion: Check worker logs for memory exhaustion or increase worker memory limits to handle task requirements
- Ray Client connection refused from external Python scripts: Confirm RAY_CLIENT_PORT mapping and use ray.init('ray://docker-host-ip:10001') with actual host IP
- Object serialization failures with custom classes: Ensure all custom modules are available on worker nodes or use Ray's runtime_env feature to distribute dependencies

