# Rook-Ceph Development Stack

> Cloud-native storage orchestrator for Kubernetes with Ceph for block, object, and file storage.

- Category: Storage & Backup
- Difficulty: advanced
- Components: ceph-mon, ceph-osd, ceph-mgr, ceph-rgw, prometheus
- Tags: rook, ceph, kubernetes, block-storage, object-storage
- Source: https://docker.recipes/storage/rook-ceph-dev

## Overview

Ceph is a distributed storage system that provides unified block, object, and file storage through a cluster of storage nodes. Originally developed by Sage Weil at UC Santa Cruz, Ceph eliminates single points of failure through its CRUSH algorithm that distributes data across multiple nodes with built-in replication. The system consists of several key daemons: ceph-mon maintains cluster maps and consensus, ceph-osd handles actual data storage and replication, ceph-mgr provides monitoring and management interfaces, and ceph-rgw offers S3-compatible object storage gateway.

This development stack combines all essential Ceph components with Prometheus monitoring to create a complete software-defined storage environment. The ceph-mon daemon acts as the cluster's brain, maintaining authoritative copies of cluster maps, while multiple ceph-osd instances simulate distributed storage nodes for redundancy testing. The ceph-mgr daemon enables the web dashboard and exposes metrics, while ceph-rgw provides object storage compatibility with AWS S3 APIs. Prometheus integration captures detailed storage metrics, IOPS performance, and cluster health indicators.

Storage architects, Kubernetes platform engineers, and infrastructure developers building cloud-native applications will find this stack invaluable for prototyping distributed storage solutions. This combination enables testing of storage failure scenarios, capacity planning, and integration patterns before deploying to production Rook-Ceph clusters. The setup provides hands-on experience with Ceph's self-healing capabilities and performance characteristics under various load conditions.

## Features

- Multi-daemon Ceph cluster with monitor, manager, and dual OSD configuration for replication testing
- Built-in Ceph Dashboard on port 7000 for visual cluster management and health monitoring
- S3-compatible RADOS Gateway supporting AWS SDK integration and bucket operations
- Native Prometheus metrics export from ceph-mgr for storage performance monitoring
- CRUSH map simulation with configurable placement groups and replication rules
- Block device support through RBD (RADOS Block Device) for persistent volume testing
- Object storage pools with configurable erasure coding and compression algorithms
- Cluster health monitoring with automatic OSD failure detection and recovery simulation

## Use cases

- Kubernetes persistent volume development with CSI driver integration testing
- S3-compatible object storage backend for applications requiring AWS S3 API compatibility
- Storage failure scenario testing and disaster recovery procedure validation
- Multi-tenant storage solutions with isolated pools and user access controls
- Container registry storage backend using Ceph's object storage capabilities
- Development of storage-intensive applications requiring block and object storage
- Performance benchmarking and capacity planning for production Ceph deployments

## Prerequisites

- Docker host with minimum 4GB RAM and 20GB available disk space
- Basic understanding of distributed storage concepts and RADOS architecture
- Familiarity with Ceph cluster components and their interdependencies
- Network configuration allowing container communication on 172.29.0.0/16 subnet
- Available ports 7000 (dashboard), 8080 (RGW), 9090 (Prometheus), 9283 (metrics)

## docker-compose.yml

```yaml
services:
  ceph-mon:
    image: ceph/ceph:latest
    hostname: ceph-mon
    environment:
      - MON_IP=172.29.0.2
      - CEPH_PUBLIC_NETWORK=172.29.0.0/16
    volumes:
      - ceph_etc:/etc/ceph
      - ceph_mon_data:/var/lib/ceph/mon
    command: mon
    networks:
      ceph-net:
        ipv4_address: 172.29.0.2
    restart: unless-stopped

  ceph-mgr:
    image: ceph/ceph:latest
    hostname: ceph-mgr
    ports:
      - "7000:7000"  # Dashboard
      - "9283:9283"  # Prometheus metrics
    environment:
      - CEPH_PUBLIC_NETWORK=172.29.0.0/16
    volumes:
      - ceph_etc:/etc/ceph
      - ceph_mgr_data:/var/lib/ceph/mgr
    command: mgr
    depends_on:
      - ceph-mon
    networks:
      - ceph-net
    restart: unless-stopped

  ceph-osd-1:
    image: ceph/ceph:latest
    hostname: ceph-osd-1
    privileged: true
    environment:
      - OSD_DEVICE=/dev/sdb
      - CEPH_PUBLIC_NETWORK=172.29.0.0/16
    volumes:
      - ceph_etc:/etc/ceph
      - ceph_osd1_data:/var/lib/ceph/osd
      - /dev:/dev
    command: osd_directory
    depends_on:
      - ceph-mon
    networks:
      - ceph-net
    restart: unless-stopped

  ceph-osd-2:
    image: ceph/ceph:latest
    hostname: ceph-osd-2
    privileged: true
    environment:
      - OSD_DEVICE=/dev/sdc
      - CEPH_PUBLIC_NETWORK=172.29.0.0/16
    volumes:
      - ceph_etc:/etc/ceph
      - ceph_osd2_data:/var/lib/ceph/osd
      - /dev:/dev
    command: osd_directory
    depends_on:
      - ceph-mon
    networks:
      - ceph-net
    restart: unless-stopped

  ceph-rgw:
    image: ceph/ceph:latest
    hostname: ceph-rgw
    ports:
      - "8080:8080"
    volumes:
      - ceph_etc:/etc/ceph
      - ceph_rgw_data:/var/lib/ceph/radosgw
    command: rgw
    depends_on:
      - ceph-mon
      - ceph-osd-1
      - ceph-osd-2
    networks:
      - ceph-net
    restart: unless-stopped

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus_data:/prometheus
    networks:
      - ceph-net
    restart: unless-stopped

volumes:
  ceph_etc:
  ceph_mon_data:
  ceph_mgr_data:
  ceph_osd1_data:
  ceph_osd2_data:
  ceph_rgw_data:
  prometheus_data:

networks:
  ceph-net:
    driver: bridge
    ipam:
      config:
        - subnet: 172.29.0.0/16
```

## .env

```bash
# Ceph Configuration
CEPH_FSID=$(uuidgen)
CEPH_MON_KEY=$(ceph-authtool --gen-print-key)
CEPH_ADMIN_KEY=$(ceph-authtool --gen-print-key)

# Dashboard
CEPH_DASHBOARD_USER=admin
CEPH_DASHBOARD_PASSWORD=secure_ceph_password
```

## Usage notes

- Ceph Dashboard at http://localhost:7000
- S3-compatible gateway at http://localhost:8080
- Prometheus metrics at http://localhost:9283
- Development setup - not for production

## Troubleshooting

- MON_IP connectivity issues: Verify ceph-mon container can bind to 172.29.0.2 and check network subnet conflicts
- OSD activation failures: Ensure privileged mode is enabled and /dev directory mount is accessible for device simulation
- Dashboard login authentication: Default admin user credentials may need manual setup through ceph-mgr container exec
- RGW S3 access denied errors: Create RGW user credentials using radosgw-admin commands within the ceph-rgw container
- Prometheus scraping failures: Confirm ceph-mgr prometheus module is enabled and metrics endpoint responds on port 9283
- Cluster health HEALTH_WARN: Check OSD count meets minimum replication requirements and placement group distribution

