docker.recipes

Rook-Ceph Development Stack

advanced

Cloud-native storage orchestrator for Kubernetes with Ceph for block, object, and file storage.

Overview

Ceph is a distributed storage system that provides unified block, object, and file storage through a cluster of storage nodes. Originally developed by Sage Weil at UC Santa Cruz, Ceph eliminates single points of failure through its CRUSH algorithm that distributes data across multiple nodes with built-in replication. The system consists of several key daemons: ceph-mon maintains cluster maps and consensus, ceph-osd handles actual data storage and replication, ceph-mgr provides monitoring and management interfaces, and ceph-rgw offers S3-compatible object storage gateway. This development stack combines all essential Ceph components with Prometheus monitoring to create a complete software-defined storage environment. The ceph-mon daemon acts as the cluster's brain, maintaining authoritative copies of cluster maps, while multiple ceph-osd instances simulate distributed storage nodes for redundancy testing. The ceph-mgr daemon enables the web dashboard and exposes metrics, while ceph-rgw provides object storage compatibility with AWS S3 APIs. Prometheus integration captures detailed storage metrics, IOPS performance, and cluster health indicators. Storage architects, Kubernetes platform engineers, and infrastructure developers building cloud-native applications will find this stack invaluable for prototyping distributed storage solutions. This combination enables testing of storage failure scenarios, capacity planning, and integration patterns before deploying to production Rook-Ceph clusters. The setup provides hands-on experience with Ceph's self-healing capabilities and performance characteristics under various load conditions.

Key Features

  • Multi-daemon Ceph cluster with monitor, manager, and dual OSD configuration for replication testing
  • Built-in Ceph Dashboard on port 7000 for visual cluster management and health monitoring
  • S3-compatible RADOS Gateway supporting AWS SDK integration and bucket operations
  • Native Prometheus metrics export from ceph-mgr for storage performance monitoring
  • CRUSH map simulation with configurable placement groups and replication rules
  • Block device support through RBD (RADOS Block Device) for persistent volume testing
  • Object storage pools with configurable erasure coding and compression algorithms
  • Cluster health monitoring with automatic OSD failure detection and recovery simulation

Common Use Cases

  • 1Kubernetes persistent volume development with CSI driver integration testing
  • 2S3-compatible object storage backend for applications requiring AWS S3 API compatibility
  • 3Storage failure scenario testing and disaster recovery procedure validation
  • 4Multi-tenant storage solutions with isolated pools and user access controls
  • 5Container registry storage backend using Ceph's object storage capabilities
  • 6Development of storage-intensive applications requiring block and object storage
  • 7Performance benchmarking and capacity planning for production Ceph deployments

Prerequisites

  • Docker host with minimum 4GB RAM and 20GB available disk space
  • Basic understanding of distributed storage concepts and RADOS architecture
  • Familiarity with Ceph cluster components and their interdependencies
  • Network configuration allowing container communication on 172.29.0.0/16 subnet
  • Available ports 7000 (dashboard), 8080 (RGW), 9090 (Prometheus), 9283 (metrics)

For development & testing. Review security settings, change default credentials, and test thoroughly before production use. See Terms

docker-compose.yml

docker-compose.yml
1services: 
2  ceph-mon: 
3    image: ceph/ceph:latest
4    hostname: ceph-mon
5    environment: 
6      - MON_IP=172.29.0.2
7      - CEPH_PUBLIC_NETWORK=172.29.0.0/16
8    volumes: 
9      - ceph_etc:/etc/ceph
10      - ceph_mon_data:/var/lib/ceph/mon
11    command: mon
12    networks: 
13      ceph-net: 
14        ipv4_address: 172.29.0.2
15    restart: unless-stopped
16
17  ceph-mgr: 
18    image: ceph/ceph:latest
19    hostname: ceph-mgr
20    ports: 
21      - "7000:7000"  # Dashboard
22      - "9283:9283"  # Prometheus metrics
23    environment: 
24      - CEPH_PUBLIC_NETWORK=172.29.0.0/16
25    volumes: 
26      - ceph_etc:/etc/ceph
27      - ceph_mgr_data:/var/lib/ceph/mgr
28    command: mgr
29    depends_on: 
30      - ceph-mon
31    networks: 
32      - ceph-net
33    restart: unless-stopped
34
35  ceph-osd-1: 
36    image: ceph/ceph:latest
37    hostname: ceph-osd-1
38    privileged: true
39    environment: 
40      - OSD_DEVICE=/dev/sdb
41      - CEPH_PUBLIC_NETWORK=172.29.0.0/16
42    volumes: 
43      - ceph_etc:/etc/ceph
44      - ceph_osd1_data:/var/lib/ceph/osd
45      - /dev:/dev
46    command: osd_directory
47    depends_on: 
48      - ceph-mon
49    networks: 
50      - ceph-net
51    restart: unless-stopped
52
53  ceph-osd-2: 
54    image: ceph/ceph:latest
55    hostname: ceph-osd-2
56    privileged: true
57    environment: 
58      - OSD_DEVICE=/dev/sdc
59      - CEPH_PUBLIC_NETWORK=172.29.0.0/16
60    volumes: 
61      - ceph_etc:/etc/ceph
62      - ceph_osd2_data:/var/lib/ceph/osd
63      - /dev:/dev
64    command: osd_directory
65    depends_on: 
66      - ceph-mon
67    networks: 
68      - ceph-net
69    restart: unless-stopped
70
71  ceph-rgw: 
72    image: ceph/ceph:latest
73    hostname: ceph-rgw
74    ports: 
75      - "8080:8080"
76    volumes: 
77      - ceph_etc:/etc/ceph
78      - ceph_rgw_data:/var/lib/ceph/radosgw
79    command: rgw
80    depends_on: 
81      - ceph-mon
82      - ceph-osd-1
83      - ceph-osd-2
84    networks: 
85      - ceph-net
86    restart: unless-stopped
87
88  prometheus: 
89    image: prom/prometheus:latest
90    ports: 
91      - "9090:9090"
92    volumes: 
93      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
94      - prometheus_data:/prometheus
95    networks: 
96      - ceph-net
97    restart: unless-stopped
98
99volumes: 
100  ceph_etc: 
101  ceph_mon_data: 
102  ceph_mgr_data: 
103  ceph_osd1_data: 
104  ceph_osd2_data: 
105  ceph_rgw_data: 
106  prometheus_data: 
107
108networks: 
109  ceph-net: 
110    driver: bridge
111    ipam: 
112      config: 
113        - subnet: 172.29.0.0/16

.env Template

.env
1# Ceph Configuration
2CEPH_FSID=$(uuidgen)
3CEPH_MON_KEY=$(ceph-authtool --gen-print-key)
4CEPH_ADMIN_KEY=$(ceph-authtool --gen-print-key)
5
6# Dashboard
7CEPH_DASHBOARD_USER=admin
8CEPH_DASHBOARD_PASSWORD=secure_ceph_password

Usage Notes

  1. 1Ceph Dashboard at http://localhost:7000
  2. 2S3-compatible gateway at http://localhost:8080
  3. 3Prometheus metrics at http://localhost:9283
  4. 4Development setup - not for production

Individual Services(6 services)

Copy individual services to mix and match with your existing compose files.

ceph-mon
ceph-mon:
  image: ceph/ceph:latest
  hostname: ceph-mon
  environment:
    - MON_IP=172.29.0.2
    - CEPH_PUBLIC_NETWORK=172.29.0.0/16
  volumes:
    - ceph_etc:/etc/ceph
    - ceph_mon_data:/var/lib/ceph/mon
  command: mon
  networks:
    ceph-net:
      ipv4_address: 172.29.0.2
  restart: unless-stopped
ceph-mgr
ceph-mgr:
  image: ceph/ceph:latest
  hostname: ceph-mgr
  ports:
    - "7000:7000"
    - "9283:9283"
  environment:
    - CEPH_PUBLIC_NETWORK=172.29.0.0/16
  volumes:
    - ceph_etc:/etc/ceph
    - ceph_mgr_data:/var/lib/ceph/mgr
  command: mgr
  depends_on:
    - ceph-mon
  networks:
    - ceph-net
  restart: unless-stopped
ceph-osd-1
ceph-osd-1:
  image: ceph/ceph:latest
  hostname: ceph-osd-1
  privileged: true
  environment:
    - OSD_DEVICE=/dev/sdb
    - CEPH_PUBLIC_NETWORK=172.29.0.0/16
  volumes:
    - ceph_etc:/etc/ceph
    - ceph_osd1_data:/var/lib/ceph/osd
    - /dev:/dev
  command: osd_directory
  depends_on:
    - ceph-mon
  networks:
    - ceph-net
  restart: unless-stopped
ceph-osd-2
ceph-osd-2:
  image: ceph/ceph:latest
  hostname: ceph-osd-2
  privileged: true
  environment:
    - OSD_DEVICE=/dev/sdc
    - CEPH_PUBLIC_NETWORK=172.29.0.0/16
  volumes:
    - ceph_etc:/etc/ceph
    - ceph_osd2_data:/var/lib/ceph/osd
    - /dev:/dev
  command: osd_directory
  depends_on:
    - ceph-mon
  networks:
    - ceph-net
  restart: unless-stopped
ceph-rgw
ceph-rgw:
  image: ceph/ceph:latest
  hostname: ceph-rgw
  ports:
    - "8080:8080"
  volumes:
    - ceph_etc:/etc/ceph
    - ceph_rgw_data:/var/lib/ceph/radosgw
  command: rgw
  depends_on:
    - ceph-mon
    - ceph-osd-1
    - ceph-osd-2
  networks:
    - ceph-net
  restart: unless-stopped
prometheus
prometheus:
  image: prom/prometheus:latest
  ports:
    - "9090:9090"
  volumes:
    - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
    - prometheus_data:/prometheus
  networks:
    - ceph-net
  restart: unless-stopped

Quick Start

terminal
1# 1. Create the compose file
2cat > docker-compose.yml << 'EOF'
3services:
4  ceph-mon:
5    image: ceph/ceph:latest
6    hostname: ceph-mon
7    environment:
8      - MON_IP=172.29.0.2
9      - CEPH_PUBLIC_NETWORK=172.29.0.0/16
10    volumes:
11      - ceph_etc:/etc/ceph
12      - ceph_mon_data:/var/lib/ceph/mon
13    command: mon
14    networks:
15      ceph-net:
16        ipv4_address: 172.29.0.2
17    restart: unless-stopped
18
19  ceph-mgr:
20    image: ceph/ceph:latest
21    hostname: ceph-mgr
22    ports:
23      - "7000:7000"  # Dashboard
24      - "9283:9283"  # Prometheus metrics
25    environment:
26      - CEPH_PUBLIC_NETWORK=172.29.0.0/16
27    volumes:
28      - ceph_etc:/etc/ceph
29      - ceph_mgr_data:/var/lib/ceph/mgr
30    command: mgr
31    depends_on:
32      - ceph-mon
33    networks:
34      - ceph-net
35    restart: unless-stopped
36
37  ceph-osd-1:
38    image: ceph/ceph:latest
39    hostname: ceph-osd-1
40    privileged: true
41    environment:
42      - OSD_DEVICE=/dev/sdb
43      - CEPH_PUBLIC_NETWORK=172.29.0.0/16
44    volumes:
45      - ceph_etc:/etc/ceph
46      - ceph_osd1_data:/var/lib/ceph/osd
47      - /dev:/dev
48    command: osd_directory
49    depends_on:
50      - ceph-mon
51    networks:
52      - ceph-net
53    restart: unless-stopped
54
55  ceph-osd-2:
56    image: ceph/ceph:latest
57    hostname: ceph-osd-2
58    privileged: true
59    environment:
60      - OSD_DEVICE=/dev/sdc
61      - CEPH_PUBLIC_NETWORK=172.29.0.0/16
62    volumes:
63      - ceph_etc:/etc/ceph
64      - ceph_osd2_data:/var/lib/ceph/osd
65      - /dev:/dev
66    command: osd_directory
67    depends_on:
68      - ceph-mon
69    networks:
70      - ceph-net
71    restart: unless-stopped
72
73  ceph-rgw:
74    image: ceph/ceph:latest
75    hostname: ceph-rgw
76    ports:
77      - "8080:8080"
78    volumes:
79      - ceph_etc:/etc/ceph
80      - ceph_rgw_data:/var/lib/ceph/radosgw
81    command: rgw
82    depends_on:
83      - ceph-mon
84      - ceph-osd-1
85      - ceph-osd-2
86    networks:
87      - ceph-net
88    restart: unless-stopped
89
90  prometheus:
91    image: prom/prometheus:latest
92    ports:
93      - "9090:9090"
94    volumes:
95      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
96      - prometheus_data:/prometheus
97    networks:
98      - ceph-net
99    restart: unless-stopped
100
101volumes:
102  ceph_etc:
103  ceph_mon_data:
104  ceph_mgr_data:
105  ceph_osd1_data:
106  ceph_osd2_data:
107  ceph_rgw_data:
108  prometheus_data:
109
110networks:
111  ceph-net:
112    driver: bridge
113    ipam:
114      config:
115        - subnet: 172.29.0.0/16
116EOF
117
118# 2. Create the .env file
119cat > .env << 'EOF'
120# Ceph Configuration
121CEPH_FSID=$(uuidgen)
122CEPH_MON_KEY=$(ceph-authtool --gen-print-key)
123CEPH_ADMIN_KEY=$(ceph-authtool --gen-print-key)
124
125# Dashboard
126CEPH_DASHBOARD_USER=admin
127CEPH_DASHBOARD_PASSWORD=secure_ceph_password
128EOF
129
130# 3. Start the services
131docker compose up -d
132
133# 4. View logs
134docker compose logs -f

One-Liner

Run this command to download and set up the recipe in one step:

terminal
1curl -fsSL https://docker.recipes/api/recipes/rook-ceph-dev/run | bash

Troubleshooting

  • MON_IP connectivity issues: Verify ceph-mon container can bind to 172.29.0.2 and check network subnet conflicts
  • OSD activation failures: Ensure privileged mode is enabled and /dev directory mount is accessible for device simulation
  • Dashboard login authentication: Default admin user credentials may need manual setup through ceph-mgr container exec
  • RGW S3 access denied errors: Create RGW user credentials using radosgw-admin commands within the ceph-rgw container
  • Prometheus scraping failures: Confirm ceph-mgr prometheus module is enabled and metrics endpoint responds on port 9283
  • Cluster health HEALTH_WARN: Check OSD count meets minimum replication requirements and placement group distribution

Community Notes

Loading...
Loading notes...

Download Recipe Kit

Get all files in a ready-to-deploy package

Includes docker-compose.yml, .env template, README, and license

Components

ceph-monceph-osdceph-mgrceph-rgwprometheus

Tags

#rook#ceph#kubernetes#block-storage#object-storage

Category

Storage & Backup

Related Recipes

Ad Space