docker.recipes

GlusterFS Distributed Storage

advanced

Scalable distributed file system with replicated volumes for high availability storage.

Overview

GlusterFS is an open-source distributed file system that aggregates storage from multiple servers into a single global namespace, providing scalability and high availability through replication and distribution. Originally developed by Gluster Inc. (acquired by Red Hat), it eliminates single points of failure by distributing data across multiple nodes and supports various volume types including replicated, distributed, and striped configurations. This stack combines GlusterFS with Heketi, a RESTful volume management interface that simplifies GlusterFS administration through API calls, making it particularly valuable for dynamic storage provisioning in containerized environments. Heketi acts as the management layer, handling volume creation, deletion, and topology management without requiring direct GlusterFS CLI interaction. Prometheus monitoring completes the stack by collecting metrics from both GlusterFS nodes and Heketi, providing visibility into storage performance, capacity utilization, and cluster health. This combination is ideal for organizations running containerized applications that require persistent, highly available storage with programmatic management capabilities, such as Kubernetes environments needing dynamic persistent volume provisioning or applications requiring distributed file access across multiple hosts.

Key Features

  • Three-node GlusterFS cluster with replica 3 configuration for maximum data redundancy
  • Heketi REST API on port 8080 for programmatic volume management and topology control
  • Automatic peer probe and trusted storage pool formation across gluster1, gluster2, and gluster3 nodes
  • FUSE-based client mounting support with native GlusterFS protocol access
  • Prometheus metrics collection for GlusterFS brick health, volume statistics, and Heketi API performance
  • Persistent volume mapping for GlusterFS configuration, logs, and data directories across all nodes
  • Fixed IP addressing (172.28.0.2-4) for consistent peer discovery and client mounting
  • Privileged container execution enabling FUSE filesystem operations and block device management

Common Use Cases

  • 1Kubernetes persistent volume backend with dynamic provisioning via Heketi integration
  • 2Shared storage for containerized applications requiring concurrent read/write access across multiple hosts
  • 3Backup and archive storage with built-in replication eliminating need for separate backup solutions
  • 4Media streaming platforms needing distributed storage for large file libraries with high availability
  • 5Development environments requiring shared code repositories and build artifacts across team members
  • 6Database clusters requiring shared storage for configuration files and backup repositories
  • 7Content management systems needing distributed file storage with automatic failover capabilities

Prerequisites

  • Minimum 4GB RAM per host (GlusterFS requires 1GB+ per node, Heketi 512MB, Prometheus 256MB)
  • Docker host with privileged container support for FUSE filesystem mounting
  • Available ports 8080 (Heketi API), 9090 (Prometheus), and 24007-24008 (GlusterFS daemon)
  • Understanding of GlusterFS volume types (replica, distribute, stripe) and brick management concepts
  • Basic knowledge of Heketi topology files and SSH key management for node authentication
  • Network configuration allowing inter-container communication on 172.28.0.0/16 subnet

For development & testing. Review security settings, change default credentials, and test thoroughly before production use. See Terms

docker-compose.yml

docker-compose.yml
1services: 
2  gluster1: 
3    image: gluster/gluster-centos:latest
4    hostname: gluster1
5    privileged: true
6    volumes: 
7      - gluster1_data:/data/glusterfs
8      - gluster1_etc:/etc/glusterfs
9      - gluster1_lib:/var/lib/glusterd
10      - gluster1_log:/var/log/glusterfs
11    networks: 
12      gluster-net: 
13        ipv4_address: 172.28.0.2
14    restart: unless-stopped
15
16  gluster2: 
17    image: gluster/gluster-centos:latest
18    hostname: gluster2
19    privileged: true
20    volumes: 
21      - gluster2_data:/data/glusterfs
22      - gluster2_etc:/etc/glusterfs
23      - gluster2_lib:/var/lib/glusterd
24      - gluster2_log:/var/log/glusterfs
25    networks: 
26      gluster-net: 
27        ipv4_address: 172.28.0.3
28    restart: unless-stopped
29
30  gluster3: 
31    image: gluster/gluster-centos:latest
32    hostname: gluster3
33    privileged: true
34    volumes: 
35      - gluster3_data:/data/glusterfs
36      - gluster3_etc:/etc/glusterfs
37      - gluster3_lib:/var/lib/glusterd
38      - gluster3_log:/var/log/glusterfs
39    networks: 
40      gluster-net: 
41        ipv4_address: 172.28.0.4
42    restart: unless-stopped
43
44  heketi: 
45    image: heketi/heketi:latest
46    ports: 
47      - "8080:8080"
48    environment: 
49      - HEKETI_EXECUTOR=ssh
50      - HEKETI_ADMIN_KEY=${HEKETI_ADMIN_KEY}
51    volumes: 
52      - heketi_config:/etc/heketi
53      - heketi_db:/var/lib/heketi
54    depends_on: 
55      - gluster1
56      - gluster2
57      - gluster3
58    networks: 
59      - gluster-net
60    restart: unless-stopped
61
62  prometheus: 
63    image: prom/prometheus:latest
64    ports: 
65      - "9090:9090"
66    volumes: 
67      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
68      - prometheus_data:/prometheus
69    networks: 
70      - gluster-net
71    restart: unless-stopped
72
73volumes: 
74  gluster1_data: 
75  gluster1_etc: 
76  gluster1_lib: 
77  gluster1_log: 
78  gluster2_data: 
79  gluster2_etc: 
80  gluster2_lib: 
81  gluster2_log: 
82  gluster3_data: 
83  gluster3_etc: 
84  gluster3_lib: 
85  gluster3_log: 
86  heketi_config: 
87  heketi_db: 
88  prometheus_data: 
89
90networks: 
91  gluster-net: 
92    driver: bridge
93    ipam: 
94      config: 
95        - subnet: 172.28.0.0/16

.env Template

.env
1# Heketi Configuration
2HEKETI_ADMIN_KEY=secure_heketi_admin_key
3
4# GlusterFS Volume Configuration
5REPLICA_COUNT=3
6VOLUME_NAME=gv0

Usage Notes

  1. 1Heketi API at http://localhost:8080
  2. 2Create volume: gluster volume create gv0 replica 3 gluster1:/data/glusterfs gluster2:/data/glusterfs gluster3:/data/glusterfs
  3. 3Mount: mount -t glusterfs gluster1:/gv0 /mnt/gluster
  4. 4Requires privileged containers for FUSE mounting

Individual Services(5 services)

Copy individual services to mix and match with your existing compose files.

gluster1
gluster1:
  image: gluster/gluster-centos:latest
  hostname: gluster1
  privileged: true
  volumes:
    - gluster1_data:/data/glusterfs
    - gluster1_etc:/etc/glusterfs
    - gluster1_lib:/var/lib/glusterd
    - gluster1_log:/var/log/glusterfs
  networks:
    gluster-net:
      ipv4_address: 172.28.0.2
  restart: unless-stopped
gluster2
gluster2:
  image: gluster/gluster-centos:latest
  hostname: gluster2
  privileged: true
  volumes:
    - gluster2_data:/data/glusterfs
    - gluster2_etc:/etc/glusterfs
    - gluster2_lib:/var/lib/glusterd
    - gluster2_log:/var/log/glusterfs
  networks:
    gluster-net:
      ipv4_address: 172.28.0.3
  restart: unless-stopped
gluster3
gluster3:
  image: gluster/gluster-centos:latest
  hostname: gluster3
  privileged: true
  volumes:
    - gluster3_data:/data/glusterfs
    - gluster3_etc:/etc/glusterfs
    - gluster3_lib:/var/lib/glusterd
    - gluster3_log:/var/log/glusterfs
  networks:
    gluster-net:
      ipv4_address: 172.28.0.4
  restart: unless-stopped
heketi
heketi:
  image: heketi/heketi:latest
  ports:
    - "8080:8080"
  environment:
    - HEKETI_EXECUTOR=ssh
    - HEKETI_ADMIN_KEY=${HEKETI_ADMIN_KEY}
  volumes:
    - heketi_config:/etc/heketi
    - heketi_db:/var/lib/heketi
  depends_on:
    - gluster1
    - gluster2
    - gluster3
  networks:
    - gluster-net
  restart: unless-stopped
prometheus
prometheus:
  image: prom/prometheus:latest
  ports:
    - "9090:9090"
  volumes:
    - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
    - prometheus_data:/prometheus
  networks:
    - gluster-net
  restart: unless-stopped

Quick Start

terminal
1# 1. Create the compose file
2cat > docker-compose.yml << 'EOF'
3services:
4  gluster1:
5    image: gluster/gluster-centos:latest
6    hostname: gluster1
7    privileged: true
8    volumes:
9      - gluster1_data:/data/glusterfs
10      - gluster1_etc:/etc/glusterfs
11      - gluster1_lib:/var/lib/glusterd
12      - gluster1_log:/var/log/glusterfs
13    networks:
14      gluster-net:
15        ipv4_address: 172.28.0.2
16    restart: unless-stopped
17
18  gluster2:
19    image: gluster/gluster-centos:latest
20    hostname: gluster2
21    privileged: true
22    volumes:
23      - gluster2_data:/data/glusterfs
24      - gluster2_etc:/etc/glusterfs
25      - gluster2_lib:/var/lib/glusterd
26      - gluster2_log:/var/log/glusterfs
27    networks:
28      gluster-net:
29        ipv4_address: 172.28.0.3
30    restart: unless-stopped
31
32  gluster3:
33    image: gluster/gluster-centos:latest
34    hostname: gluster3
35    privileged: true
36    volumes:
37      - gluster3_data:/data/glusterfs
38      - gluster3_etc:/etc/glusterfs
39      - gluster3_lib:/var/lib/glusterd
40      - gluster3_log:/var/log/glusterfs
41    networks:
42      gluster-net:
43        ipv4_address: 172.28.0.4
44    restart: unless-stopped
45
46  heketi:
47    image: heketi/heketi:latest
48    ports:
49      - "8080:8080"
50    environment:
51      - HEKETI_EXECUTOR=ssh
52      - HEKETI_ADMIN_KEY=${HEKETI_ADMIN_KEY}
53    volumes:
54      - heketi_config:/etc/heketi
55      - heketi_db:/var/lib/heketi
56    depends_on:
57      - gluster1
58      - gluster2
59      - gluster3
60    networks:
61      - gluster-net
62    restart: unless-stopped
63
64  prometheus:
65    image: prom/prometheus:latest
66    ports:
67      - "9090:9090"
68    volumes:
69      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
70      - prometheus_data:/prometheus
71    networks:
72      - gluster-net
73    restart: unless-stopped
74
75volumes:
76  gluster1_data:
77  gluster1_etc:
78  gluster1_lib:
79  gluster1_log:
80  gluster2_data:
81  gluster2_etc:
82  gluster2_lib:
83  gluster2_log:
84  gluster3_data:
85  gluster3_etc:
86  gluster3_lib:
87  gluster3_log:
88  heketi_config:
89  heketi_db:
90  prometheus_data:
91
92networks:
93  gluster-net:
94    driver: bridge
95    ipam:
96      config:
97        - subnet: 172.28.0.0/16
98EOF
99
100# 2. Create the .env file
101cat > .env << 'EOF'
102# Heketi Configuration
103HEKETI_ADMIN_KEY=secure_heketi_admin_key
104
105# GlusterFS Volume Configuration
106REPLICA_COUNT=3
107VOLUME_NAME=gv0
108EOF
109
110# 3. Start the services
111docker compose up -d
112
113# 4. View logs
114docker compose logs -f

One-Liner

Run this command to download and set up the recipe in one step:

terminal
1curl -fsSL https://docker.recipes/api/recipes/glusterfs-cluster/run | bash

Troubleshooting

  • Error 'Transport endpoint is not connected': Restart glusterd service in affected containers and re-run peer probe commands
  • Heketi returns 'Unable to create volume': Check GlusterFS brick space availability and ensure all nodes are in trusted pool
  • Mount fails with 'No such file or directory': Verify volume exists with 'gluster volume list' and volume is started
  • Prometheus shows no GlusterFS metrics: Install and configure gluster_exporter on each GlusterFS node for metrics collection
  • Volume creation hangs indefinitely: Check network connectivity between nodes and verify brick directories are accessible
  • Split-brain detected in logs: Use 'gluster volume heal' commands to resolve file conflicts and restore replica consistency

Community Notes

Loading...
Loading notes...

Download Recipe Kit

Get all files in a ready-to-deploy package

Includes docker-compose.yml, .env template, README, and license

Components

glusterfsheketiprometheus

Tags

#glusterfs#distributed-storage#ha#replicated#scalable

Category

Storage & Backup

Related Recipes

Ad Space