# GlusterFS Distributed Storage

> Scalable distributed file system with replicated volumes for high availability storage.

- Category: Storage & Backup
- Difficulty: advanced
- Components: glusterfs, heketi, prometheus
- Tags: glusterfs, distributed-storage, ha, replicated, scalable
- Source: https://docker.recipes/storage/glusterfs-cluster

## Overview

GlusterFS is an open-source distributed file system that aggregates storage from multiple servers into a single global namespace, providing scalability and high availability through replication and distribution. Originally developed by Gluster Inc. (acquired by Red Hat), it eliminates single points of failure by distributing data across multiple nodes and supports various volume types including replicated, distributed, and striped configurations. This stack combines GlusterFS with Heketi, a RESTful volume management interface that simplifies GlusterFS administration through API calls, making it particularly valuable for dynamic storage provisioning in containerized environments. Heketi acts as the management layer, handling volume creation, deletion, and topology management without requiring direct GlusterFS CLI interaction. Prometheus monitoring completes the stack by collecting metrics from both GlusterFS nodes and Heketi, providing visibility into storage performance, capacity utilization, and cluster health. This combination is ideal for organizations running containerized applications that require persistent, highly available storage with programmatic management capabilities, such as Kubernetes environments needing dynamic persistent volume provisioning or applications requiring distributed file access across multiple hosts.

## Features

- Three-node GlusterFS cluster with replica 3 configuration for maximum data redundancy
- Heketi REST API on port 8080 for programmatic volume management and topology control
- Automatic peer probe and trusted storage pool formation across gluster1, gluster2, and gluster3 nodes
- FUSE-based client mounting support with native GlusterFS protocol access
- Prometheus metrics collection for GlusterFS brick health, volume statistics, and Heketi API performance
- Persistent volume mapping for GlusterFS configuration, logs, and data directories across all nodes
- Fixed IP addressing (172.28.0.2-4) for consistent peer discovery and client mounting
- Privileged container execution enabling FUSE filesystem operations and block device management

## Use cases

- Kubernetes persistent volume backend with dynamic provisioning via Heketi integration
- Shared storage for containerized applications requiring concurrent read/write access across multiple hosts
- Backup and archive storage with built-in replication eliminating need for separate backup solutions
- Media streaming platforms needing distributed storage for large file libraries with high availability
- Development environments requiring shared code repositories and build artifacts across team members
- Database clusters requiring shared storage for configuration files and backup repositories
- Content management systems needing distributed file storage with automatic failover capabilities

## Prerequisites

- Minimum 4GB RAM per host (GlusterFS requires 1GB+ per node, Heketi 512MB, Prometheus 256MB)
- Docker host with privileged container support for FUSE filesystem mounting
- Available ports 8080 (Heketi API), 9090 (Prometheus), and 24007-24008 (GlusterFS daemon)
- Understanding of GlusterFS volume types (replica, distribute, stripe) and brick management concepts
- Basic knowledge of Heketi topology files and SSH key management for node authentication
- Network configuration allowing inter-container communication on 172.28.0.0/16 subnet

## docker-compose.yml

```yaml
services:
  gluster1:
    image: gluster/gluster-centos:latest
    hostname: gluster1
    privileged: true
    volumes:
      - gluster1_data:/data/glusterfs
      - gluster1_etc:/etc/glusterfs
      - gluster1_lib:/var/lib/glusterd
      - gluster1_log:/var/log/glusterfs
    networks:
      gluster-net:
        ipv4_address: 172.28.0.2
    restart: unless-stopped

  gluster2:
    image: gluster/gluster-centos:latest
    hostname: gluster2
    privileged: true
    volumes:
      - gluster2_data:/data/glusterfs
      - gluster2_etc:/etc/glusterfs
      - gluster2_lib:/var/lib/glusterd
      - gluster2_log:/var/log/glusterfs
    networks:
      gluster-net:
        ipv4_address: 172.28.0.3
    restart: unless-stopped

  gluster3:
    image: gluster/gluster-centos:latest
    hostname: gluster3
    privileged: true
    volumes:
      - gluster3_data:/data/glusterfs
      - gluster3_etc:/etc/glusterfs
      - gluster3_lib:/var/lib/glusterd
      - gluster3_log:/var/log/glusterfs
    networks:
      gluster-net:
        ipv4_address: 172.28.0.4
    restart: unless-stopped

  heketi:
    image: heketi/heketi:latest
    ports:
      - "8080:8080"
    environment:
      - HEKETI_EXECUTOR=ssh
      - HEKETI_ADMIN_KEY=${HEKETI_ADMIN_KEY}
    volumes:
      - heketi_config:/etc/heketi
      - heketi_db:/var/lib/heketi
    depends_on:
      - gluster1
      - gluster2
      - gluster3
    networks:
      - gluster-net
    restart: unless-stopped

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus_data:/prometheus
    networks:
      - gluster-net
    restart: unless-stopped

volumes:
  gluster1_data:
  gluster1_etc:
  gluster1_lib:
  gluster1_log:
  gluster2_data:
  gluster2_etc:
  gluster2_lib:
  gluster2_log:
  gluster3_data:
  gluster3_etc:
  gluster3_lib:
  gluster3_log:
  heketi_config:
  heketi_db:
  prometheus_data:

networks:
  gluster-net:
    driver: bridge
    ipam:
      config:
        - subnet: 172.28.0.0/16
```

## .env

```bash
# Heketi Configuration
HEKETI_ADMIN_KEY=secure_heketi_admin_key

# GlusterFS Volume Configuration
REPLICA_COUNT=3
VOLUME_NAME=gv0
```

## Usage notes

- Heketi API at http://localhost:8080
- Create volume: gluster volume create gv0 replica 3 gluster1:/data/glusterfs gluster2:/data/glusterfs gluster3:/data/glusterfs
- Mount: mount -t glusterfs gluster1:/gv0 /mnt/gluster
- Requires privileged containers for FUSE mounting

## Troubleshooting

- Error 'Transport endpoint is not connected': Restart glusterd service in affected containers and re-run peer probe commands
- Heketi returns 'Unable to create volume': Check GlusterFS brick space availability and ensure all nodes are in trusted pool
- Mount fails with 'No such file or directory': Verify volume exists with 'gluster volume list' and volume is started
- Prometheus shows no GlusterFS metrics: Install and configure gluster_exporter on each GlusterFS node for metrics collection
- Volume creation hangs indefinitely: Check network connectivity between nodes and verify brick directories are accessible
- Split-brain detected in logs: Use 'gluster volume heal' commands to resolve file conflicts and restore replica consistency

