# SeaweedFS

> Fast distributed storage system.

- Category: Storage & Backup
- Difficulty: intermediate
- Components: seaweedfs-master, seaweedfs-volume
- Tags: seaweedfs, distributed, s3, fast
- Source: https://docker.recipes/storage/seaweedfs

## Overview

SeaweedFS is a distributed object storage system designed for handling billions of small files with high performance and low latency. Originally created by Chris Lu, SeaweedFS follows Facebook's Haystack design philosophy, storing small files directly without requiring a separate metadata database for file locations. The system uses an innovative approach where the master server manages volume locations and file IDs, while volume servers handle the actual file storage with O(1) disk seek complexity.

This Docker stack combines the seaweedfs-master node for cluster coordination and metadata management with seaweedfs-volume servers for data storage, plus a filer component that provides POSIX and S3-compatible interfaces. The master server assigns file IDs and tracks volume locations across the cluster, while volume servers store files in append-only mode for maximum write performance. The filer acts as a bridge, translating traditional file system operations and S3 API calls into SeaweedFS's native operations.

This configuration is ideal for organizations dealing with massive collections of small to medium-sized files like images, documents, or media assets that need fast access times and horizontal scalability. Unlike traditional distributed file systems that struggle with small file performance, SeaweedFS excels at scenarios requiring millions of file operations per second while maintaining strong consistency and providing both object storage and file system interfaces.

## Features

- O(1) file lookup performance using volume-based architecture without central metadata bottlenecks
- Dual API support providing both S3-compatible object storage and POSIX file system interfaces
- Master-directed file placement with automatic volume allocation and load balancing
- Append-only volume storage optimized for write-heavy workloads with configurable replication
- Built-in HTTP interface for direct file access without requiring separate gateway services
- Elastic volume management supporting up to 100 volumes per server with dynamic scaling
- Cross-datacenter replication with rack-aware placement for high availability deployments
- Integrated web UI for cluster monitoring and volume management through master interface

## Use cases

- Content delivery networks storing millions of images and media files with fast retrieval requirements
- Document management systems handling large collections of PDFs, spreadsheets, and office files
- Photo sharing platforms and social media applications with high-volume image upload workflows
- Backup and archival systems requiring S3-compatible storage with better small file performance
- Development environments needing local object storage for testing cloud-native applications
- Video surveillance systems storing thousands of small video clips with rapid access needs
- E-commerce platforms managing product images and digital assets across multiple regions

## Prerequisites

- Minimum 2GB RAM for master server and 4GB per volume server for production deployments
- Docker Engine 20.10+ with Docker Compose v2 for container orchestration support
- Available ports 9333, 19333 for master, 8080, 18080 for volume, and 8888, 18888 for filer
- Understanding of object storage concepts and S3 API for application integration
- Basic knowledge of distributed systems for configuring replication and rack topology
- Sufficient disk space on Docker host as volume data persists in seaweed_data volume

## docker-compose.yml

```yaml
services:
  master:
    image: chrislusf/seaweedfs:latest
    container_name: seaweedfs-master
    restart: unless-stopped
    command: master -ip=master -port=9333
    ports:
      - "9333:9333"
      - "19333:19333"
    networks:
      - seaweedfs

  volume:
    image: chrislusf/seaweedfs:latest
    container_name: seaweedfs-volume
    restart: unless-stopped
    command: volume -mserver=master:9333 -port=8080 -ip=volume -max=100
    volumes:
      - seaweed_data:/data
    ports:
      - "8080:8080"
      - "18080:18080"
    depends_on:
      - master
    networks:
      - seaweedfs

  filer:
    image: chrislusf/seaweedfs:latest
    container_name: seaweedfs-filer
    restart: unless-stopped
    command: filer -master=master:9333 -port=8888
    ports:
      - "8888:8888"
      - "18888:18888"
    depends_on:
      - master
      - volume
    networks:
      - seaweedfs

volumes:
  seaweed_data:

networks:
  seaweedfs:
    driver: bridge
```

## .env

```bash
# Scale by adding more volume servers
```

## Usage notes

- Docs: https://github.com/seaweedfs/seaweedfs/wiki
- Master UI at http://localhost:9333 - cluster management
- Filer at http://localhost:8888 - S3/POSIX interface
- Optimized for small files (images, PDFs) - O(1) lookup
- Scale by adding volume servers: -max=100 sets max volumes per server
- S3 API: weed s3 -filer=localhost:8888 -port=8333

## Troubleshooting

- Master not accessible at localhost:9333: Check if master container started successfully and port binding conflicts with existing services
- Volume server shows 'cannot connect to master': Verify master container is running and network connectivity between master and volume containers
- Filer returns 'no volume server found': Ensure volume server registered with master by checking master UI and volume container logs
- S3 operations fail with connection errors: Start S3 gateway with 'docker exec seaweedfs-filer weed s3 -filer=localhost:8888' command
- Files not persisting after container restart: Verify seaweed_data volume mount and check volume server data directory permissions
- High memory usage on volume server: Reduce max volumes per server from 100 or increase container memory limits for production use

