SeaweedFS

intermediate

Fast distributed storage system.

Overview

SeaweedFS is a distributed object storage system designed for handling billions of small files with high performance and low latency. Originally created by Chris Lu, SeaweedFS follows Facebook's Haystack design philosophy, storing small files directly without requiring a separate metadata database for file locations. The system uses an innovative approach where the master server manages volume locations and file IDs, while volume servers handle the actual file storage with O(1) disk seek complexity. This Docker stack combines the seaweedfs-master node for cluster coordination and metadata management with seaweedfs-volume servers for data storage, plus a filer component that provides POSIX and S3-compatible interfaces. The master server assigns file IDs and tracks volume locations across the cluster, while volume servers store files in append-only mode for maximum write performance. The filer acts as a bridge, translating traditional file system operations and S3 API calls into SeaweedFS's native operations. This configuration is ideal for organizations dealing with massive collections of small to medium-sized files like images, documents, or media assets that need fast access times and horizontal scalability. Unlike traditional distributed file systems that struggle with small file performance, SeaweedFS excels at scenarios requiring millions of file operations per second while maintaining strong consistency and providing both object storage and file system interfaces.

Key Features

O(1) file lookup performance using volume-based architecture without central metadata bottlenecks
Dual API support providing both S3-compatible object storage and POSIX file system interfaces
Master-directed file placement with automatic volume allocation and load balancing
Append-only volume storage optimized for write-heavy workloads with configurable replication
Built-in HTTP interface for direct file access without requiring separate gateway services
Elastic volume management supporting up to 100 volumes per server with dynamic scaling
Cross-datacenter replication with rack-aware placement for high availability deployments
Integrated web UI for cluster monitoring and volume management through master interface

Common Use Cases

1Content delivery networks storing millions of images and media files with fast retrieval requirements
2Document management systems handling large collections of PDFs, spreadsheets, and office files
3Photo sharing platforms and social media applications with high-volume image upload workflows
4Backup and archival systems requiring S3-compatible storage with better small file performance
5Development environments needing local object storage for testing cloud-native applications
6Video surveillance systems storing thousands of small video clips with rapid access needs
7E-commerce platforms managing product images and digital assets across multiple regions

Prerequisites

Minimum 2GB RAM for master server and 4GB per volume server for production deployments
Docker Engine 20.10+ with Docker Compose v2 for container orchestration support
Available ports 9333, 19333 for master, 8080, 18080 for volume, and 8888, 18888 for filer
Understanding of object storage concepts and S3 API for application integration
Basic knowledge of distributed systems for configuring replication and rack topology
Sufficient disk space on Docker host as volume data persists in seaweed_data volume

For development & testing. Review security settings, change default credentials, and test thoroughly before production use. See Terms

docker-compose.yml

1services: 
2  master: 
3    image: chrislusf/seaweedfs:latest
4    container_name: seaweedfs-master
5    restart: unless-stopped
6    command: master -ip=master -port=9333
7    ports: 
8      - "9333:9333"
9      - "19333:19333"
10    networks: 
11      - seaweedfs
12
13  volume: 
14    image: chrislusf/seaweedfs:latest
15    container_name: seaweedfs-volume
16    restart: unless-stopped
17    command: volume -mserver=master:9333 -port=8080 -ip=volume -max=100
18    volumes: 
19      - seaweed_data:/data
20    ports: 
21      - "8080:8080"
22      - "18080:18080"
23    depends_on: 
24      - master
25    networks: 
26      - seaweedfs
27
28  filer: 
29    image: chrislusf/seaweedfs:latest
30    container_name: seaweedfs-filer
31    restart: unless-stopped
32    command: filer -master=master:9333 -port=8888
33    ports: 
34      - "8888:8888"
35      - "18888:18888"
36    depends_on: 
37      - master
38      - volume
39    networks: 
40      - seaweedfs
41
42volumes: 
43  seaweed_data: 
44
45networks: 
46  seaweedfs: 
47    driver: bridge

.env Template

.env

1# Scale by adding more volume servers

Usage Notes

1Docs: https://github.com/seaweedfs/seaweedfs/wiki
2Master UI at http://localhost:9333 - cluster management
3Filer at http://localhost:8888 - S3/POSIX interface
4Optimized for small files (images, PDFs) - O(1) lookup
5Scale by adding volume servers: -max=100 sets max volumes per server
6S3 API: weed s3 -filer=localhost:8888 -port=8333

Individual Services(3 services)

Copy individual services to mix and match with your existing compose files.

master

master:
  image: chrislusf/seaweedfs:latest
  container_name: seaweedfs-master
  restart: unless-stopped
  command: master -ip=master -port=9333
  ports:
    - "9333:9333"
    - "19333:19333"
  networks:
    - seaweedfs

volume

volume:
  image: chrislusf/seaweedfs:latest
  container_name: seaweedfs-volume
  restart: unless-stopped
  command: volume -mserver=master:9333 -port=8080 -ip=volume -max=100
  volumes:
    - seaweed_data:/data
  ports:
    - "8080:8080"
    - "18080:18080"
  depends_on:
    - master
  networks:
    - seaweedfs

filer

filer:
  image: chrislusf/seaweedfs:latest
  container_name: seaweedfs-filer
  restart: unless-stopped
  command: filer -master=master:9333 -port=8888
  ports:
    - "8888:8888"
    - "18888:18888"
  depends_on:
    - master
    - volume
  networks:
    - seaweedfs

Quick Start

terminal

1# 1. Create the compose file
2cat > docker-compose.yml << 'EOF'
3services:
4  master:
5    image: chrislusf/seaweedfs:latest
6    container_name: seaweedfs-master
7    restart: unless-stopped
8    command: master -ip=master -port=9333
9    ports:
10      - "9333:9333"
11      - "19333:19333"
12    networks:
13      - seaweedfs
14
15  volume:
16    image: chrislusf/seaweedfs:latest
17    container_name: seaweedfs-volume
18    restart: unless-stopped
19    command: volume -mserver=master:9333 -port=8080 -ip=volume -max=100
20    volumes:
21      - seaweed_data:/data
22    ports:
23      - "8080:8080"
24      - "18080:18080"
25    depends_on:
26      - master
27    networks:
28      - seaweedfs
29
30  filer:
31    image: chrislusf/seaweedfs:latest
32    container_name: seaweedfs-filer
33    restart: unless-stopped
34    command: filer -master=master:9333 -port=8888
35    ports:
36      - "8888:8888"
37      - "18888:18888"
38    depends_on:
39      - master
40      - volume
41    networks:
42      - seaweedfs
43
44volumes:
45  seaweed_data:
46
47networks:
48  seaweedfs:
49    driver: bridge
50EOF
51
52# 2. Create the .env file
53cat > .env << 'EOF'
54# Scale by adding more volume servers
55EOF
56
57# 3. Start the services
58docker compose up -d
59
60# 4. View logs
61docker compose logs -f

One-Liner

Run this command to download and set up the recipe in one step:

terminal

1curl -fsSL https://docker.recipes/api/recipes/seaweedfs/run | bash

Troubleshooting

Master not accessible at localhost:9333: Check if master container started successfully and port binding conflicts with existing services
Volume server shows 'cannot connect to master': Verify master container is running and network connectivity between master and volume containers
Filer returns 'no volume server found': Ensure volume server registered with master by checking master UI and volume container logs
S3 operations fail with connection errors: Start S3 gateway with 'docker exec seaweedfs-filer weed s3 -filer=localhost:8888' command
Files not persisting after container restart: Verify seaweed_data volume mount and check volume server data directory permissions
High memory usage on volume server: Reduce max volumes per server from 100 or increase container memory limits for production use

Community Notes

Loading notes...

Download Recipe Kit

Get all files in a ready-to-deploy package

Includes docker-compose.yml, .env template, README, and license

Components

seaweedfs-masterseaweedfs-volume

Related Recipes

Ad Space

Shortcuts: C CopyF FavoriteD Download