etcd Cluster

intermediate

Distributed reliable key-value store for critical data coordination.

Overview

etcd is a distributed, strongly consistent key-value store originally developed by CoreOS and now maintained by the Cloud Native Computing Foundation. Built on the Raft consensus algorithm, etcd serves as the backbone for critical distributed system coordination, most notably as the primary datastore for Kubernetes cluster state. Its design prioritizes consistency and reliability over performance, making it ideal for storing configuration data, metadata, and coordination information that distributed systems depend on. This three-node etcd cluster configuration implements a production-grade distributed setup that can tolerate one node failure while maintaining cluster availability and data consistency. The cluster uses the Raft consensus protocol to elect leaders, replicate data, and ensure that all nodes maintain identical state, providing the strong consistency guarantees that critical infrastructure components require. Each node participates in both client serving (port 2379) and peer communication (port 2380) to maintain cluster membership and data synchronization. Organizations running Kubernetes clusters, microservices architectures requiring distributed coordination, or any system needing reliable configuration management will benefit from this etcd cluster setup. The three-node configuration strikes the optimal balance between fault tolerance and resource efficiency, providing high availability while minimizing the overhead associated with larger cluster sizes that offer diminishing returns in most scenarios.

Key Features

Raft consensus algorithm ensuring strong consistency across all cluster members
Multi-version concurrency control (MVCC) for efficient read operations and historical queries
Watch API for real-time notification of key changes and cluster state updates
Lease-based TTL system for automatic key expiration and resource cleanup
ACID transactions supporting conditional operations and atomic multi-key updates
Built-in snapshot and restore capabilities for backup and disaster recovery
gRPC-based API with both HTTP/JSON and binary protocol support
Cluster membership management with automatic leader election and failover

Common Use Cases

1Kubernetes cluster datastore for storing cluster state, configuration, and secrets
2Microservices service discovery and configuration management across distributed environments
3Distributed system leader election for coordinating singleton processes
4Feature flag and configuration distribution with real-time updates across services
5Distributed locking and coordination for preventing race conditions in multi-instance applications
6Container orchestration metadata storage for custom scheduling and deployment systems
7CI/CD pipeline coordination and state management across distributed build agents

Prerequisites

Minimum 1.5GB RAM total (512MB per etcd node) for stable cluster operation
Understanding of distributed systems concepts including consensus algorithms and CAP theorem
Network connectivity allowing communication on ports 2379 (client) and 2380 (peer)
Basic familiarity with etcdctl command-line tool for cluster management and debugging
Knowledge of Raft consensus behavior including leader election and split-brain scenarios
SSD storage recommended for optimal write performance and reduced commit latency

For development & testing. Review security settings, change default credentials, and test thoroughly before production use. See Terms

docker-compose.yml

1services: 
2  etcd1: 
3    image: quay.io/coreos/etcd:latest
4    container_name: etcd1
5    command: 
6      - etcd
7      - --name=etcd1
8      - --initial-advertise-peer-urls=http://etcd1:2380
9      - --listen-peer-urls=http://0.0.0.0:2380
10      - --listen-client-urls=http://0.0.0.0:2379
11      - --advertise-client-urls=http://etcd1:2379
12      - --initial-cluster=etcd1=http://etcd1:2380,etcd2=http://etcd2:2380,etcd3=http://etcd3:2380
13      - --initial-cluster-state=new
14    volumes: 
15      - etcd1_data:/etcd-data
16    ports: 
17      - "2379:2379"
18    networks: 
19      - etcd-network
20
21  etcd2: 
22    image: quay.io/coreos/etcd:latest
23    container_name: etcd2
24    command: 
25      - etcd
26      - --name=etcd2
27      - --initial-advertise-peer-urls=http://etcd2:2380
28      - --listen-peer-urls=http://0.0.0.0:2380
29      - --listen-client-urls=http://0.0.0.0:2379
30      - --advertise-client-urls=http://etcd2:2379
31      - --initial-cluster=etcd1=http://etcd1:2380,etcd2=http://etcd2:2380,etcd3=http://etcd3:2380
32      - --initial-cluster-state=new
33    volumes: 
34      - etcd2_data:/etcd-data
35    networks: 
36      - etcd-network
37
38  etcd3: 
39    image: quay.io/coreos/etcd:latest
40    container_name: etcd3
41    command: 
42      - etcd
43      - --name=etcd3
44      - --initial-advertise-peer-urls=http://etcd3:2380
45      - --listen-peer-urls=http://0.0.0.0:2380
46      - --listen-client-urls=http://0.0.0.0:2379
47      - --advertise-client-urls=http://etcd3:2379
48      - --initial-cluster=etcd1=http://etcd1:2380,etcd2=http://etcd2:2380,etcd3=http://etcd3:2380
49      - --initial-cluster-state=new
50    volumes: 
51      - etcd3_data:/etcd-data
52    networks: 
53      - etcd-network
54
55volumes: 
56  etcd1_data: 
57  etcd2_data: 
58  etcd3_data: 
59
60networks: 
61  etcd-network: 
62    driver: bridge

.env Template

.env

1# etcd cluster configuration

Usage Notes

1Docs: https://etcd.io/docs/
2Client API on port 2379 | Peer port 2380
3Use etcdctl: ETCDCTL_API=3 etcdctl --endpoints=localhost:2379 put key value
4Raft consensus protocol - requires 3+ nodes for HA
5Used by Kubernetes for cluster state storage
6Monitor: etcdctl endpoint health --cluster

Individual Services(3 services)

Copy individual services to mix and match with your existing compose files.

etcd1

etcd1:
  image: quay.io/coreos/etcd:latest
  container_name: etcd1
  command:
    - etcd
    - "--name=etcd1"
    - "--initial-advertise-peer-urls=http://etcd1:2380"
    - "--listen-peer-urls=http://0.0.0.0:2380"
    - "--listen-client-urls=http://0.0.0.0:2379"
    - "--advertise-client-urls=http://etcd1:2379"
    - "--initial-cluster=etcd1=http://etcd1:2380,etcd2=http://etcd2:2380,etcd3=http://etcd3:2380"
    - "--initial-cluster-state=new"
  volumes:
    - etcd1_data:/etcd-data
  ports:
    - "2379:2379"
  networks:
    - etcd-network

etcd2

etcd2:
  image: quay.io/coreos/etcd:latest
  container_name: etcd2
  command:
    - etcd
    - "--name=etcd2"
    - "--initial-advertise-peer-urls=http://etcd2:2380"
    - "--listen-peer-urls=http://0.0.0.0:2380"
    - "--listen-client-urls=http://0.0.0.0:2379"
    - "--advertise-client-urls=http://etcd2:2379"
    - "--initial-cluster=etcd1=http://etcd1:2380,etcd2=http://etcd2:2380,etcd3=http://etcd3:2380"
    - "--initial-cluster-state=new"
  volumes:
    - etcd2_data:/etcd-data
  networks:
    - etcd-network

etcd3

etcd3:
  image: quay.io/coreos/etcd:latest
  container_name: etcd3
  command:
    - etcd
    - "--name=etcd3"
    - "--initial-advertise-peer-urls=http://etcd3:2380"
    - "--listen-peer-urls=http://0.0.0.0:2380"
    - "--listen-client-urls=http://0.0.0.0:2379"
    - "--advertise-client-urls=http://etcd3:2379"
    - "--initial-cluster=etcd1=http://etcd1:2380,etcd2=http://etcd2:2380,etcd3=http://etcd3:2380"
    - "--initial-cluster-state=new"
  volumes:
    - etcd3_data:/etcd-data
  networks:
    - etcd-network

Quick Start

terminal

1# 1. Create the compose file
2cat > docker-compose.yml << 'EOF'
3services:
4  etcd1:
5    image: quay.io/coreos/etcd:latest
6    container_name: etcd1
7    command:
8      - etcd
9      - --name=etcd1
10      - --initial-advertise-peer-urls=http://etcd1:2380
11      - --listen-peer-urls=http://0.0.0.0:2380
12      - --listen-client-urls=http://0.0.0.0:2379
13      - --advertise-client-urls=http://etcd1:2379
14      - --initial-cluster=etcd1=http://etcd1:2380,etcd2=http://etcd2:2380,etcd3=http://etcd3:2380
15      - --initial-cluster-state=new
16    volumes:
17      - etcd1_data:/etcd-data
18    ports:
19      - "2379:2379"
20    networks:
21      - etcd-network
22
23  etcd2:
24    image: quay.io/coreos/etcd:latest
25    container_name: etcd2
26    command:
27      - etcd
28      - --name=etcd2
29      - --initial-advertise-peer-urls=http://etcd2:2380
30      - --listen-peer-urls=http://0.0.0.0:2380
31      - --listen-client-urls=http://0.0.0.0:2379
32      - --advertise-client-urls=http://etcd2:2379
33      - --initial-cluster=etcd1=http://etcd1:2380,etcd2=http://etcd2:2380,etcd3=http://etcd3:2380
34      - --initial-cluster-state=new
35    volumes:
36      - etcd2_data:/etcd-data
37    networks:
38      - etcd-network
39
40  etcd3:
41    image: quay.io/coreos/etcd:latest
42    container_name: etcd3
43    command:
44      - etcd
45      - --name=etcd3
46      - --initial-advertise-peer-urls=http://etcd3:2380
47      - --listen-peer-urls=http://0.0.0.0:2380
48      - --listen-client-urls=http://0.0.0.0:2379
49      - --advertise-client-urls=http://etcd3:2379
50      - --initial-cluster=etcd1=http://etcd1:2380,etcd2=http://etcd2:2380,etcd3=http://etcd3:2380
51      - --initial-cluster-state=new
52    volumes:
53      - etcd3_data:/etcd-data
54    networks:
55      - etcd-network
56
57volumes:
58  etcd1_data:
59  etcd2_data:
60  etcd3_data:
61
62networks:
63  etcd-network:
64    driver: bridge
65EOF
66
67# 2. Create the .env file
68cat > .env << 'EOF'
69# etcd cluster configuration
70EOF
71
72# 3. Start the services
73docker compose up -d
74
75# 4. View logs
76docker compose logs -f

One-Liner

Run this command to download and set up the recipe in one step:

terminal

1curl -fsSL https://docker.recipes/api/recipes/etcd/run | bash

Troubleshooting

etcd cluster unhealthy after startup: Verify all nodes can reach each other on port 2380 and check Docker network connectivity
Leader election failures or frequent re-elections: Increase heartbeat and election timeouts, check for network latency between nodes
Database space exceeded errors: Run 'etcdctl defrag' on each node and consider enabling auto-compaction
Split-brain scenarios with multiple leaders: Ensure odd number of nodes and verify --initial-cluster configuration matches across all members
High memory usage and slow queries: Enable periodic compaction and check for excessive watch subscriptions
Authentication failures after cluster restart: Verify RBAC settings and ensure auth tokens are properly configured across all nodes

Community Notes

Loading notes...

Download Recipe Kit

Get all files in a ready-to-deploy package

Includes docker-compose.yml, .env template, README, and license

Components

etcd

Related Recipes

Ad Space

Shortcuts: C CopyF FavoriteD Download