# etcd Cluster

> Distributed reliable key-value store for critical data coordination.

- Category: Database Stacks
- Difficulty: intermediate
- Components: etcd
- Tags: etcd, distributed, key-value, coordination, kubernetes
- Source: https://docker.recipes/databases/etcd

## Overview

etcd is a distributed, strongly consistent key-value store originally developed by CoreOS and now maintained by the Cloud Native Computing Foundation. Built on the Raft consensus algorithm, etcd serves as the backbone for critical distributed system coordination, most notably as the primary datastore for Kubernetes cluster state. Its design prioritizes consistency and reliability over performance, making it ideal for storing configuration data, metadata, and coordination information that distributed systems depend on. This three-node etcd cluster configuration implements a production-grade distributed setup that can tolerate one node failure while maintaining cluster availability and data consistency. The cluster uses the Raft consensus protocol to elect leaders, replicate data, and ensure that all nodes maintain identical state, providing the strong consistency guarantees that critical infrastructure components require. Each node participates in both client serving (port 2379) and peer communication (port 2380) to maintain cluster membership and data synchronization. Organizations running Kubernetes clusters, microservices architectures requiring distributed coordination, or any system needing reliable configuration management will benefit from this etcd cluster setup. The three-node configuration strikes the optimal balance between fault tolerance and resource efficiency, providing high availability while minimizing the overhead associated with larger cluster sizes that offer diminishing returns in most scenarios.

## Features

- Raft consensus algorithm ensuring strong consistency across all cluster members
- Multi-version concurrency control (MVCC) for efficient read operations and historical queries
- Watch API for real-time notification of key changes and cluster state updates
- Lease-based TTL system for automatic key expiration and resource cleanup
- ACID transactions supporting conditional operations and atomic multi-key updates
- Built-in snapshot and restore capabilities for backup and disaster recovery
- gRPC-based API with both HTTP/JSON and binary protocol support
- Cluster membership management with automatic leader election and failover

## Use cases

- Kubernetes cluster datastore for storing cluster state, configuration, and secrets
- Microservices service discovery and configuration management across distributed environments
- Distributed system leader election for coordinating singleton processes
- Feature flag and configuration distribution with real-time updates across services
- Distributed locking and coordination for preventing race conditions in multi-instance applications
- Container orchestration metadata storage for custom scheduling and deployment systems
- CI/CD pipeline coordination and state management across distributed build agents

## Prerequisites

- Minimum 1.5GB RAM total (512MB per etcd node) for stable cluster operation
- Understanding of distributed systems concepts including consensus algorithms and CAP theorem
- Network connectivity allowing communication on ports 2379 (client) and 2380 (peer)
- Basic familiarity with etcdctl command-line tool for cluster management and debugging
- Knowledge of Raft consensus behavior including leader election and split-brain scenarios
- SSD storage recommended for optimal write performance and reduced commit latency

## docker-compose.yml

```yaml
services:
  etcd1:
    image: quay.io/coreos/etcd:latest
    container_name: etcd1
    command:
      - etcd
      - --name=etcd1
      - --initial-advertise-peer-urls=http://etcd1:2380
      - --listen-peer-urls=http://0.0.0.0:2380
      - --listen-client-urls=http://0.0.0.0:2379
      - --advertise-client-urls=http://etcd1:2379
      - --initial-cluster=etcd1=http://etcd1:2380,etcd2=http://etcd2:2380,etcd3=http://etcd3:2380
      - --initial-cluster-state=new
    volumes:
      - etcd1_data:/etcd-data
    ports:
      - "2379:2379"
    networks:
      - etcd-network

  etcd2:
    image: quay.io/coreos/etcd:latest
    container_name: etcd2
    command:
      - etcd
      - --name=etcd2
      - --initial-advertise-peer-urls=http://etcd2:2380
      - --listen-peer-urls=http://0.0.0.0:2380
      - --listen-client-urls=http://0.0.0.0:2379
      - --advertise-client-urls=http://etcd2:2379
      - --initial-cluster=etcd1=http://etcd1:2380,etcd2=http://etcd2:2380,etcd3=http://etcd3:2380
      - --initial-cluster-state=new
    volumes:
      - etcd2_data:/etcd-data
    networks:
      - etcd-network

  etcd3:
    image: quay.io/coreos/etcd:latest
    container_name: etcd3
    command:
      - etcd
      - --name=etcd3
      - --initial-advertise-peer-urls=http://etcd3:2380
      - --listen-peer-urls=http://0.0.0.0:2380
      - --listen-client-urls=http://0.0.0.0:2379
      - --advertise-client-urls=http://etcd3:2379
      - --initial-cluster=etcd1=http://etcd1:2380,etcd2=http://etcd2:2380,etcd3=http://etcd3:2380
      - --initial-cluster-state=new
    volumes:
      - etcd3_data:/etcd-data
    networks:
      - etcd-network

volumes:
  etcd1_data:
  etcd2_data:
  etcd3_data:

networks:
  etcd-network:
    driver: bridge
```

## .env

```bash
# etcd cluster configuration
```

## Usage notes

- Docs: https://etcd.io/docs/
- Client API on port 2379 | Peer port 2380
- Use etcdctl: ETCDCTL_API=3 etcdctl --endpoints=localhost:2379 put key value
- Raft consensus protocol - requires 3+ nodes for HA
- Used by Kubernetes for cluster state storage
- Monitor: etcdctl endpoint health --cluster

## Troubleshooting

- etcd cluster unhealthy after startup: Verify all nodes can reach each other on port 2380 and check Docker network connectivity
- Leader election failures or frequent re-elections: Increase heartbeat and election timeouts, check for network latency between nodes
- Database space exceeded errors: Run 'etcdctl defrag' on each node and consider enabling auto-compaction
- Split-brain scenarios with multiple leaders: Ensure odd number of nodes and verify --initial-cluster configuration matches across all members
- High memory usage and slow queries: Enable periodic compaction and check for excessive watch subscriptions
- Authentication failures after cluster restart: Verify RBAC settings and ensure auth tokens are properly configured across all nodes

