# HashiCorp Vault Secrets Management

> Secrets management with Vault HA cluster, Consul backend, and UI.

- Category: Security & Networking
- Difficulty: advanced
- Components: vault, consul, nginx, prometheus
- Tags: vault, hashicorp, secrets, encryption, pki
- Source: https://docker.recipes/security/vault-cluster

## Overview

HashiCorp Vault is an industry-leading secrets management platform designed to secure, store, and tightly control access to tokens, passwords, certificates, API keys, and other sensitive data. Originally developed by HashiCorp in 2015, Vault addresses the critical security challenge of secrets sprawl in modern infrastructure, where applications and systems require access to numerous credentials and encryption keys. Unlike traditional password managers or basic key stores, Vault provides dynamic secrets generation, automatic credential rotation, and fine-grained policy-based access controls that integrate with existing identity systems.

This production-grade stack combines Vault with HashiCorp Consul as a highly available storage backend, creating a resilient cluster that can survive individual node failures without losing access to critical secrets. Consul provides distributed consensus through the Raft algorithm and service discovery capabilities, while Grafana and Prometheus deliver comprehensive monitoring of vault operations, unsealing events, authentication attempts, and performance metrics. The three-node Vault cluster ensures no single point of failure, with automatic leadership election and data replication across all instances.

This configuration targets organizations implementing zero-trust security models, DevOps teams managing multi-cloud deployments, and enterprises requiring compliance with regulations like SOX, PCI-DSS, or GDPR. The combination of Vault's encryption-as-a-service capabilities with Consul's distributed architecture makes this stack particularly valuable for companies running microservices architectures where traditional secrets management approaches become unwieldy and insecure.

## Features

- High availability three-node Vault cluster with automatic failover and leader election
- Consul-backed distributed storage with Raft consensus for data consistency
- Dynamic secrets generation for databases, cloud providers, and SSH access
- Transit encryption engine for encryption-as-a-service operations
- PKI engine for automated certificate lifecycle management
- Policy-based access control with fine-grained permissions and audit logging
- Prometheus metrics collection for vault operations and cluster health
- Grafana dashboards for monitoring unsealing events, authentication patterns, and secret access

## Use cases

- Enterprise secrets management replacing hardcoded credentials in applications
- Database credential rotation for PostgreSQL, MySQL, and MongoDB instances
- PKI infrastructure for internal certificate authority and automated cert provisioning
- Multi-cloud API key management across AWS, Azure, and Google Cloud Platform
- Kubernetes secrets injection using vault-agent or CSI drivers
- Compliance environments requiring audit trails and encryption key management
- DevOps pipelines needing secure credential distribution to CI/CD systems

## Prerequisites

- Docker Engine 20.10+ with at least 4GB available memory for the complete stack
- Basic understanding of HashiCorp Vault concepts including unsealing and root tokens
- Network access to ports 8200 (Vault UI), 8500 (Consul UI), 9090 (Prometheus), and 3000 (Grafana)
- Familiarity with Vault CLI commands for initialization and configuration
- Understanding of high availability concepts and distributed systems basics

## docker-compose.yml

```yaml
services:
  consul:
    image: hashicorp/consul:latest
    command: agent -server -bootstrap-expect=1 -ui -client=0.0.0.0
    ports:
      - "8500:8500"
    volumes:
      - consul_data:/consul/data
    networks:
      - vault_net

  vault-1:
    image: hashicorp/vault:latest
    cap_add:
      - IPC_LOCK
    environment:
      - VAULT_ADDR=http://0.0.0.0:8200
      - VAULT_API_ADDR=http://vault-1:8200
      - VAULT_CLUSTER_ADDR=http://vault-1:8201
    volumes:
      - ./vault/config.hcl:/vault/config/config.hcl
      - vault_1:/vault/data
    command: server
    ports:
      - "8200:8200"
    depends_on:
      - consul
    networks:
      - vault_net

  vault-2:
    image: hashicorp/vault:latest
    cap_add:
      - IPC_LOCK
    environment:
      - VAULT_ADDR=http://0.0.0.0:8200
      - VAULT_API_ADDR=http://vault-2:8200
      - VAULT_CLUSTER_ADDR=http://vault-2:8201
    volumes:
      - ./vault/config.hcl:/vault/config/config.hcl
      - vault_2:/vault/data
    command: server
    depends_on:
      - consul
    networks:
      - vault_net

  vault-3:
    image: hashicorp/vault:latest
    cap_add:
      - IPC_LOCK
    environment:
      - VAULT_ADDR=http://0.0.0.0:8200
      - VAULT_API_ADDR=http://vault-3:8200
      - VAULT_CLUSTER_ADDR=http://vault-3:8201
    volumes:
      - ./vault/config.hcl:/vault/config/config.hcl
      - vault_3:/vault/data
    command: server
    depends_on:
      - consul
    networks:
      - vault_net

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - prometheus_data:/prometheus
    networks:
      - vault_net

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
    volumes:
      - grafana_data:/var/lib/grafana
    networks:
      - vault_net

volumes:
  consul_data:
  vault_1:
  vault_2:
  vault_3:
  prometheus_data:
  grafana_data:

networks:
  vault_net:
```

## .env

```bash
# Vault Cluster
GRAFANA_PASSWORD=secure_grafana_password

# Vault at http://localhost:8200
# Initialize with: vault operator init
```

## Usage notes

- Vault UI at http://localhost:8200
- Initialize with vault operator init
- Consul backend for HA storage
- PKI, transit, and KV engines
- Auto-unseal in production

## Troubleshooting

- Vault shows 'sealed' status after restart: Run 'vault operator unseal' with threshold number of unseal keys on each node
- Consul connection refused errors: Verify Consul container started successfully and is accessible on port 8500
- Permission denied when accessing vault data: Check that containers have IPC_LOCK capability and proper volume mounts
- Vault cluster split-brain condition: Restart all Vault nodes simultaneously to force re-election
- Prometheus not scraping vault metrics: Configure Vault telemetry in config.hcl and verify prometheus_retention_time setting
- Grafana showing no vault data: Import HashiCorp Vault dashboard templates and configure Prometheus data source correctly

