# NATS JetStream Cluster

> High-performance messaging system with JetStream persistence and streaming.

- Category: Message Queues & Brokers
- Difficulty: intermediate
- Components: nats, nats-box, prometheus, grafana
- Tags: nats, jetstream, messaging, streaming, high-performance
- Source: https://docker.recipes/messaging/nats-jetstream-cluster

## Overview

NATS is a high-performance, cloud-native messaging system originally developed by Derek Collison in 2010, designed to provide extremely fast and lightweight communication for distributed systems. Unlike traditional message brokers, NATS focuses on simplicity and speed, delivering sub-millisecond latency with minimal resource overhead. JetStream, introduced as NATS 2.2's persistence layer, adds durability and streaming capabilities while maintaining NATS' performance characteristics. This cluster configuration creates a three-node NATS JetStream cluster that provides both high availability and data persistence, eliminating the single point of failure inherent in standalone deployments. The integration with Prometheus captures detailed metrics about message throughput, consumer lag, and cluster health, while Grafana transforms this telemetry into visual dashboards for real-time monitoring. The nats-box utility container provides a complete CLI toolkit for stream management, consumer testing, and administrative operations. This stack particularly excels in scenarios requiring both real-time messaging performance and guaranteed message delivery, making it ideal for financial trading platforms, IoT data ingestion, and microservices architectures where message loss is unacceptable but traditional heavyweight brokers introduce too much latency.

## Features

- Three-node NATS cluster with automatic failover and split-brain protection
- JetStream persistence engine with configurable retention policies and storage limits
- Pull and push consumer patterns with acknowledgment-based delivery guarantees
- Stream replication across cluster nodes for data durability and availability
- Built-in monitoring endpoints exposing JetStream metrics and cluster topology
- Subject-based routing with wildcard support for flexible message distribution
- Queue groups for load balancing consumers across multiple application instances
- Real-time Grafana dashboards showing message rates, stream sizes, and consumer lag

## Use cases

- Financial trading systems requiring sub-millisecond message delivery with audit trails
- IoT sensor networks collecting telemetry data with guaranteed persistence
- Microservices architectures needing both request-reply and event streaming patterns
- Real-time chat applications with message history and offline delivery
- Log aggregation systems processing high-volume structured data streams
- Event-driven architectures implementing CQRS patterns with event sourcing
- Gaming backends managing player state synchronization and match coordination

## Prerequisites

- Minimum 768MB RAM available (256MB per NATS node plus monitoring stack)
- Docker Engine 20.10+ with Docker Compose v2 for multi-container orchestration
- Ports 4222, 8222, 9090, and 3000 available on the host system
- Basic understanding of pub/sub messaging patterns and stream processing concepts
- Familiarity with PromQL for customizing Prometheus monitoring queries
- SSD storage recommended for JetStream data volumes to ensure optimal performance

## docker-compose.yml

```yaml
services:
  nats1:
    image: nats:latest
    hostname: nats1
    ports:
      - "4222:4222"
      - "8222:8222"
    command: >
      -js
      -sd /data
      -cluster nats://0.0.0.0:6222
      -cluster_name nats-cluster
      -routes nats://nats2:6222,nats://nats3:6222
      -m 8222
    volumes:
      - nats1_data:/data
    networks:
      - nats-net
    restart: unless-stopped

  nats2:
    image: nats:latest
    hostname: nats2
    command: >
      -js
      -sd /data
      -cluster nats://0.0.0.0:6222
      -cluster_name nats-cluster
      -routes nats://nats1:6222,nats://nats3:6222
    volumes:
      - nats2_data:/data
    networks:
      - nats-net
    restart: unless-stopped

  nats3:
    image: nats:latest
    hostname: nats3
    command: >
      -js
      -sd /data
      -cluster nats://0.0.0.0:6222
      -cluster_name nats-cluster
      -routes nats://nats1:6222,nats://nats2:6222
    volumes:
      - nats3_data:/data
    networks:
      - nats-net
    restart: unless-stopped

  nats-box:
    image: natsio/nats-box:latest
    command: sleep infinity
    depends_on:
      - nats1
    networks:
      - nats-net

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus_data:/prometheus
    networks:
      - nats-net
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
    volumes:
      - grafana_data:/var/lib/grafana
    depends_on:
      - prometheus
    networks:
      - nats-net
    restart: unless-stopped

volumes:
  nats1_data:
  nats2_data:
  nats3_data:
  prometheus_data:
  grafana_data:

networks:
  nats-net:
    driver: bridge
```

## .env

```bash
# NATS Configuration
NATS_CLUSTER_NAME=nats-cluster

# Grafana
GRAFANA_PASSWORD=secure_grafana_password
```

## Usage notes

- NATS client connection at localhost:4222
- Monitoring endpoint at http://localhost:8222
- Use nats-box: docker compose exec nats-box nats
- JetStream enabled for persistence

## Troubleshooting

- Cluster formation fails with 'no route to host': Ensure all NATS containers can resolve each other's hostnames and verify the nats-net network is created properly
- JetStream consumers show increasing lag: Check if consumer acknowledgment timeouts are too short or if processing applications have crashed using 'nats consumer info'
- Memory usage grows unbounded: Configure stream retention policies with max_msgs or max_bytes limits to prevent infinite message accumulation
- Split-brain scenario with multiple leaders: Restart all NATS containers simultaneously to force cluster re-election and check for network partitioning issues
- Prometheus shows gaps in NATS metrics: Verify that port 8222 monitoring endpoint is accessible and confirm prometheus.yml includes all NATS node targets
- Messages lost during node restart: Ensure JetStream streams have replication factor set to 3 using 'nats stream add --replicas=3' for full cluster redundancy

