# NATS + JetStream + Monitoring

> Cloud native messaging system with persistence.

- Category: Message Queues & Brokers
- Difficulty: intermediate
- Components: nats, nats-surveyor, prometheus, grafana
- Tags: nats, messaging, jetstream, cloud-native, pub-sub
- Source: https://docker.recipes/messaging/nats-complete

## Overview

NATS is a high-performance, cloud-native messaging system originally developed by Derek Collison in 2010, designed to be simple, secure, and performant for modern distributed applications. Unlike traditional message brokers that focus on complex routing and delivery guarantees, NATS prioritizes speed and simplicity, making it ideal for microservices architectures, IoT systems, and real-time applications where low latency is critical.

This stack combines NATS with JetStream persistence capabilities, NATS Surveyor for metrics collection, Prometheus for time-series monitoring, and Grafana for visualization. JetStream transforms NATS from a pure fire-and-forget messaging system into one with durable streams, consumer groups, and exactly-once delivery semantics. NATS Surveyor acts as a specialized metrics exporter that understands NATS internals, feeding connection counts, message rates, and JetStream consumer lag into Prometheus for alerting and historical analysis.

Development teams building event-driven architectures, platform engineers managing microservices communication, and organizations migrating from heavier message brokers like RabbitMQ will find this stack particularly valuable. The monitoring layer provides crucial visibility into message throughput, consumer health, and system performance that's essential for production NATS deployments, especially when using JetStream's persistence features for critical business events.

## Features

- JetStream persistence with configurable retention policies and replay capabilities
- Pub/Sub messaging with subject-based routing and wildcard subscriptions
- Request/Reply pattern for synchronous communication between services
- Queue groups for load balancing messages across consumer instances
- NATS Surveyor metrics collection with account-level observability
- Built-in key-value store using JetStream for configuration and state management
- WebSocket support for browser-based clients and real-time web applications
- Multi-tenancy through NATS accounts with isolated message spaces

## Use cases

- Microservices event bus for order processing and inventory management systems
- IoT telemetry collection with JetStream persistence for offline device recovery
- Real-time chat applications using NATS WebSocket gateway and subject filtering
- Financial trading systems requiring ultra-low latency market data distribution
- DevOps automation pipelines with reliable job queuing and status updates
- Gaming backends for matchmaking, leaderboards, and real-time player interactions
- Monitoring agent coordination across distributed infrastructure deployments

## Prerequisites

- Minimum 512MB RAM available (256MB for NATS, 256MB for monitoring stack)
- Ports 3000, 4222, 7777, 8222, and 9090 available on the host system
- Basic understanding of pub/sub messaging patterns and subject hierarchies
- Familiarity with Prometheus metrics and PromQL query language
- Docker Compose 3.8+ with volume and network support
- Understanding of JetStream concepts like streams, consumers, and acknowledgments

## docker-compose.yml

```yaml
services:
  nats:
    image: nats:latest
    command:
      - "--jetstream"
      - "--store_dir=/data"
      - "--http_port=8222"
    volumes:
      - nats-data:/data
    ports:
      - "4222:4222"
      - "8222:8222"
    networks:
      - nats-network
    restart: unless-stopped

  nats-surveyor:
    image: natsio/nats-surveyor:latest
    command:
      - "-s=nats://nats:4222"
      - "--accounts"
      - "--observe=*"
    ports:
      - "7777:7777"
    depends_on:
      - nats
    networks:
      - nats-network
    restart: unless-stopped

  prometheus:
    image: prom/prometheus:latest
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus
    ports:
      - "9090:9090"
    depends_on:
      - nats-surveyor
    networks:
      - nats-network
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
    volumes:
      - grafana-data:/var/lib/grafana
    ports:
      - "3000:3000"
    depends_on:
      - prometheus
    networks:
      - nats-network
    restart: unless-stopped

volumes:
  nats-data:
  prometheus-data:
  grafana-data:

networks:
  nats-network:
    driver: bridge
```

## .env

```bash
# NATS
GRAFANA_PASSWORD=secure_grafana_password

# NATS at nats://localhost:4222
# Monitoring at http://localhost:8222
```

## Usage notes

- NATS at nats://localhost:4222
- Monitoring at http://localhost:8222
- Grafana at http://localhost:3000
- JetStream for persistence
- Built-in key-value store

## Troubleshooting

- Error 'nats: no servers available for connection': Check that NATS container is running and port 4222 is accessible
- JetStream 'insufficient resources' error: Increase memory limits or reduce stream retention settings in NATS configuration
- NATS Surveyor showing 0 connections: Verify the connection string 'nats://nats:4222' and ensure containers are on the same network
- Grafana showing 'no data points' for NATS metrics: Check Prometheus targets page and verify nats-surveyor:7777 is being scraped successfully
- High memory usage in NATS container: Review JetStream stream configurations and set appropriate max_age or max_msgs limits
- Consumer lag alerts firing: Check if consumer applications are processing messages and consider scaling consumer instances

