# NSQ

> Real-time distributed messaging platform.

- Category: Message Queues & Brokers
- Difficulty: intermediate
- Components: nsq
- Tags: nsq, distributed, messaging, golang
- Source: https://docker.recipes/messaging/nsq

## Overview

NSQ is a realtime distributed messaging platform developed by Bitly, designed to operate at scale and handle billions of messages per day. Built in Go, NSQ emphasizes simplicity, fault-tolerance, and high availability through its distributed topology that eliminates single points of failure. The platform consists of three core daemons: nsqd (message queue daemon), nsqlookupd (topology discovery service), and nsqadmin (web-based administration interface).

This Docker stack deploys a complete NSQ cluster with nsqlookupd providing service discovery on ports 4160/4161, nsqd handling message queuing on ports 4150/4151, and nsqadmin offering web-based monitoring and administration on port 4171. The architecture allows producers and consumers to discover each other dynamically through nsqlookupd, while nsqd nodes can be scaled horizontally without coordination. Messages are distributed across available nsqd instances, and the system continues operating even if individual components fail.

Developers building event-driven applications, DevOps teams implementing microservice communication, and organizations requiring reliable message delivery will find this NSQ deployment valuable. The platform excels in scenarios requiring guaranteed message delivery, real-time data processing pipelines, and distributed system coordination where traditional message brokers might introduce complexity or single points of failure.

## Features

- Distributed topology with automatic service discovery via nsqlookupd eliminating configuration management overhead
- Guaranteed message delivery with configurable retry mechanisms and dead letter queue support
- Memory and disk-based message persistence with configurable sync intervals for durability
- Built-in message deduplication and ordering guarantees within individual topics and channels
- HTTP and TCP protocol support for both publishing and consuming messages with JSON API
- Real-time web interface through nsqadmin for monitoring message rates, queue depths, and node health
- Automatic load balancing across multiple nsqd instances without manual partitioning
- Channel-based fan-out messaging allowing multiple consumer groups per topic

## Use cases

- Event-driven microservices architecture requiring reliable inter-service communication
- Real-time analytics pipelines processing user activity streams and application metrics
- Distributed job processing systems with task queuing and worker coordination
- IoT data ingestion platforms handling sensor data from thousands of devices
- E-commerce order processing workflows requiring guaranteed delivery and audit trails
- Chat applications and notification systems requiring real-time message distribution
- Log aggregation and processing systems collecting data from distributed applications

## Prerequisites

- Docker Engine 20.10+ and Docker Compose v2 for container orchestration support
- Minimum 512MB RAM available for the NSQ cluster with additional memory for message buffering
- Available ports 4150-4151 (nsqd), 4160-4161 (nsqlookupd), and 4171 (nsqadmin) on the host system
- Basic understanding of pub/sub messaging patterns and distributed systems concepts
- Network connectivity between containers and familiarity with NSQ's HTTP API for integration
- Understanding of Go-based application logging and debugging for troubleshooting NSQ components

## docker-compose.yml

```yaml
services:
  nsqlookupd:
    image: nsqio/nsq:latest
    container_name: nsqlookupd
    command: /nsqlookupd
    ports:
      - "4160:4160"
      - "4161:4161"

  nsqd:
    image: nsqio/nsq:latest
    container_name: nsqd
    command: /nsqd --lookupd-tcp-address=nsqlookupd:4160
    ports:
      - "4150:4150"
      - "4151:4151"
    depends_on:
      - nsqlookupd

  nsqadmin:
    image: nsqio/nsq:latest
    container_name: nsqadmin
    command: /nsqadmin --lookupd-http-address=nsqlookupd:4161
    ports:
      - "4171:4171"
    depends_on:
      - nsqlookupd
```

## .env

```bash
# No additional config needed
```

## Usage notes

- Docs: https://nsq.io/overview/design.html
- Admin UI at http://localhost:4171, nsqd on ports 4150 (TCP) / 4151 (HTTP)
- nsqlookupd handles service discovery on ports 4160/4161
- Publish: curl -d 'message' http://localhost:4151/pub?topic=test
- No single point of failure - fully distributed
- Written in Go - simple, performant, fault-tolerant

## Troubleshooting

- nsqd fails to connect to nsqlookupd: Verify nsqlookupd is running and accessible on port 4160, check Docker network connectivity between containers
- Messages not being delivered to consumers: Confirm topic and channel names match exactly, check nsqadmin interface for queue depth and consumer connections
- nsqadmin shows 'No nsqd nodes' error: Ensure nsqd has registered with nsqlookupd by checking logs, verify nsqlookupd HTTP address configuration
- High memory usage in nsqd: Adjust message retention settings and sync intervals, consider scaling to multiple nsqd instances to distribute load
- Connection refused on HTTP endpoints: Verify port mappings match container internal ports, check firewall rules and Docker network configuration
- Topics not appearing in nsqadmin: Publish at least one message to create the topic, ensure nsqd is properly connected to nsqlookupd for discovery

