docker.recipes

NSQ

intermediate

Real-time distributed messaging platform.

Overview

NSQ is a realtime distributed messaging platform developed by Bitly, designed to operate at scale and handle billions of messages per day. Built in Go, NSQ emphasizes simplicity, fault-tolerance, and high availability through its distributed topology that eliminates single points of failure. The platform consists of three core daemons: nsqd (message queue daemon), nsqlookupd (topology discovery service), and nsqadmin (web-based administration interface). This Docker stack deploys a complete NSQ cluster with nsqlookupd providing service discovery on ports 4160/4161, nsqd handling message queuing on ports 4150/4151, and nsqadmin offering web-based monitoring and administration on port 4171. The architecture allows producers and consumers to discover each other dynamically through nsqlookupd, while nsqd nodes can be scaled horizontally without coordination. Messages are distributed across available nsqd instances, and the system continues operating even if individual components fail. Developers building event-driven applications, DevOps teams implementing microservice communication, and organizations requiring reliable message delivery will find this NSQ deployment valuable. The platform excels in scenarios requiring guaranteed message delivery, real-time data processing pipelines, and distributed system coordination where traditional message brokers might introduce complexity or single points of failure.

Key Features

  • Distributed topology with automatic service discovery via nsqlookupd eliminating configuration management overhead
  • Guaranteed message delivery with configurable retry mechanisms and dead letter queue support
  • Memory and disk-based message persistence with configurable sync intervals for durability
  • Built-in message deduplication and ordering guarantees within individual topics and channels
  • HTTP and TCP protocol support for both publishing and consuming messages with JSON API
  • Real-time web interface through nsqadmin for monitoring message rates, queue depths, and node health
  • Automatic load balancing across multiple nsqd instances without manual partitioning
  • Channel-based fan-out messaging allowing multiple consumer groups per topic

Common Use Cases

  • 1Event-driven microservices architecture requiring reliable inter-service communication
  • 2Real-time analytics pipelines processing user activity streams and application metrics
  • 3Distributed job processing systems with task queuing and worker coordination
  • 4IoT data ingestion platforms handling sensor data from thousands of devices
  • 5E-commerce order processing workflows requiring guaranteed delivery and audit trails
  • 6Chat applications and notification systems requiring real-time message distribution
  • 7Log aggregation and processing systems collecting data from distributed applications

Prerequisites

  • Docker Engine 20.10+ and Docker Compose v2 for container orchestration support
  • Minimum 512MB RAM available for the NSQ cluster with additional memory for message buffering
  • Available ports 4150-4151 (nsqd), 4160-4161 (nsqlookupd), and 4171 (nsqadmin) on the host system
  • Basic understanding of pub/sub messaging patterns and distributed systems concepts
  • Network connectivity between containers and familiarity with NSQ's HTTP API for integration
  • Understanding of Go-based application logging and debugging for troubleshooting NSQ components

For development & testing. Review security settings, change default credentials, and test thoroughly before production use. See Terms

docker-compose.yml

docker-compose.yml
1services: 
2  nsqlookupd: 
3    image: nsqio/nsq:latest
4    container_name: nsqlookupd
5    command: /nsqlookupd
6    ports: 
7      - "4160:4160"
8      - "4161:4161"
9
10  nsqd: 
11    image: nsqio/nsq:latest
12    container_name: nsqd
13    command: /nsqd --lookupd-tcp-address=nsqlookupd:4160
14    ports: 
15      - "4150:4150"
16      - "4151:4151"
17    depends_on: 
18      - nsqlookupd
19
20  nsqadmin: 
21    image: nsqio/nsq:latest
22    container_name: nsqadmin
23    command: /nsqadmin --lookupd-http-address=nsqlookupd:4161
24    ports: 
25      - "4171:4171"
26    depends_on: 
27      - nsqlookupd

.env Template

.env
1# No additional config needed

Usage Notes

  1. 1Docs: https://nsq.io/overview/design.html
  2. 2Admin UI at http://localhost:4171, nsqd on ports 4150 (TCP) / 4151 (HTTP)
  3. 3nsqlookupd handles service discovery on ports 4160/4161
  4. 4Publish: curl -d 'message' http://localhost:4151/pub?topic=test
  5. 5No single point of failure - fully distributed
  6. 6Written in Go - simple, performant, fault-tolerant

Individual Services(3 services)

Copy individual services to mix and match with your existing compose files.

nsqlookupd
nsqlookupd:
  image: nsqio/nsq:latest
  container_name: nsqlookupd
  command: /nsqlookupd
  ports:
    - "4160:4160"
    - "4161:4161"
nsqd
nsqd:
  image: nsqio/nsq:latest
  container_name: nsqd
  command: /nsqd --lookupd-tcp-address=nsqlookupd:4160
  ports:
    - "4150:4150"
    - "4151:4151"
  depends_on:
    - nsqlookupd
nsqadmin
nsqadmin:
  image: nsqio/nsq:latest
  container_name: nsqadmin
  command: /nsqadmin --lookupd-http-address=nsqlookupd:4161
  ports:
    - "4171:4171"
  depends_on:
    - nsqlookupd

Quick Start

terminal
1# 1. Create the compose file
2cat > docker-compose.yml << 'EOF'
3services:
4  nsqlookupd:
5    image: nsqio/nsq:latest
6    container_name: nsqlookupd
7    command: /nsqlookupd
8    ports:
9      - "4160:4160"
10      - "4161:4161"
11
12  nsqd:
13    image: nsqio/nsq:latest
14    container_name: nsqd
15    command: /nsqd --lookupd-tcp-address=nsqlookupd:4160
16    ports:
17      - "4150:4150"
18      - "4151:4151"
19    depends_on:
20      - nsqlookupd
21
22  nsqadmin:
23    image: nsqio/nsq:latest
24    container_name: nsqadmin
25    command: /nsqadmin --lookupd-http-address=nsqlookupd:4161
26    ports:
27      - "4171:4171"
28    depends_on:
29      - nsqlookupd
30EOF
31
32# 2. Create the .env file
33cat > .env << 'EOF'
34# No additional config needed
35EOF
36
37# 3. Start the services
38docker compose up -d
39
40# 4. View logs
41docker compose logs -f

One-Liner

Run this command to download and set up the recipe in one step:

terminal
1curl -fsSL https://docker.recipes/api/recipes/nsq/run | bash

Troubleshooting

  • nsqd fails to connect to nsqlookupd: Verify nsqlookupd is running and accessible on port 4160, check Docker network connectivity between containers
  • Messages not being delivered to consumers: Confirm topic and channel names match exactly, check nsqadmin interface for queue depth and consumer connections
  • nsqadmin shows 'No nsqd nodes' error: Ensure nsqd has registered with nsqlookupd by checking logs, verify nsqlookupd HTTP address configuration
  • High memory usage in nsqd: Adjust message retention settings and sync intervals, consider scaling to multiple nsqd instances to distribute load
  • Connection refused on HTTP endpoints: Verify port mappings match container internal ports, check firewall rules and Docker network configuration
  • Topics not appearing in nsqadmin: Publish at least one message to create the topic, ensure nsqd is properly connected to nsqlookupd for discovery

Community Notes

Loading...
Loading notes...

Download Recipe Kit

Get all files in a ready-to-deploy package

Includes docker-compose.yml, .env template, README, and license

Components

nsq

Tags

#nsq#distributed#messaging#golang

Category

Message Queues & Brokers

Related Recipes

Ad Space