OpenSearch Cluster

advanced

Open source search and analytics suite with OpenSearch Dashboards and multi-node cluster.

Overview

OpenSearch is a community-driven, open-source search and analytics engine forked from Elasticsearch 7.10.2, developed by Amazon Web Services after Elastic changed Elasticsearch's licensing model. It provides full-text search capabilities, real-time analytics, and log aggregation features while maintaining compatibility with existing Elasticsearch APIs and tools. The platform includes built-in security features, machine learning capabilities, and distributed search functionality designed to handle petabyte-scale data workloads across multiple nodes. This multi-node OpenSearch cluster combines OpenSearch nodes for distributed search and storage, OpenSearch Dashboards for data visualization and cluster management, Logstash for data ingestion and transformation, and NGINX as a load balancer to distribute queries across cluster nodes. The configuration establishes a resilient two-node cluster with automatic failover, shared data replication, and centralized log processing pipeline that can ingest data from multiple sources while providing unified search capabilities across structured and unstructured data. This stack serves organizations migrating from Elasticsearch seeking an open-source alternative, DevOps teams building centralized logging infrastructure, and data analysts requiring scalable search capabilities with rich visualization tools. The combination delivers enterprise-grade search functionality without licensing restrictions, making it particularly valuable for companies processing large volumes of log data, implementing site search functionality, or building analytics platforms that require real-time data processing and complex query capabilities.

Key Features

Multi-node OpenSearch cluster with automatic leader election and data replication across opensearch-node1 and opensearch-node2
OpenSearch Dashboards web interface providing Kibana-compatible visualizations, index management, and cluster health monitoring
Logstash data processing pipeline with OpenSearch output plugin for ETL operations and log parsing from multiple sources
NGINX load balancer distributing OpenSearch API requests across cluster nodes for improved performance and availability
Built-in OpenSearch Security plugin with role-based access control, field-level security, and audit logging capabilities
Memory-optimized JVM configuration with 512MB heap allocation per node and memory lock prevention of swapping
Elasticsearch API compatibility allowing existing Elasticsearch clients and tools to work without modification
Cluster-wide search federation enabling queries across all nodes with automatic result aggregation and scoring

Common Use Cases

1Centralized log aggregation for microservices architectures collecting application logs, system metrics, and security events
2E-commerce product search implementations requiring faceted search, auto-completion, and relevance scoring capabilities
3Security information and event management (SIEM) systems analyzing network traffic, threat detection, and compliance reporting
4Business intelligence dashboards aggregating data from multiple databases for real-time analytics and reporting
5Content management systems implementing full-text search across documents, media files, and user-generated content
6DevOps monitoring platforms correlating infrastructure metrics, application performance data, and deployment tracking
7Data lake search layer providing query capabilities over structured and unstructured data stored in various formats

Prerequisites

Minimum 4GB RAM available (2GB for OpenSearch cluster, 1GB for Dashboards, 1GB for Logstash and system overhead)
Docker host with vm.max_map_count set to at least 262144 for OpenSearch memory mapping requirements
Available ports 5601 (Dashboards), 9200 (OpenSearch API), 5044 (Logstash beats input), and 9600 (Logstash monitoring)
Basic understanding of Elasticsearch query DSL and JSON document structure for effective data indexing and searching
Logstash pipeline configuration files in ./logstash/pipeline directory defining input sources and data transformation rules
NGINX configuration file at ./nginx.conf specifying upstream OpenSearch nodes and load balancing strategy

For development & testing. Review security settings, change default credentials, and test thoroughly before production use. See Terms

docker-compose.yml

1services: 
2  opensearch-node1: 
3    image: opensearchproject/opensearch:latest
4    environment: 
5      cluster.name: opensearch-cluster
6      node.name: opensearch-node1
7      discovery.seed_hosts: opensearch-node1,opensearch-node2
8      cluster.initial_cluster_manager_nodes: opensearch-node1,opensearch-node2
9      bootstrap.memory_lock: "true"
10      OPENSEARCH_JAVA_OPTS: "-Xms512m -Xmx512m"
11      OPENSEARCH_INITIAL_ADMIN_PASSWORD: ${OPENSEARCH_PASSWORD}
12    ulimits: 
13      memlock: 
14        soft: -1
15        hard: -1
16      nofile: 
17        soft: 65536
18        hard: 65536
19    volumes: 
20      - opensearch-data1:/usr/share/opensearch/data
21    networks: 
22      - opensearch-net
23    restart: unless-stopped
24
25  opensearch-node2: 
26    image: opensearchproject/opensearch:latest
27    environment: 
28      cluster.name: opensearch-cluster
29      node.name: opensearch-node2
30      discovery.seed_hosts: opensearch-node1,opensearch-node2
31      cluster.initial_cluster_manager_nodes: opensearch-node1,opensearch-node2
32      bootstrap.memory_lock: "true"
33      OPENSEARCH_JAVA_OPTS: "-Xms512m -Xmx512m"
34      OPENSEARCH_INITIAL_ADMIN_PASSWORD: ${OPENSEARCH_PASSWORD}
35    ulimits: 
36      memlock: 
37        soft: -1
38        hard: -1
39      nofile: 
40        soft: 65536
41        hard: 65536
42    volumes: 
43      - opensearch-data2:/usr/share/opensearch/data
44    networks: 
45      - opensearch-net
46    restart: unless-stopped
47
48  opensearch-dashboards: 
49    image: opensearchproject/opensearch-dashboards:latest
50    ports: 
51      - "5601:5601"
52    environment: 
53      OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]'
54    depends_on: 
55      - opensearch-node1
56      - opensearch-node2
57    networks: 
58      - opensearch-net
59    restart: unless-stopped
60
61  logstash: 
62    image: opensearchproject/logstash-oss-with-opensearch-output-plugin:latest
63    volumes: 
64      - ./logstash/pipeline:/usr/share/logstash/pipeline:ro
65    ports: 
66      - "5044:5044"
67      - "9600:9600"
68    depends_on: 
69      - opensearch-node1
70    networks: 
71      - opensearch-net
72    restart: unless-stopped
73
74  nginx: 
75    image: nginx:alpine
76    ports: 
77      - "9200:9200"
78    volumes: 
79      - ./nginx.conf:/etc/nginx/nginx.conf:ro
80    depends_on: 
81      - opensearch-node1
82      - opensearch-node2
83    networks: 
84      - opensearch-net
85    restart: unless-stopped
86
87volumes: 
88  opensearch-data1: 
89  opensearch-data2: 
90
91networks: 
92  opensearch-net: 
93    driver: bridge

.env Template

.env

1# OpenSearch Admin Password
2OPENSEARCH_PASSWORD=SecurePassword123!
3
4# Must meet complexity requirements:
5# - At least 8 characters
6# - Uppercase letter
7# - Lowercase letter
8# - Number
9# - Special character

Usage Notes

1Dashboards at http://localhost:5601
2Default user: admin
3Elasticsearch-compatible API
4Includes security plugin by default

Individual Services(5 services)

Copy individual services to mix and match with your existing compose files.

opensearch-node1

opensearch-node1:
  image: opensearchproject/opensearch:latest
  environment:
    cluster.name: opensearch-cluster
    node.name: opensearch-node1
    discovery.seed_hosts: opensearch-node1,opensearch-node2
    cluster.initial_cluster_manager_nodes: opensearch-node1,opensearch-node2
    bootstrap.memory_lock: "true"
    OPENSEARCH_JAVA_OPTS: "-Xms512m -Xmx512m"
    OPENSEARCH_INITIAL_ADMIN_PASSWORD: ${OPENSEARCH_PASSWORD}
  ulimits:
    memlock:
      soft: -1
      hard: -1
    nofile:
      soft: 65536
      hard: 65536
  volumes:
    - opensearch-data1:/usr/share/opensearch/data
  networks:
    - opensearch-net
  restart: unless-stopped

opensearch-node2

opensearch-node2:
  image: opensearchproject/opensearch:latest
  environment:
    cluster.name: opensearch-cluster
    node.name: opensearch-node2
    discovery.seed_hosts: opensearch-node1,opensearch-node2
    cluster.initial_cluster_manager_nodes: opensearch-node1,opensearch-node2
    bootstrap.memory_lock: "true"
    OPENSEARCH_JAVA_OPTS: "-Xms512m -Xmx512m"
    OPENSEARCH_INITIAL_ADMIN_PASSWORD: ${OPENSEARCH_PASSWORD}
  ulimits:
    memlock:
      soft: -1
      hard: -1
    nofile:
      soft: 65536
      hard: 65536
  volumes:
    - opensearch-data2:/usr/share/opensearch/data
  networks:
    - opensearch-net
  restart: unless-stopped

opensearch-dashboards

opensearch-dashboards:
  image: opensearchproject/opensearch-dashboards:latest
  ports:
    - "5601:5601"
  environment:
    OPENSEARCH_HOSTS: "[\"https://opensearch-node1:9200\",\"https://opensearch-node2:9200\"]"
  depends_on:
    - opensearch-node1
    - opensearch-node2
  networks:
    - opensearch-net
  restart: unless-stopped

logstash

logstash:
  image: opensearchproject/logstash-oss-with-opensearch-output-plugin:latest
  volumes:
    - ./logstash/pipeline:/usr/share/logstash/pipeline:ro
  ports:
    - "5044:5044"
    - "9600:9600"
  depends_on:
    - opensearch-node1
  networks:
    - opensearch-net
  restart: unless-stopped

nginx

nginx:
  image: nginx:alpine
  ports:
    - "9200:9200"
  volumes:
    - ./nginx.conf:/etc/nginx/nginx.conf:ro
  depends_on:
    - opensearch-node1
    - opensearch-node2
  networks:
    - opensearch-net
  restart: unless-stopped

Quick Start

terminal

1# 1. Create the compose file
2cat > docker-compose.yml << 'EOF'
3services:
4  opensearch-node1:
5    image: opensearchproject/opensearch:latest
6    environment:
7      cluster.name: opensearch-cluster
8      node.name: opensearch-node1
9      discovery.seed_hosts: opensearch-node1,opensearch-node2
10      cluster.initial_cluster_manager_nodes: opensearch-node1,opensearch-node2
11      bootstrap.memory_lock: "true"
12      OPENSEARCH_JAVA_OPTS: "-Xms512m -Xmx512m"
13      OPENSEARCH_INITIAL_ADMIN_PASSWORD: ${OPENSEARCH_PASSWORD}
14    ulimits:
15      memlock:
16        soft: -1
17        hard: -1
18      nofile:
19        soft: 65536
20        hard: 65536
21    volumes:
22      - opensearch-data1:/usr/share/opensearch/data
23    networks:
24      - opensearch-net
25    restart: unless-stopped
26
27  opensearch-node2:
28    image: opensearchproject/opensearch:latest
29    environment:
30      cluster.name: opensearch-cluster
31      node.name: opensearch-node2
32      discovery.seed_hosts: opensearch-node1,opensearch-node2
33      cluster.initial_cluster_manager_nodes: opensearch-node1,opensearch-node2
34      bootstrap.memory_lock: "true"
35      OPENSEARCH_JAVA_OPTS: "-Xms512m -Xmx512m"
36      OPENSEARCH_INITIAL_ADMIN_PASSWORD: ${OPENSEARCH_PASSWORD}
37    ulimits:
38      memlock:
39        soft: -1
40        hard: -1
41      nofile:
42        soft: 65536
43        hard: 65536
44    volumes:
45      - opensearch-data2:/usr/share/opensearch/data
46    networks:
47      - opensearch-net
48    restart: unless-stopped
49
50  opensearch-dashboards:
51    image: opensearchproject/opensearch-dashboards:latest
52    ports:
53      - "5601:5601"
54    environment:
55      OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]'
56    depends_on:
57      - opensearch-node1
58      - opensearch-node2
59    networks:
60      - opensearch-net
61    restart: unless-stopped
62
63  logstash:
64    image: opensearchproject/logstash-oss-with-opensearch-output-plugin:latest
65    volumes:
66      - ./logstash/pipeline:/usr/share/logstash/pipeline:ro
67    ports:
68      - "5044:5044"
69      - "9600:9600"
70    depends_on:
71      - opensearch-node1
72    networks:
73      - opensearch-net
74    restart: unless-stopped
75
76  nginx:
77    image: nginx:alpine
78    ports:
79      - "9200:9200"
80    volumes:
81      - ./nginx.conf:/etc/nginx/nginx.conf:ro
82    depends_on:
83      - opensearch-node1
84      - opensearch-node2
85    networks:
86      - opensearch-net
87    restart: unless-stopped
88
89volumes:
90  opensearch-data1:
91  opensearch-data2:
92
93networks:
94  opensearch-net:
95    driver: bridge
96EOF
97
98# 2. Create the .env file
99cat > .env << 'EOF'
100# OpenSearch Admin Password
101OPENSEARCH_PASSWORD=SecurePassword123!
102
103# Must meet complexity requirements:
104# - At least 8 characters
105# - Uppercase letter
106# - Lowercase letter
107# - Number
108# - Special character
109EOF
110
111# 3. Start the services
112docker compose up -d
113
114# 4. View logs
115docker compose logs -f

One-Liner

Run this command to download and set up the recipe in one step:

terminal

1curl -fsSL https://docker.recipes/api/recipes/opensearch-stack/run | bash

Troubleshooting

OpenSearch bootstrap check failures with memory lock errors: Increase Docker container memory limits and set vm.max_map_count=262144 on host system
Cluster formation timeout with nodes unable to discover each other: Verify opensearch-net network connectivity and ensure discovery.seed_hosts includes all node names
OpenSearch Dashboards connection refused to cluster: Check OPENSEARCH_HOSTS environment variable URLs match actual node addresses and verify SSL certificate trust
Logstash pipeline startup failures: Validate pipeline configuration syntax in mounted ./logstash/pipeline directory and ensure OpenSearch cluster is accessible
NGINX 502 Bad Gateway errors: Confirm opensearch-node1 and opensearch-node2 services are running and nginx.conf upstream configuration matches service names
High memory usage causing container restarts: Adjust OPENSEARCH_JAVA_OPTS heap size based on available system memory and enable swap accounting if needed

Community Notes

Loading notes...

Download Recipe Kit

Get all files in a ready-to-deploy package

Includes docker-compose.yml, .env template, README, and license

Components

opensearchopensearch-dashboardslogstashnginx

Related Recipes

Ad Space

Shortcuts: C CopyF FavoriteD Download