# OpenSearch Cluster

> Open source search and analytics suite with OpenSearch Dashboards and multi-node cluster.

- Category: Database Stacks
- Difficulty: advanced
- Components: opensearch, opensearch-dashboards, logstash, nginx
- Tags: opensearch, search, analytics, elasticsearch-alternative
- Source: https://docker.recipes/databases/opensearch-stack

## Overview

OpenSearch is a community-driven, open-source search and analytics engine forked from Elasticsearch 7.10.2, developed by Amazon Web Services after Elastic changed Elasticsearch's licensing model. It provides full-text search capabilities, real-time analytics, and log aggregation features while maintaining compatibility with existing Elasticsearch APIs and tools. The platform includes built-in security features, machine learning capabilities, and distributed search functionality designed to handle petabyte-scale data workloads across multiple nodes.

This multi-node OpenSearch cluster combines OpenSearch nodes for distributed search and storage, OpenSearch Dashboards for data visualization and cluster management, Logstash for data ingestion and transformation, and NGINX as a load balancer to distribute queries across cluster nodes. The configuration establishes a resilient two-node cluster with automatic failover, shared data replication, and centralized log processing pipeline that can ingest data from multiple sources while providing unified search capabilities across structured and unstructured data.

This stack serves organizations migrating from Elasticsearch seeking an open-source alternative, DevOps teams building centralized logging infrastructure, and data analysts requiring scalable search capabilities with rich visualization tools. The combination delivers enterprise-grade search functionality without licensing restrictions, making it particularly valuable for companies processing large volumes of log data, implementing site search functionality, or building analytics platforms that require real-time data processing and complex query capabilities.

## Features

- Multi-node OpenSearch cluster with automatic leader election and data replication across opensearch-node1 and opensearch-node2
- OpenSearch Dashboards web interface providing Kibana-compatible visualizations, index management, and cluster health monitoring
- Logstash data processing pipeline with OpenSearch output plugin for ETL operations and log parsing from multiple sources
- NGINX load balancer distributing OpenSearch API requests across cluster nodes for improved performance and availability
- Built-in OpenSearch Security plugin with role-based access control, field-level security, and audit logging capabilities
- Memory-optimized JVM configuration with 512MB heap allocation per node and memory lock prevention of swapping
- Elasticsearch API compatibility allowing existing Elasticsearch clients and tools to work without modification
- Cluster-wide search federation enabling queries across all nodes with automatic result aggregation and scoring

## Use cases

- Centralized log aggregation for microservices architectures collecting application logs, system metrics, and security events
- E-commerce product search implementations requiring faceted search, auto-completion, and relevance scoring capabilities
- Security information and event management (SIEM) systems analyzing network traffic, threat detection, and compliance reporting
- Business intelligence dashboards aggregating data from multiple databases for real-time analytics and reporting
- Content management systems implementing full-text search across documents, media files, and user-generated content
- DevOps monitoring platforms correlating infrastructure metrics, application performance data, and deployment tracking
- Data lake search layer providing query capabilities over structured and unstructured data stored in various formats

## Prerequisites

- Minimum 4GB RAM available (2GB for OpenSearch cluster, 1GB for Dashboards, 1GB for Logstash and system overhead)
- Docker host with vm.max_map_count set to at least 262144 for OpenSearch memory mapping requirements
- Available ports 5601 (Dashboards), 9200 (OpenSearch API), 5044 (Logstash beats input), and 9600 (Logstash monitoring)
- Basic understanding of Elasticsearch query DSL and JSON document structure for effective data indexing and searching
- Logstash pipeline configuration files in ./logstash/pipeline directory defining input sources and data transformation rules
- NGINX configuration file at ./nginx.conf specifying upstream OpenSearch nodes and load balancing strategy

## docker-compose.yml

```yaml
services:
  opensearch-node1:
    image: opensearchproject/opensearch:latest
    environment:
      cluster.name: opensearch-cluster
      node.name: opensearch-node1
      discovery.seed_hosts: opensearch-node1,opensearch-node2
      cluster.initial_cluster_manager_nodes: opensearch-node1,opensearch-node2
      bootstrap.memory_lock: "true"
      OPENSEARCH_JAVA_OPTS: "-Xms512m -Xmx512m"
      OPENSEARCH_INITIAL_ADMIN_PASSWORD: ${OPENSEARCH_PASSWORD}
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data
    networks:
      - opensearch-net
    restart: unless-stopped

  opensearch-node2:
    image: opensearchproject/opensearch:latest
    environment:
      cluster.name: opensearch-cluster
      node.name: opensearch-node2
      discovery.seed_hosts: opensearch-node1,opensearch-node2
      cluster.initial_cluster_manager_nodes: opensearch-node1,opensearch-node2
      bootstrap.memory_lock: "true"
      OPENSEARCH_JAVA_OPTS: "-Xms512m -Xmx512m"
      OPENSEARCH_INITIAL_ADMIN_PASSWORD: ${OPENSEARCH_PASSWORD}
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch-data2:/usr/share/opensearch/data
    networks:
      - opensearch-net
    restart: unless-stopped

  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:latest
    ports:
      - "5601:5601"
    environment:
      OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]'
    depends_on:
      - opensearch-node1
      - opensearch-node2
    networks:
      - opensearch-net
    restart: unless-stopped

  logstash:
    image: opensearchproject/logstash-oss-with-opensearch-output-plugin:latest
    volumes:
      - ./logstash/pipeline:/usr/share/logstash/pipeline:ro
    ports:
      - "5044:5044"
      - "9600:9600"
    depends_on:
      - opensearch-node1
    networks:
      - opensearch-net
    restart: unless-stopped

  nginx:
    image: nginx:alpine
    ports:
      - "9200:9200"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - opensearch-node1
      - opensearch-node2
    networks:
      - opensearch-net
    restart: unless-stopped

volumes:
  opensearch-data1:
  opensearch-data2:

networks:
  opensearch-net:
    driver: bridge
```

## .env

```bash
# OpenSearch Admin Password
OPENSEARCH_PASSWORD=SecurePassword123!

# Must meet complexity requirements:
# - At least 8 characters
# - Uppercase letter
# - Lowercase letter
# - Number
# - Special character
```

## Usage notes

- Dashboards at http://localhost:5601
- Default user: admin
- Elasticsearch-compatible API
- Includes security plugin by default

## Troubleshooting

- OpenSearch bootstrap check failures with memory lock errors: Increase Docker container memory limits and set vm.max_map_count=262144 on host system
- Cluster formation timeout with nodes unable to discover each other: Verify opensearch-net network connectivity and ensure discovery.seed_hosts includes all node names
- OpenSearch Dashboards connection refused to cluster: Check OPENSEARCH_HOSTS environment variable URLs match actual node addresses and verify SSL certificate trust
- Logstash pipeline startup failures: Validate pipeline configuration syntax in mounted ./logstash/pipeline directory and ensure OpenSearch cluster is accessible
- NGINX 502 Bad Gateway errors: Confirm opensearch-node1 and opensearch-node2 services are running and nginx.conf upstream configuration matches service names
- High memory usage causing container restarts: Adjust OPENSEARCH_JAVA_OPTS heap size based on available system memory and enable swap accounting if needed

