# Elasticsearch + Kibana

> Elasticsearch search engine with Kibana visualization dashboard.

- Category: Database Stacks
- Difficulty: intermediate
- Components: elasticsearch, kibana
- Tags: elasticsearch, search, analytics, kibana
- Source: https://docker.recipes/databases/elasticsearch-kibana

## Overview

Elasticsearch is a distributed, RESTful search and analytics engine built on Apache Lucene that has become the de facto standard for full-text search and log analytics. Originally developed by Shay Banon in 2010, Elasticsearch revolutionized how developers approach search functionality by providing near real-time indexing, powerful aggregations, and a simple REST API. As the core component of the Elastic Stack, it powers everything from application search features to enterprise-wide observability platforms.

This stack combines Elasticsearch's search capabilities with Kibana, the visualization and management layer that transforms raw data into interactive dashboards, charts, and reports. Kibana acts as the window into your Elasticsearch data, providing both end-users with powerful search interfaces and administrators with cluster management tools. Together, they solve the challenge of making large volumes of structured and unstructured data searchable, analyzable, and visually comprehensible.

This combination is ideal for development teams building search-driven applications, operations teams implementing centralized logging, and data analysts who need to explore and visualize large datasets. The stack excels in scenarios requiring real-time insights, complex aggregations, and intuitive data exploration interfaces, making it valuable for both technical and non-technical users who need to extract meaningful insights from their data.

## Features

- Full-text search with relevance scoring and advanced query DSL
- Real-time data visualization with Kibana dashboards and charts
- Distributed architecture with automatic sharding and replication
- RESTful JSON API for programmatic data access and management
- Powerful aggregations for analytics including histograms, terms, and metrics
- Index lifecycle management for automatic data retention policies
- Interactive data exploration with Kibana Discover and filtering
- Machine learning capabilities for anomaly detection and forecasting

## Use cases

- Application search functionality for e-commerce, documentation, or content sites
- Centralized logging and log analysis for microservices architectures
- Security information and event management (SIEM) for threat detection
- Business intelligence dashboards for sales, marketing, and operational metrics
- Application performance monitoring (APM) and observability platforms
- Time-series data analysis for IoT sensors and monitoring systems
- Content management systems requiring sophisticated search capabilities

## Prerequisites

- Minimum 4GB RAM available (2GB for Elasticsearch, additional for Kibana)
- Docker Engine 20.10+ and Docker Compose v2 for optimal compatibility
- Available ports 9200 (Elasticsearch API), 9300 (cluster communication), and 5601 (Kibana)
- Basic understanding of REST APIs and JSON for data indexing and querying
- Sufficient disk space for data storage and index growth over time

## docker-compose.yml

```yaml
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
    container_name: elasticsearch
    restart: unless-stopped
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    volumes:
      - es_data:/usr/share/elasticsearch/data
    ports:
      - "9200:9200"
      - "9300:9300"
    networks:
      - elastic-network

  kibana:
    image: docker.elastic.co/kibana/kibana:8.11.0
    container_name: kibana
    restart: unless-stopped
    environment:
      ELASTICSEARCH_HOSTS: http://elasticsearch:9200
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch
    networks:
      - elastic-network

volumes:
  es_data:

networks:
  elastic-network:
    driver: bridge
```

## .env

```bash
# Elasticsearch heap size (adjust based on available memory)
ES_JAVA_OPTS=-Xms512m -Xmx512m
```

## Usage notes

- Docs: https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html
- Access Kibana at http://localhost:5601 | ES API at http://localhost:9200
- Check cluster health: curl localhost:9200/_cluster/health?pretty
- Security disabled - enable xpack.security for production
- Adjust ES_JAVA_OPTS heap size (recommended: 50% of available RAM, max 32GB)
- Data stored in es_data volume - snapshot for backups

## Troubleshooting

- Elasticsearch exits with 'max virtual memory areas vm.max_map_count too low': Run 'sysctl -w vm.max_map_count=262144' on host system
- Kibana shows 'Kibana server is not ready yet': Wait 2-3 minutes for Elasticsearch to fully initialize before Kibana connects
- OutOfMemoryError in Elasticsearch logs: Increase ES_JAVA_OPTS heap size or reduce it if exceeding available system RAM
- Elasticsearch cluster status yellow or red: Check disk space availability and ensure proper index replica settings for single-node setup
- Connection refused on port 9200: Verify Elasticsearch container started successfully and check Docker container logs for JVM errors

