# Elastic Stack (ELK) Complete

> Full Elasticsearch, Logstash, Kibana stack with Beats agents and APM.

- Category: Monitoring & Observability
- Difficulty: advanced
- Components: elasticsearch, logstash, kibana, filebeat, metricbeat, apm-server
- Tags: elasticsearch, logstash, kibana, beats, apm
- Source: https://docker.recipes/monitoring/elastic-stack-complete

## Overview

Elasticsearch is a distributed, RESTful search and analytics engine built on Apache Lucene that serves as the backbone of the Elastic Stack. Originally developed by Shay Banon in 2010, Elasticsearch revolutionized log analytics and full-text search by providing near real-time indexing, distributed architecture with automatic sharding, and powerful aggregation capabilities. Its RESTful API and JSON-based queries made it accessible to developers while its horizontal scaling capabilities attracted enterprises dealing with massive data volumes. This complete Elastic Stack deployment combines Elasticsearch with Logstash for data processing, Kibana for visualization, Filebeat and Metricbeat for data collection, and APM Server for application performance monitoring. Together, these components create a comprehensive observability platform that ingests, processes, stores, and visualizes machine data from any source. The stack transforms raw logs, metrics, and traces into actionable insights through Kibana's rich dashboards and Elasticsearch's powerful search capabilities. Organizations seeking unified observability, security teams building SIEM solutions, and DevOps engineers monitoring distributed systems will find this stack invaluable. The combination provides end-to-end visibility from application traces captured by APM Server to system metrics collected by Metricbeat, all processed through Logstash pipelines and made searchable in Elasticsearch. This deployment particularly benefits teams managing microservices architectures, containerized applications, or complex distributed systems where correlating data across multiple sources is crucial for troubleshooting and performance optimization.

## Features

- Full-text search with relevance scoring and advanced query DSL across all ingested data
- Near real-time indexing and search capabilities with sub-second query response times
- Logstash pipeline processing with 200+ input, filter, and output plugins for data transformation
- Kibana Canvas for pixel-perfect data presentation and custom visualization creation
- Machine learning anomaly detection for automatic identification of unusual patterns in logs and metrics
- APM distributed tracing for end-to-end transaction visibility across microservices
- Beat agents for lightweight data shipping from containers, systems, and applications
- Index lifecycle management with automatic hot-warm-cold data tiering based on age and usage patterns

## Use cases

- Centralized logging for microservices architectures with distributed tracing correlation
- Security information and event management (SIEM) for threat detection and compliance reporting
- Application performance monitoring with code-level visibility and error tracking
- Infrastructure monitoring with real-time dashboards for system metrics and container health
- E-commerce search functionality with faceted navigation and personalized recommendations
- Business intelligence analytics with custom dashboards and automated alerting
- Compliance logging for financial services and healthcare organizations requiring audit trails

## Prerequisites

- Minimum 8GB RAM recommended (Elasticsearch requires 2GB minimum, additional memory for other components)
- Docker Engine 20.10+ with Docker Compose V2 for multi-container orchestration
- Available ports 5601, 9200, 9300, 8200, 5044, 5000, and 9600 for service communication
- Understanding of Elasticsearch mapping concepts and Logstash pipeline configuration syntax
- Basic knowledge of Kibana query languages (KQL/Lucene) for effective data exploration
- Familiarity with YAML configuration for Beat agents and APM instrumentation setup

## docker-compose.yml

```yaml
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=true
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
    volumes:
      - es_data:/usr/share/elasticsearch/data
    ports:
      - "9200:9200"
      - "9300:9300"
    networks:
      - elastic_net

  logstash:
    image: docker.elastic.co/logstash/logstash:8.11.0
    volumes:
      - ./logstash/pipeline:/usr/share/logstash/pipeline:ro
      - ./logstash/config:/usr/share/logstash/config:ro
    ports:
      - "5044:5044"
      - "5000:5000/tcp"
      - "5000:5000/udp"
      - "9600:9600"
    environment:
      - xpack.monitoring.elasticsearch.hosts=http://elasticsearch:9200
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
    depends_on:
      - elasticsearch
    networks:
      - elastic_net

  kibana:
    image: docker.elastic.co/kibana/kibana:8.11.0
    ports:
      - "5601:5601"
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
    depends_on:
      - elasticsearch
    networks:
      - elastic_net

  filebeat:
    image: docker.elastic.co/beats/filebeat:8.11.0
    user: root
    volumes:
      - ./filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
    depends_on:
      - elasticsearch
      - logstash
    networks:
      - elastic_net

  metricbeat:
    image: docker.elastic.co/beats/metricbeat:8.11.0
    user: root
    volumes:
      - ./metricbeat/metricbeat.yml:/usr/share/metricbeat/metricbeat.yml:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
      - /proc:/hostfs/proc:ro
      - /:/hostfs:ro
    environment:
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
    depends_on:
      - elasticsearch
    networks:
      - elastic_net

  apm-server:
    image: docker.elastic.co/apm/apm-server:8.11.0
    ports:
      - "8200:8200"
    environment:
      - output.elasticsearch.hosts=["elasticsearch:9200"]
      - output.elasticsearch.username=elastic
      - output.elasticsearch.password=${ELASTIC_PASSWORD}
    depends_on:
      - elasticsearch
    networks:
      - elastic_net

volumes:
  es_data:

networks:
  elastic_net:
```

## .env

```bash
# Elastic Stack
ELASTIC_PASSWORD=secure_elastic_password
KIBANA_PASSWORD=secure_kibana_password

# Kibana at http://localhost:5601
# Elasticsearch at http://localhost:9200
# APM at http://localhost:8200
```

## Usage notes

- Kibana at http://localhost:5601
- Elasticsearch at http://localhost:9200
- APM Server at http://localhost:8200
- Logstash at port 5044 (Beats), 5000 (TCP/UDP)
- Default user: elastic/your_password

## Troubleshooting

- Elasticsearch cluster health red: Check disk space and memory allocation, ensure ES_JAVA_OPTS heap size doesn't exceed 50% of available RAM
- Kibana 'Unable to connect to Elasticsearch' error: Verify ELASTICSEARCH_HOSTS environment variable and ensure Elasticsearch is fully started before Kibana
- Logstash pipeline not processing data: Check pipeline configuration syntax in ./logstash/pipeline directory and verify input/output plugin connectivity
- Filebeat not shipping container logs: Ensure Docker socket mount permissions and verify filebeat.yml autodiscover configuration for Docker containers
- APM Server rejecting agent data: Confirm APM agents are configured with correct server URL (localhost:8200) and authentication credentials match ELASTIC_PASSWORD
- High memory usage on Elasticsearch: Adjust ES_JAVA_OPTS heap size, implement index lifecycle policies, and consider fielddata circuit breaker settings

