# Elastic APM Stack

> Elasticsearch APM with Kibana for application monitoring.

- Category: Monitoring & Observability
- Difficulty: intermediate
- Components: elasticsearch, kibana, apm-server
- Tags: elastic, apm, observability, kibana
- Source: https://docker.recipes/monitoring/elastic-apm-stack

## Overview

Elasticsearch is a distributed search and analytics engine built on Apache Lucene that has become the cornerstone of modern observability platforms. Originally developed by Elastic N.V., it provides real-time indexing and search capabilities across massive datasets, making it ideal for log analytics, application performance monitoring, and business intelligence. Its distributed architecture with automatic sharding and replication ensures high availability and horizontal scalability. This Elastic APM Stack combines Elasticsearch with Kibana (the visualization and management interface) and APM Server (the data collection component) to create a comprehensive application performance monitoring solution. APM Server receives performance data from instrumented applications via lightweight agents, processes and enriches this data, then stores it in Elasticsearch indices optimized for time-series analysis. Kibana provides pre-built dashboards and visualizations specifically designed for APM data, including service maps, transaction traces, and error analytics. Development teams and DevOps engineers seeking deep visibility into application performance, error tracking, and dependency mapping will find this stack invaluable. Unlike basic logging solutions, this combination provides distributed tracing capabilities that track requests across microservices, automatic service discovery and mapping, and machine learning-powered anomaly detection for proactive issue identification.

## Features

- Distributed tracing across microservices with automatic span correlation and timing analysis
- Real-time application metrics collection including response times, throughput, and error rates
- Service topology mapping that automatically discovers and visualizes application dependencies
- Machine learning anomaly detection for proactive identification of performance degradations
- Custom dashboards and alerting through Kibana's visualization engine
- Support for multiple programming languages through official APM agents
- Index lifecycle management for automatic data retention and cost optimization
- Cross-cluster search capabilities for monitoring distributed application environments

## Use cases

- Microservices performance monitoring in production environments with distributed request tracing
- E-commerce platform optimization by identifying slow database queries and API bottlenecks
- SaaS application error tracking and root cause analysis across multiple service tiers
- DevOps teams implementing observability-driven development practices with CI/CD integration
- Enterprise applications requiring compliance auditing and performance SLA monitoring
- Startup development teams establishing performance baselines during rapid scaling phases
- Legacy application modernization projects needing visibility during cloud migration

## Prerequisites

- Minimum 4GB RAM available for Elasticsearch heap allocation and optimal query performance
- Docker and Docker Compose installed with at least 20GB available disk space for indices
- Network ports 9200, 5601, and 8200 available and not conflicting with existing services
- Basic understanding of APM concepts including transactions, spans, and distributed tracing
- Application instrumentation knowledge for integrating APM agents into your codebase
- JVM tuning experience recommended for production Elasticsearch deployments

## docker-compose.yml

```yaml
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
    container_name: elastic-apm-es
    restart: unless-stopped
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    volumes:
      - es_data:/usr/share/elasticsearch/data
    ports:
      - "${ES_PORT:-9200}:9200"

  kibana:
    image: docker.elastic.co/kibana/kibana:8.11.0
    container_name: elastic-apm-kibana
    restart: unless-stopped
    ports:
      - "${KIBANA_PORT:-5601}:5601"
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    depends_on:
      - elasticsearch

  apm-server:
    image: docker.elastic.co/apm/apm-server:8.11.0
    container_name: apm-server
    restart: unless-stopped
    ports:
      - "${APM_PORT:-8200}:8200"
    environment:
      - output.elasticsearch.hosts=["elasticsearch:9200"]
    depends_on:
      - elasticsearch

volumes:
  es_data:
```

## .env

```bash
# Elastic APM
ES_PORT=9200
KIBANA_PORT=5601
APM_PORT=8200
```

## Usage notes

- Kibana at http://localhost:5601
- APM Server at http://localhost:8200
- Configure APM agents with server URL
- View APM in Kibana Observability

## Troubleshooting

- Elasticsearch container fails to start with 'max virtual memory areas vm.max_map_count too low': Increase vm.max_map_count to 262144 using sysctl -w vm.max_map_count=262144
- APM agents cannot connect showing 'connection refused' errors: Verify APM server is accessible at port 8200 and check firewall rules between application and container network
- Kibana displays 'Elasticsearch cluster did not respond' warnings: Wait 30-60 seconds for Elasticsearch to complete startup and index creation before accessing Kibana
- High memory usage causing container restarts: Adjust ES_JAVA_OPTS heap settings based on available system memory, recommended minimum -Xms1g -Xmx1g for production
- APM data not appearing in Kibana despite successful agent connections: Check APM server logs for parsing errors and verify agent compatibility with server version
- Disk space filling rapidly with APM indices: Configure index lifecycle policies in Kibana to automatically delete old data after specified retention periods

