Elastic APM Stack

intermediate

Elasticsearch APM with Kibana for application monitoring.

Overview

Elasticsearch is a distributed search and analytics engine built on Apache Lucene that has become the cornerstone of modern observability platforms. Originally developed by Elastic N.V., it provides real-time indexing and search capabilities across massive datasets, making it ideal for log analytics, application performance monitoring, and business intelligence. Its distributed architecture with automatic sharding and replication ensures high availability and horizontal scalability. This Elastic APM Stack combines Elasticsearch with Kibana (the visualization and management interface) and APM Server (the data collection component) to create a comprehensive application performance monitoring solution. APM Server receives performance data from instrumented applications via lightweight agents, processes and enriches this data, then stores it in Elasticsearch indices optimized for time-series analysis. Kibana provides pre-built dashboards and visualizations specifically designed for APM data, including service maps, transaction traces, and error analytics. Development teams and DevOps engineers seeking deep visibility into application performance, error tracking, and dependency mapping will find this stack invaluable. Unlike basic logging solutions, this combination provides distributed tracing capabilities that track requests across microservices, automatic service discovery and mapping, and machine learning-powered anomaly detection for proactive issue identification.

Key Features

Distributed tracing across microservices with automatic span correlation and timing analysis
Real-time application metrics collection including response times, throughput, and error rates
Service topology mapping that automatically discovers and visualizes application dependencies
Machine learning anomaly detection for proactive identification of performance degradations
Custom dashboards and alerting through Kibana's visualization engine
Support for multiple programming languages through official APM agents
Index lifecycle management for automatic data retention and cost optimization
Cross-cluster search capabilities for monitoring distributed application environments

Common Use Cases

1Microservices performance monitoring in production environments with distributed request tracing
2E-commerce platform optimization by identifying slow database queries and API bottlenecks
3SaaS application error tracking and root cause analysis across multiple service tiers
4DevOps teams implementing observability-driven development practices with CI/CD integration
5Enterprise applications requiring compliance auditing and performance SLA monitoring
6Startup development teams establishing performance baselines during rapid scaling phases
7Legacy application modernization projects needing visibility during cloud migration

Prerequisites

Minimum 4GB RAM available for Elasticsearch heap allocation and optimal query performance
Docker and Docker Compose installed with at least 20GB available disk space for indices
Network ports 9200, 5601, and 8200 available and not conflicting with existing services
Basic understanding of APM concepts including transactions, spans, and distributed tracing
Application instrumentation knowledge for integrating APM agents into your codebase
JVM tuning experience recommended for production Elasticsearch deployments

For development & testing. Review security settings, change default credentials, and test thoroughly before production use. See Terms

docker-compose.yml

1services: 
2  elasticsearch: 
3    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
4    container_name: elastic-apm-es
5    restart: unless-stopped
6    environment: 
7      - discovery.type=single-node
8      - xpack.security.enabled=false
9      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
10    volumes: 
11      - es_data:/usr/share/elasticsearch/data
12    ports: 
13      - "${ES_PORT:-9200}:9200"
14
15  kibana: 
16    image: docker.elastic.co/kibana/kibana:8.11.0
17    container_name: elastic-apm-kibana
18    restart: unless-stopped
19    ports: 
20      - "${KIBANA_PORT:-5601}:5601"
21    environment: 
22      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
23    depends_on: 
24      - elasticsearch
25
26  apm-server: 
27    image: docker.elastic.co/apm/apm-server:8.11.0
28    container_name: apm-server
29    restart: unless-stopped
30    ports: 
31      - "${APM_PORT:-8200}:8200"
32    environment: 
33      - output.elasticsearch.hosts=["elasticsearch:9200"]
34    depends_on: 
35      - elasticsearch
36
37volumes: 
38  es_data:

.env Template

.env

1# Elastic APM
2ES_PORT=9200
3KIBANA_PORT=5601
4APM_PORT=8200

Usage Notes

1Kibana at http://localhost:5601
2APM Server at http://localhost:8200
3Configure APM agents with server URL
4View APM in Kibana Observability

Individual Services(3 services)

Copy individual services to mix and match with your existing compose files.

elasticsearch

elasticsearch:
  image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
  container_name: elastic-apm-es
  restart: unless-stopped
  environment:
    - discovery.type=single-node
    - xpack.security.enabled=false
    - ES_JAVA_OPTS=-Xms512m -Xmx512m
  volumes:
    - es_data:/usr/share/elasticsearch/data
  ports:
    - ${ES_PORT:-9200}:9200

kibana

kibana:
  image: docker.elastic.co/kibana/kibana:8.11.0
  container_name: elastic-apm-kibana
  restart: unless-stopped
  ports:
    - ${KIBANA_PORT:-5601}:5601
  environment:
    - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
  depends_on:
    - elasticsearch

apm-server

apm-server:
  image: docker.elastic.co/apm/apm-server:8.11.0
  container_name: apm-server
  restart: unless-stopped
  ports:
    - ${APM_PORT:-8200}:8200
  environment:
    - output.elasticsearch.hosts=["elasticsearch:9200"]
  depends_on:
    - elasticsearch

Quick Start

terminal

1# 1. Create the compose file
2cat > docker-compose.yml << 'EOF'
3services:
4  elasticsearch:
5    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
6    container_name: elastic-apm-es
7    restart: unless-stopped
8    environment:
9      - discovery.type=single-node
10      - xpack.security.enabled=false
11      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
12    volumes:
13      - es_data:/usr/share/elasticsearch/data
14    ports:
15      - "${ES_PORT:-9200}:9200"
16
17  kibana:
18    image: docker.elastic.co/kibana/kibana:8.11.0
19    container_name: elastic-apm-kibana
20    restart: unless-stopped
21    ports:
22      - "${KIBANA_PORT:-5601}:5601"
23    environment:
24      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
25    depends_on:
26      - elasticsearch
27
28  apm-server:
29    image: docker.elastic.co/apm/apm-server:8.11.0
30    container_name: apm-server
31    restart: unless-stopped
32    ports:
33      - "${APM_PORT:-8200}:8200"
34    environment:
35      - output.elasticsearch.hosts=["elasticsearch:9200"]
36    depends_on:
37      - elasticsearch
38
39volumes:
40  es_data:
41EOF
42
43# 2. Create the .env file
44cat > .env << 'EOF'
45# Elastic APM
46ES_PORT=9200
47KIBANA_PORT=5601
48APM_PORT=8200
49EOF
50
51# 3. Start the services
52docker compose up -d
53
54# 4. View logs
55docker compose logs -f

One-Liner

Run this command to download and set up the recipe in one step:

terminal

1curl -fsSL https://docker.recipes/api/recipes/elastic-apm-stack/run | bash

Troubleshooting

Elasticsearch container fails to start with 'max virtual memory areas vm.max_map_count too low': Increase vm.max_map_count to 262144 using sysctl -w vm.max_map_count=262144
APM agents cannot connect showing 'connection refused' errors: Verify APM server is accessible at port 8200 and check firewall rules between application and container network
Kibana displays 'Elasticsearch cluster did not respond' warnings: Wait 30-60 seconds for Elasticsearch to complete startup and index creation before accessing Kibana
High memory usage causing container restarts: Adjust ES_JAVA_OPTS heap settings based on available system memory, recommended minimum -Xms1g -Xmx1g for production
APM data not appearing in Kibana despite successful agent connections: Check APM server logs for parsing errors and verify agent compatibility with server version
Disk space filling rapidly with APM indices: Configure index lifecycle policies in Kibana to automatically delete old data after specified retention periods

Community Notes

Loading notes...

Download Recipe Kit

Get all files in a ready-to-deploy package

Includes docker-compose.yml, .env template, README, and license

Components

elasticsearchkibanaapm-server

Related Recipes

Ad Space

Shortcuts: C CopyF FavoriteD Download