Elastic APM Stack

intermediate

Elasticsearch APM with Kibana for application monitoring.

[i]Overview

Elasticsearch is a distributed search and analytics engine built on Apache Lucene that has become the cornerstone of modern observability platforms. Originally developed by Elastic N.V., it provides real-time indexing and search capabilities across massive datasets, making it ideal for log analytics, application performance monitoring, and business intelligence. Its distributed architecture with automatic sharding and replication ensures high availability and horizontal scalability. This Elastic APM Stack combines Elasticsearch with Kibana (the visualization and management interface) and APM Server (the data collection component) to create a comprehensive application performance monitoring solution. APM Server receives performance data from instrumented applications via lightweight agents, processes and enriches this data, then stores it in Elasticsearch indices optimized for time-series analysis. Kibana provides pre-built dashboards and visualizations specifically designed for APM data, including service maps, transaction traces, and error analytics. Development teams and DevOps engineers seeking deep visibility into application performance, error tracking, and dependency mapping will find this stack invaluable. Unlike basic logging solutions, this combination provides distributed tracing capabilities that track requests across microservices, automatic service discovery and mapping, and machine learning-powered anomaly detection for proactive issue identification.

[*]Key Features

[+]Distributed tracing across microservices with automatic span correlation and timing analysis
[+]Real-time application metrics collection including response times, throughput, and error rates
[+]Service topology mapping that automatically discovers and visualizes application dependencies
[+]Machine learning anomaly detection for proactive identification of performance degradations
[+]Custom dashboards and alerting through Kibana's visualization engine
[+]Support for multiple programming languages through official APM agents
[+]Index lifecycle management for automatic data retention and cost optimization
[+]Cross-cluster search capabilities for monitoring distributed application environments

[#]Common Use Cases

[1]Microservices performance monitoring in production environments with distributed request tracing
[2]E-commerce platform optimization by identifying slow database queries and API bottlenecks
[3]SaaS application error tracking and root cause analysis across multiple service tiers
[4]DevOps teams implementing observability-driven development practices with CI/CD integration
[5]Enterprise applications requiring compliance auditing and performance SLA monitoring
[6]Startup development teams establishing performance baselines during rapid scaling phases
[7]Legacy application modernization projects needing visibility during cloud migration

[!]Prerequisites

[!]Minimum 4GB RAM available for Elasticsearch heap allocation and optimal query performance
[!]Docker and Docker Compose installed with at least 20GB available disk space for indices
[!]Network ports 9200, 5601, and 8200 available and not conflicting with existing services
[!]Basic understanding of APM concepts including transactions, spans, and distributed tracing
[!]Application instrumentation knowledge for integrating APM agents into your codebase
[!]JVM tuning experience recommended for production Elasticsearch deployments

[!]

WARNING: For development & testing. Review security settings, change default credentials, and test thoroughly before production use. See Terms

[$]docker-compose.yml

[docker-compose.yml]

1services: 
2  elasticsearch: 
3    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
4    container_name: elastic-apm-es
5    restart: unless-stopped
6    environment: 
7      - discovery.type=single-node
8      - xpack.security.enabled=false
9      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
10    volumes: 
11      - es_data:/usr/share/elasticsearch/data
12    ports: 
13      - "${ES_PORT:-9200}:9200"
14
15  kibana: 
16    image: docker.elastic.co/kibana/kibana:8.11.0
17    container_name: elastic-apm-kibana
18    restart: unless-stopped
19    ports: 
20      - "${KIBANA_PORT:-5601}:5601"
21    environment: 
22      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
23    depends_on: 
24      - elasticsearch
25
26  apm-server: 
27    image: docker.elastic.co/apm/apm-server:8.11.0
28    container_name: apm-server
29    restart: unless-stopped
30    ports: 
31      - "${APM_PORT:-8200}:8200"
32    environment: 
33      - output.elasticsearch.hosts=["elasticsearch:9200"]
34    depends_on: 
35      - elasticsearch
36
37volumes: 
38  es_data:

[$].env Template

[.env]

1# Elastic APM
2ES_PORT=9200
3KIBANA_PORT=5601
4APM_PORT=8200

[i]Usage Notes

[1]Kibana at http://localhost:5601
[2]APM Server at http://localhost:8200
[3]Configure APM agents with server URL
[4]View APM in Kibana Observability

Individual Services(3 services)

Copy individual services to mix and match with your existing compose files.

elasticsearch

elasticsearch:
  image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
  container_name: elastic-apm-es
  restart: unless-stopped
  environment:
    - discovery.type=single-node
    - xpack.security.enabled=false
    - ES_JAVA_OPTS=-Xms512m -Xmx512m
  volumes:
    - es_data:/usr/share/elasticsearch/data
  ports:
    - ${ES_PORT:-9200}:9200

kibana

kibana:
  image: docker.elastic.co/kibana/kibana:8.11.0
  container_name: elastic-apm-kibana
  restart: unless-stopped
  ports:
    - ${KIBANA_PORT:-5601}:5601
  environment:
    - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
  depends_on:
    - elasticsearch

apm-server

apm-server:
  image: docker.elastic.co/apm/apm-server:8.11.0
  container_name: apm-server
  restart: unless-stopped
  ports:
    - ${APM_PORT:-8200}:8200
  environment:
    - output.elasticsearch.hosts=["elasticsearch:9200"]
  depends_on:
    - elasticsearch

[>]Quick Start

[terminal]

1# 1. Create the compose file
2cat > docker-compose.yml << 'EOF'
3services:
4  elasticsearch:
5    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
6    container_name: elastic-apm-es
7    restart: unless-stopped
8    environment:
9      - discovery.type=single-node
10      - xpack.security.enabled=false
11      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
12    volumes:
13      - es_data:/usr/share/elasticsearch/data
14    ports:
15      - "${ES_PORT:-9200}:9200"
16
17  kibana:
18    image: docker.elastic.co/kibana/kibana:8.11.0
19    container_name: elastic-apm-kibana
20    restart: unless-stopped
21    ports:
22      - "${KIBANA_PORT:-5601}:5601"
23    environment:
24      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
25    depends_on:
26      - elasticsearch
27
28  apm-server:
29    image: docker.elastic.co/apm/apm-server:8.11.0
30    container_name: apm-server
31    restart: unless-stopped
32    ports:
33      - "${APM_PORT:-8200}:8200"
34    environment:
35      - output.elasticsearch.hosts=["elasticsearch:9200"]
36    depends_on:
37      - elasticsearch
38
39volumes:
40  es_data:
41EOF
42
43# 2. Create the .env file
44cat > .env << 'EOF'
45# Elastic APM
46ES_PORT=9200
47KIBANA_PORT=5601
48APM_PORT=8200
49EOF
50
51# 3. Start the services
52docker compose up -d
53
54# 4. View logs
55docker compose logs -f

[>]One-Liner

Run this command to download and set up the recipe in one step:

[terminal]

1curl -fsSL https://docker.recipes/api/recipes/elastic-apm-stack/run | bash

[?]Troubleshooting

[!]Elasticsearch container fails to start with 'max virtual memory areas vm.max_map_count too low': Increase vm.max_map_count to 262144 using sysctl -w vm.max_map_count=262144
[!]APM agents cannot connect showing 'connection refused' errors: Verify APM server is accessible at port 8200 and check firewall rules between application and container network
[!]Kibana displays 'Elasticsearch cluster did not respond' warnings: Wait 30-60 seconds for Elasticsearch to complete startup and index creation before accessing Kibana
[!]High memory usage causing container restarts: Adjust ES_JAVA_OPTS heap settings based on available system memory, recommended minimum -Xms1g -Xmx1g for production
[!]APM data not appearing in Kibana despite successful agent connections: Check APM server logs for parsing errors and verify agent compatibility with server version
[!]Disk space filling rapidly with APM indices: Configure index lifecycle policies in Kibana to automatically delete old data after specified retention periods

Community Notes

Loading notes...

## Download Recipe Kit

Get all files in a ready-to-deploy package

Includes docker-compose.yml, .env template, README, and license

## Components

elasticsearchkibanaapm-server

## Tags

#elastic#apm#observability#kibana

## Category

Monitoring & Observability

## Related

Shortcuts: C CopyF FavoriteD Download