Elastic Stack (ELK) Complete

advanced

Full Elasticsearch, Logstash, Kibana stack with Beats agents and APM.

Overview

Elasticsearch is a distributed, RESTful search and analytics engine built on Apache Lucene that serves as the backbone of the Elastic Stack. Originally developed by Shay Banon in 2010, Elasticsearch revolutionized log analytics and full-text search by providing near real-time indexing, distributed architecture with automatic sharding, and powerful aggregation capabilities. Its RESTful API and JSON-based queries made it accessible to developers while its horizontal scaling capabilities attracted enterprises dealing with massive data volumes. This complete Elastic Stack deployment combines Elasticsearch with Logstash for data processing, Kibana for visualization, Filebeat and Metricbeat for data collection, and APM Server for application performance monitoring. Together, these components create a comprehensive observability platform that ingests, processes, stores, and visualizes machine data from any source. The stack transforms raw logs, metrics, and traces into actionable insights through Kibana's rich dashboards and Elasticsearch's powerful search capabilities. Organizations seeking unified observability, security teams building SIEM solutions, and DevOps engineers monitoring distributed systems will find this stack invaluable. The combination provides end-to-end visibility from application traces captured by APM Server to system metrics collected by Metricbeat, all processed through Logstash pipelines and made searchable in Elasticsearch. This deployment particularly benefits teams managing microservices architectures, containerized applications, or complex distributed systems where correlating data across multiple sources is crucial for troubleshooting and performance optimization.

Key Features

Full-text search with relevance scoring and advanced query DSL across all ingested data
Near real-time indexing and search capabilities with sub-second query response times
Logstash pipeline processing with 200+ input, filter, and output plugins for data transformation
Kibana Canvas for pixel-perfect data presentation and custom visualization creation
Machine learning anomaly detection for automatic identification of unusual patterns in logs and metrics
APM distributed tracing for end-to-end transaction visibility across microservices
Beat agents for lightweight data shipping from containers, systems, and applications
Index lifecycle management with automatic hot-warm-cold data tiering based on age and usage patterns

Common Use Cases

1Centralized logging for microservices architectures with distributed tracing correlation
2Security information and event management (SIEM) for threat detection and compliance reporting
3Application performance monitoring with code-level visibility and error tracking
4Infrastructure monitoring with real-time dashboards for system metrics and container health
5E-commerce search functionality with faceted navigation and personalized recommendations
6Business intelligence analytics with custom dashboards and automated alerting
7Compliance logging for financial services and healthcare organizations requiring audit trails

Prerequisites

Minimum 8GB RAM recommended (Elasticsearch requires 2GB minimum, additional memory for other components)
Docker Engine 20.10+ with Docker Compose V2 for multi-container orchestration
Available ports 5601, 9200, 9300, 8200, 5044, 5000, and 9600 for service communication
Understanding of Elasticsearch mapping concepts and Logstash pipeline configuration syntax
Basic knowledge of Kibana query languages (KQL/Lucene) for effective data exploration
Familiarity with YAML configuration for Beat agents and APM instrumentation setup

For development & testing. Review security settings, change default credentials, and test thoroughly before production use. See Terms

docker-compose.yml

1services: 
2  elasticsearch: 
3    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
4    environment: 
5      - discovery.type=single-node
6      - xpack.security.enabled=true
7      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
8      - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
9    volumes: 
10      - es_data:/usr/share/elasticsearch/data
11    ports: 
12      - "9200:9200"
13      - "9300:9300"
14    networks: 
15      - elastic_net
16
17  logstash: 
18    image: docker.elastic.co/logstash/logstash:8.11.0
19    volumes: 
20      - ./logstash/pipeline:/usr/share/logstash/pipeline:ro
21      - ./logstash/config:/usr/share/logstash/config:ro
22    ports: 
23      - "5044:5044"
24      - "5000:5000/tcp"
25      - "5000:5000/udp"
26      - "9600:9600"
27    environment: 
28      - xpack.monitoring.elasticsearch.hosts=http://elasticsearch:9200
29      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
30    depends_on: 
31      - elasticsearch
32    networks: 
33      - elastic_net
34
35  kibana: 
36    image: docker.elastic.co/kibana/kibana:8.11.0
37    ports: 
38      - "5601:5601"
39    environment: 
40      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
41      - ELASTICSEARCH_USERNAME=kibana_system
42      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
43    depends_on: 
44      - elasticsearch
45    networks: 
46      - elastic_net
47
48  filebeat: 
49    image: docker.elastic.co/beats/filebeat:8.11.0
50    user: root
51    volumes: 
52      - ./filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
53      - /var/lib/docker/containers:/var/lib/docker/containers:ro
54      - /var/run/docker.sock:/var/run/docker.sock:ro
55    environment: 
56      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
57    depends_on: 
58      - elasticsearch
59      - logstash
60    networks: 
61      - elastic_net
62
63  metricbeat: 
64    image: docker.elastic.co/beats/metricbeat:8.11.0
65    user: root
66    volumes: 
67      - ./metricbeat/metricbeat.yml:/usr/share/metricbeat/metricbeat.yml:ro
68      - /var/run/docker.sock:/var/run/docker.sock:ro
69      - /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
70      - /proc:/hostfs/proc:ro
71      - /:/hostfs:ro
72    environment: 
73      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
74    depends_on: 
75      - elasticsearch
76    networks: 
77      - elastic_net
78
79  apm-server: 
80    image: docker.elastic.co/apm/apm-server:8.11.0
81    ports: 
82      - "8200:8200"
83    environment: 
84      - output.elasticsearch.hosts=["elasticsearch:9200"]
85      - output.elasticsearch.username=elastic
86      - output.elasticsearch.password=${ELASTIC_PASSWORD}
87    depends_on: 
88      - elasticsearch
89    networks: 
90      - elastic_net
91
92volumes: 
93  es_data: 
94
95networks: 
96  elastic_net:

.env Template

.env

1# Elastic Stack
2ELASTIC_PASSWORD=secure_elastic_password
3KIBANA_PASSWORD=secure_kibana_password
4
5# Kibana at http://localhost:5601
6# Elasticsearch at http://localhost:9200
7# APM at http://localhost:8200

Usage Notes

1Kibana at http://localhost:5601
2Elasticsearch at http://localhost:9200
3APM Server at http://localhost:8200
4Logstash at port 5044 (Beats), 5000 (TCP/UDP)
5Default user: elastic/your_password

Individual Services(6 services)

Copy individual services to mix and match with your existing compose files.

elasticsearch

elasticsearch:
  image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
  environment:
    - discovery.type=single-node
    - xpack.security.enabled=true
    - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
    - ES_JAVA_OPTS=-Xms2g -Xmx2g
  volumes:
    - es_data:/usr/share/elasticsearch/data
  ports:
    - "9200:9200"
    - "9300:9300"
  networks:
    - elastic_net

logstash

logstash:
  image: docker.elastic.co/logstash/logstash:8.11.0
  volumes:
    - ./logstash/pipeline:/usr/share/logstash/pipeline:ro
    - ./logstash/config:/usr/share/logstash/config:ro
  ports:
    - "5044:5044"
    - 5000:5000/tcp
    - 5000:5000/udp
    - "9600:9600"
  environment:
    - xpack.monitoring.elasticsearch.hosts=http://elasticsearch:9200
    - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
  depends_on:
    - elasticsearch
  networks:
    - elastic_net

kibana

kibana:
  image: docker.elastic.co/kibana/kibana:8.11.0
  ports:
    - "5601:5601"
  environment:
    - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    - ELASTICSEARCH_USERNAME=kibana_system
    - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
  depends_on:
    - elasticsearch
  networks:
    - elastic_net

filebeat

filebeat:
  image: docker.elastic.co/beats/filebeat:8.11.0
  user: root
  volumes:
    - ./filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
    - /var/lib/docker/containers:/var/lib/docker/containers:ro
    - /var/run/docker.sock:/var/run/docker.sock:ro
  environment:
    - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
  depends_on:
    - elasticsearch
    - logstash
  networks:
    - elastic_net

metricbeat

metricbeat:
  image: docker.elastic.co/beats/metricbeat:8.11.0
  user: root
  volumes:
    - ./metricbeat/metricbeat.yml:/usr/share/metricbeat/metricbeat.yml:ro
    - /var/run/docker.sock:/var/run/docker.sock:ro
    - /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
    - /proc:/hostfs/proc:ro
    - /:/hostfs:ro
  environment:
    - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
  depends_on:
    - elasticsearch
  networks:
    - elastic_net

apm-server

apm-server:
  image: docker.elastic.co/apm/apm-server:8.11.0
  ports:
    - "8200:8200"
  environment:
    - output.elasticsearch.hosts=["elasticsearch:9200"]
    - output.elasticsearch.username=elastic
    - output.elasticsearch.password=${ELASTIC_PASSWORD}
  depends_on:
    - elasticsearch
  networks:
    - elastic_net

Quick Start

terminal

1# 1. Create the compose file
2cat > docker-compose.yml << 'EOF'
3services:
4  elasticsearch:
5    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
6    environment:
7      - discovery.type=single-node
8      - xpack.security.enabled=true
9      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
10      - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
11    volumes:
12      - es_data:/usr/share/elasticsearch/data
13    ports:
14      - "9200:9200"
15      - "9300:9300"
16    networks:
17      - elastic_net
18
19  logstash:
20    image: docker.elastic.co/logstash/logstash:8.11.0
21    volumes:
22      - ./logstash/pipeline:/usr/share/logstash/pipeline:ro
23      - ./logstash/config:/usr/share/logstash/config:ro
24    ports:
25      - "5044:5044"
26      - "5000:5000/tcp"
27      - "5000:5000/udp"
28      - "9600:9600"
29    environment:
30      - xpack.monitoring.elasticsearch.hosts=http://elasticsearch:9200
31      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
32    depends_on:
33      - elasticsearch
34    networks:
35      - elastic_net
36
37  kibana:
38    image: docker.elastic.co/kibana/kibana:8.11.0
39    ports:
40      - "5601:5601"
41    environment:
42      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
43      - ELASTICSEARCH_USERNAME=kibana_system
44      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
45    depends_on:
46      - elasticsearch
47    networks:
48      - elastic_net
49
50  filebeat:
51    image: docker.elastic.co/beats/filebeat:8.11.0
52    user: root
53    volumes:
54      - ./filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
55      - /var/lib/docker/containers:/var/lib/docker/containers:ro
56      - /var/run/docker.sock:/var/run/docker.sock:ro
57    environment:
58      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
59    depends_on:
60      - elasticsearch
61      - logstash
62    networks:
63      - elastic_net
64
65  metricbeat:
66    image: docker.elastic.co/beats/metricbeat:8.11.0
67    user: root
68    volumes:
69      - ./metricbeat/metricbeat.yml:/usr/share/metricbeat/metricbeat.yml:ro
70      - /var/run/docker.sock:/var/run/docker.sock:ro
71      - /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
72      - /proc:/hostfs/proc:ro
73      - /:/hostfs:ro
74    environment:
75      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
76    depends_on:
77      - elasticsearch
78    networks:
79      - elastic_net
80
81  apm-server:
82    image: docker.elastic.co/apm/apm-server:8.11.0
83    ports:
84      - "8200:8200"
85    environment:
86      - output.elasticsearch.hosts=["elasticsearch:9200"]
87      - output.elasticsearch.username=elastic
88      - output.elasticsearch.password=${ELASTIC_PASSWORD}
89    depends_on:
90      - elasticsearch
91    networks:
92      - elastic_net
93
94volumes:
95  es_data:
96
97networks:
98  elastic_net:
99EOF
100
101# 2. Create the .env file
102cat > .env << 'EOF'
103# Elastic Stack
104ELASTIC_PASSWORD=secure_elastic_password
105KIBANA_PASSWORD=secure_kibana_password
106
107# Kibana at http://localhost:5601
108# Elasticsearch at http://localhost:9200
109# APM at http://localhost:8200
110EOF
111
112# 3. Start the services
113docker compose up -d
114
115# 4. View logs
116docker compose logs -f

One-Liner

Run this command to download and set up the recipe in one step:

terminal

1curl -fsSL https://docker.recipes/api/recipes/elastic-stack-complete/run | bash

Troubleshooting

Elasticsearch cluster health red: Check disk space and memory allocation, ensure ES_JAVA_OPTS heap size doesn't exceed 50% of available RAM
Kibana 'Unable to connect to Elasticsearch' error: Verify ELASTICSEARCH_HOSTS environment variable and ensure Elasticsearch is fully started before Kibana
Logstash pipeline not processing data: Check pipeline configuration syntax in ./logstash/pipeline directory and verify input/output plugin connectivity
Filebeat not shipping container logs: Ensure Docker socket mount permissions and verify filebeat.yml autodiscover configuration for Docker containers
APM Server rejecting agent data: Confirm APM agents are configured with correct server URL (localhost:8200) and authentication credentials match ELASTIC_PASSWORD
High memory usage on Elasticsearch: Adjust ES_JAVA_OPTS heap size, implement index lifecycle policies, and consider fielddata circuit breaker settings

Community Notes

Loading notes...

Download Recipe Kit

Get all files in a ready-to-deploy package

Includes docker-compose.yml, .env template, README, and license

Components

elasticsearchlogstashkibanafilebeatmetricbeatapm-server

Related Recipes

Ad Space

Shortcuts: C CopyF FavoriteD Download