docker.recipes

Jaeger Distributed Tracing

intermediate

Jaeger for distributed tracing with Elasticsearch.

Overview

Jaeger is an open-source, end-to-end distributed tracing system originally developed by Uber Technologies to monitor and troubleshoot microservices-based architectures. It helps developers understand request flows across complex distributed systems by tracking individual requests as they traverse multiple services, providing crucial visibility into performance bottlenecks, error propagation, and service dependencies. Jaeger implements the OpenTracing specification and supports OpenTelemetry standards, making it a cornerstone tool for observability in cloud-native environments. This stack combines Jaeger with Elasticsearch as the storage backend and OpenTelemetry Collector as the telemetry data pipeline processor. Elasticsearch provides scalable, searchable storage for trace data with advanced querying capabilities, while the OpenTelemetry Collector acts as a vendor-agnostic telemetry processor that can receive, transform, and forward trace data from multiple sources. This configuration creates a production-grade tracing infrastructure that can handle high-throughput environments and complex trace data processing requirements. Development teams working with microservices, site reliability engineers monitoring distributed systems, and DevOps teams implementing observability strategies will benefit from this stack. The Jaeger-Elasticsearch combination offers superior trace search performance and data retention capabilities compared to Jaeger's in-memory storage, while the OpenTelemetry Collector provides flexibility to collect traces from various instrumentation libraries and process them before storage, making this setup ideal for organizations requiring robust distributed tracing at scale.

Key Features

  • Elasticsearch backend storage for scalable trace data persistence with full-text search capabilities
  • OpenTelemetry Protocol (OTLP) support for both gRPC and HTTP trace ingestion
  • Service dependency graph visualization showing real-time service topology
  • Distributed context propagation tracking requests across service boundaries
  • Performance anomaly detection through trace sampling and analysis
  • Multi-tenant trace isolation with advanced filtering and search operators
  • Adaptive sampling strategies to manage trace volume without losing critical data
  • Integration with Elasticsearch aggregations for custom trace analytics and dashboards

Common Use Cases

  • 1Microservices performance monitoring in e-commerce platforms with complex checkout flows
  • 2Root cause analysis for distributed system failures in financial trading applications
  • 3API latency optimization for mobile backends serving millions of users
  • 4Service mesh observability for Kubernetes clusters running containerized workloads
  • 5Third-party integration monitoring for SaaS platforms with external API dependencies
  • 6Database query optimization through trace analysis in data-intensive applications
  • 7Error propagation tracking in event-driven architectures using message queues

Prerequisites

  • Minimum 6GB RAM available (4GB for Elasticsearch, 2GB for Jaeger and collector)
  • Docker Engine 20.10+ with Docker Compose V2 support
  • Ports 16686, 4317, 4318, and 4316 available on the host system
  • Basic understanding of distributed tracing concepts and OpenTelemetry standards
  • OpenTelemetry Collector configuration file (otel-config.yaml) in project directory
  • Application instrumentation with OpenTelemetry SDKs or Jaeger client libraries

For development & testing. Review security settings, change default credentials, and test thoroughly before production use. See Terms

docker-compose.yml

docker-compose.yml
1services: 
2  elasticsearch: 
3    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
4    container_name: jaeger-es
5    restart: unless-stopped
6    environment: 
7      - discovery.type=single-node
8      - xpack.security.enabled=false
9    volumes: 
10      - es_data:/usr/share/elasticsearch/data
11
12  jaeger: 
13    image: jaegertracing/all-in-one:latest
14    container_name: jaeger
15    restart: unless-stopped
16    ports: 
17      - "${JAEGER_UI_PORT:-16686}:16686"
18      - "${OTLP_GRPC_PORT:-4317}:4317"
19      - "${OTLP_HTTP_PORT:-4318}:4318"
20    environment: 
21      - SPAN_STORAGE_TYPE=elasticsearch
22      - ES_SERVER_URLS=http://elasticsearch:9200
23    depends_on: 
24      - elasticsearch
25
26  otel-collector: 
27    image: otel/opentelemetry-collector:latest
28    container_name: otel-collector
29    restart: unless-stopped
30    ports: 
31      - "${OTEL_GRPC_PORT:-4316}:4317"
32    volumes: 
33      - ./otel-config.yaml:/etc/otel-config.yaml:ro
34    command: ["--config=/etc/otel-config.yaml"]
35
36volumes: 
37  es_data:

.env Template

.env
1# Jaeger Tracing
2JAEGER_UI_PORT=16686
3OTLP_GRPC_PORT=4317
4OTLP_HTTP_PORT=4318
5OTEL_GRPC_PORT=4316

Usage Notes

  1. 1Jaeger UI at http://localhost:16686
  2. 2Send traces to port 4317 (OTLP)
  3. 3View service dependencies

Individual Services(3 services)

Copy individual services to mix and match with your existing compose files.

elasticsearch
elasticsearch:
  image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
  container_name: jaeger-es
  restart: unless-stopped
  environment:
    - discovery.type=single-node
    - xpack.security.enabled=false
  volumes:
    - es_data:/usr/share/elasticsearch/data
jaeger
jaeger:
  image: jaegertracing/all-in-one:latest
  container_name: jaeger
  restart: unless-stopped
  ports:
    - ${JAEGER_UI_PORT:-16686}:16686
    - ${OTLP_GRPC_PORT:-4317}:4317
    - ${OTLP_HTTP_PORT:-4318}:4318
  environment:
    - SPAN_STORAGE_TYPE=elasticsearch
    - ES_SERVER_URLS=http://elasticsearch:9200
  depends_on:
    - elasticsearch
otel-collector
otel-collector:
  image: otel/opentelemetry-collector:latest
  container_name: otel-collector
  restart: unless-stopped
  ports:
    - ${OTEL_GRPC_PORT:-4316}:4317
  volumes:
    - ./otel-config.yaml:/etc/otel-config.yaml:ro
  command:
    - "--config=/etc/otel-config.yaml"

Quick Start

terminal
1# 1. Create the compose file
2cat > docker-compose.yml << 'EOF'
3services:
4  elasticsearch:
5    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
6    container_name: jaeger-es
7    restart: unless-stopped
8    environment:
9      - discovery.type=single-node
10      - xpack.security.enabled=false
11    volumes:
12      - es_data:/usr/share/elasticsearch/data
13
14  jaeger:
15    image: jaegertracing/all-in-one:latest
16    container_name: jaeger
17    restart: unless-stopped
18    ports:
19      - "${JAEGER_UI_PORT:-16686}:16686"
20      - "${OTLP_GRPC_PORT:-4317}:4317"
21      - "${OTLP_HTTP_PORT:-4318}:4318"
22    environment:
23      - SPAN_STORAGE_TYPE=elasticsearch
24      - ES_SERVER_URLS=http://elasticsearch:9200
25    depends_on:
26      - elasticsearch
27
28  otel-collector:
29    image: otel/opentelemetry-collector:latest
30    container_name: otel-collector
31    restart: unless-stopped
32    ports:
33      - "${OTEL_GRPC_PORT:-4316}:4317"
34    volumes:
35      - ./otel-config.yaml:/etc/otel-config.yaml:ro
36    command: ["--config=/etc/otel-config.yaml"]
37
38volumes:
39  es_data:
40EOF
41
42# 2. Create the .env file
43cat > .env << 'EOF'
44# Jaeger Tracing
45JAEGER_UI_PORT=16686
46OTLP_GRPC_PORT=4317
47OTLP_HTTP_PORT=4318
48OTEL_GRPC_PORT=4316
49EOF
50
51# 3. Start the services
52docker compose up -d
53
54# 4. View logs
55docker compose logs -f

One-Liner

Run this command to download and set up the recipe in one step:

terminal
1curl -fsSL https://docker.recipes/api/recipes/jaeger-tracing-stack/run | bash

Troubleshooting

  • Elasticsearch container exits with 'max virtual memory areas vm.max_map_count too low': Run 'sudo sysctl -w vm.max_map_count=262144' on the host system
  • Jaeger UI shows 'No traces found' despite sending data: Check that applications are sending traces to the correct OTLP endpoints (4317 for gRPC, 4318 for HTTP)
  • OpenTelemetry Collector fails to start with 'config file not found': Ensure otel-config.yaml exists in the project root directory with proper YAML formatting
  • High memory usage in Elasticsearch: Configure index lifecycle policies to automatically delete old trace data after retention period
  • Traces not appearing in Jaeger UI: Verify Jaeger environment variable ES_SERVER_URLS points to correct Elasticsearch endpoint and check collector pipeline configuration
  • Connection refused errors between services: Ensure all containers are on the same Docker network and using correct internal service names for communication

Community Notes

Loading...
Loading notes...

Download Recipe Kit

Get all files in a ready-to-deploy package

Includes docker-compose.yml, .env template, README, and license

Components

jaegerelasticsearchotel-collector

Tags

#jaeger#tracing#observability#opentelemetry

Category

Monitoring & Observability

Related Recipes

Ad Space