# Jaeger Distributed Tracing

> Jaeger for distributed tracing with Elasticsearch.

- Category: Monitoring & Observability
- Difficulty: intermediate
- Components: jaeger, elasticsearch, otel-collector
- Tags: jaeger, tracing, observability, opentelemetry
- Source: https://docker.recipes/monitoring/jaeger-tracing-stack

## Overview

Jaeger is an open-source, end-to-end distributed tracing system originally developed by Uber Technologies to monitor and troubleshoot microservices-based architectures. It helps developers understand request flows across complex distributed systems by tracking individual requests as they traverse multiple services, providing crucial visibility into performance bottlenecks, error propagation, and service dependencies. Jaeger implements the OpenTracing specification and supports OpenTelemetry standards, making it a cornerstone tool for observability in cloud-native environments.

This stack combines Jaeger with Elasticsearch as the storage backend and OpenTelemetry Collector as the telemetry data pipeline processor. Elasticsearch provides scalable, searchable storage for trace data with advanced querying capabilities, while the OpenTelemetry Collector acts as a vendor-agnostic telemetry processor that can receive, transform, and forward trace data from multiple sources. This configuration creates a production-grade tracing infrastructure that can handle high-throughput environments and complex trace data processing requirements.

Development teams working with microservices, site reliability engineers monitoring distributed systems, and DevOps teams implementing observability strategies will benefit from this stack. The Jaeger-Elasticsearch combination offers superior trace search performance and data retention capabilities compared to Jaeger's in-memory storage, while the OpenTelemetry Collector provides flexibility to collect traces from various instrumentation libraries and process them before storage, making this setup ideal for organizations requiring robust distributed tracing at scale.

## Features

- Elasticsearch backend storage for scalable trace data persistence with full-text search capabilities
- OpenTelemetry Protocol (OTLP) support for both gRPC and HTTP trace ingestion
- Service dependency graph visualization showing real-time service topology
- Distributed context propagation tracking requests across service boundaries
- Performance anomaly detection through trace sampling and analysis
- Multi-tenant trace isolation with advanced filtering and search operators
- Adaptive sampling strategies to manage trace volume without losing critical data
- Integration with Elasticsearch aggregations for custom trace analytics and dashboards

## Use cases

- Microservices performance monitoring in e-commerce platforms with complex checkout flows
- Root cause analysis for distributed system failures in financial trading applications
- API latency optimization for mobile backends serving millions of users
- Service mesh observability for Kubernetes clusters running containerized workloads
- Third-party integration monitoring for SaaS platforms with external API dependencies
- Database query optimization through trace analysis in data-intensive applications
- Error propagation tracking in event-driven architectures using message queues

## Prerequisites

- Minimum 6GB RAM available (4GB for Elasticsearch, 2GB for Jaeger and collector)
- Docker Engine 20.10+ with Docker Compose V2 support
- Ports 16686, 4317, 4318, and 4316 available on the host system
- Basic understanding of distributed tracing concepts and OpenTelemetry standards
- OpenTelemetry Collector configuration file (otel-config.yaml) in project directory
- Application instrumentation with OpenTelemetry SDKs or Jaeger client libraries

## docker-compose.yml

```yaml
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
    container_name: jaeger-es
    restart: unless-stopped
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
    volumes:
      - es_data:/usr/share/elasticsearch/data

  jaeger:
    image: jaegertracing/all-in-one:latest
    container_name: jaeger
    restart: unless-stopped
    ports:
      - "${JAEGER_UI_PORT:-16686}:16686"
      - "${OTLP_GRPC_PORT:-4317}:4317"
      - "${OTLP_HTTP_PORT:-4318}:4318"
    environment:
      - SPAN_STORAGE_TYPE=elasticsearch
      - ES_SERVER_URLS=http://elasticsearch:9200
    depends_on:
      - elasticsearch

  otel-collector:
    image: otel/opentelemetry-collector:latest
    container_name: otel-collector
    restart: unless-stopped
    ports:
      - "${OTEL_GRPC_PORT:-4316}:4317"
    volumes:
      - ./otel-config.yaml:/etc/otel-config.yaml:ro
    command: ["--config=/etc/otel-config.yaml"]

volumes:
  es_data:
```

## .env

```bash
# Jaeger Tracing
JAEGER_UI_PORT=16686
OTLP_GRPC_PORT=4317
OTLP_HTTP_PORT=4318
OTEL_GRPC_PORT=4316
```

## Usage notes

- Jaeger UI at http://localhost:16686
- Send traces to port 4317 (OTLP)
- View service dependencies

## Troubleshooting

- Elasticsearch container exits with 'max virtual memory areas vm.max_map_count too low': Run 'sudo sysctl -w vm.max_map_count=262144' on the host system
- Jaeger UI shows 'No traces found' despite sending data: Check that applications are sending traces to the correct OTLP endpoints (4317 for gRPC, 4318 for HTTP)
- OpenTelemetry Collector fails to start with 'config file not found': Ensure otel-config.yaml exists in the project root directory with proper YAML formatting
- High memory usage in Elasticsearch: Configure index lifecycle policies to automatically delete old trace data after retention period
- Traces not appearing in Jaeger UI: Verify Jaeger environment variable ES_SERVER_URLS points to correct Elasticsearch endpoint and check collector pipeline configuration
- Connection refused errors between services: Ensure all containers are on the same Docker network and using correct internal service names for communication

