# Grafana Tempo

> High-scale distributed tracing backend with minimal dependencies.

- Category: Monitoring & Observability
- Difficulty: intermediate
- Components: tempo, grafana
- Tags: tempo, grafana, tracing, distributed, opentelemetry
- Source: https://docker.recipes/monitoring/grafana-tempo

## Overview

Grafana Tempo is a high-volume, minimal-dependency distributed tracing backend designed to handle massive scale trace ingestion and storage. Built by Grafana Labs, Tempo focuses on cost-effective trace storage by leveraging object storage backends and requiring only basic infrastructure. It accepts traces in multiple formats including OpenTelemetry, Jaeger, and Zipkin, making it compatible with existing instrumentation without vendor lock-in. This deployment provides a complete distributed tracing solution by combining Tempo's powerful trace backend with Grafana's visualization capabilities. The stack deploys two focused services: Tempo handles trace ingestion and storage with multiple protocol endpoints, while Grafana provides the user interface for querying and visualizing trace data through its built-in Tempo data source integration. Organizations seeking to implement distributed tracing at scale will find this combination particularly valuable because it eliminates the complexity and cost of traditional tracing backends like Jaeger with Elasticsearch. The pairing enables teams to correlate traces with metrics and logs in a unified interface, providing complete observability coverage for microservices architectures and distributed systems.

## Features

- Multi-protocol trace ingestion supporting OpenTelemetry OTLP (gRPC/HTTP), Jaeger, and Zipkin formats
- Object storage backend support for cost-effective long-term trace retention at massive scale
- TraceQL query language for advanced trace search and filtering capabilities
- Grafana integration with native Tempo data source for trace visualization and exploration
- Minimal infrastructure requirements with no external database dependencies
- Automatic trace sampling and retention policies to manage storage costs
- Service map generation from trace data to visualize microservice dependencies
- Correlation with logs and metrics through Grafana's unified observability interface

## Use cases

- Microservices performance monitoring and latency analysis across distributed architectures
- Root cause analysis for complex distributed system failures and bottlenecks
- API response time optimization through detailed request flow tracing
- Service dependency mapping and architecture visualization in cloud-native environments
- Development team debugging of inter-service communication issues
- SRE incident response with trace-driven investigation workflows
- Cost-effective observability implementation for startups and growing engineering teams

## Prerequisites

- Docker and Docker Compose with at least 1GB available RAM for both services
- Understanding of distributed tracing concepts and OpenTelemetry instrumentation
- Application instrumentation already configured to export traces via OTLP, Jaeger, or Zipkin
- Tempo configuration file (tempo.yaml) prepared with storage and receiver settings
- Available ports 3000 (Grafana), 3200 (Tempo API), 4317 (OTLP gRPC), and 4318 (OTLP HTTP)
- Basic knowledge of Grafana dashboard creation and data source configuration

## docker-compose.yml

```yaml
services:
  tempo:
    image: grafana/tempo:latest
    container_name: tempo
    restart: unless-stopped
    command:
      - -config.file=/etc/tempo/tempo.yaml
    volumes:
      - ./tempo/tempo.yaml:/etc/tempo/tempo.yaml:ro
      - tempo_data:/tmp/tempo
    ports:
      - "3200:3200"
      - "4317:4317"
      - "4318:4318"
    networks:
      - tempo-network

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    restart: unless-stopped
    environment:
      GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_PASSWORD}
    ports:
      - "3000:3000"
    depends_on:
      - tempo
    networks:
      - tempo-network

volumes:
  tempo_data:

networks:
  tempo-network:
    driver: bridge
```

## .env

```bash
GRAFANA_PASSWORD=admin
```

## Usage notes

- Docs: https://grafana.com/docs/tempo/latest/
- Grafana at http://localhost:3000 (admin/GRAFANA_PASSWORD)
- Tempo API on port 3200, add as Tempo datasource in Grafana
- OTLP gRPC on 4317, HTTP on 4318 for trace ingestion
- Create tempo/tempo.yaml config before starting
- Pairs with Loki (logs) and Mimir (metrics) for full observability

## Troubleshooting

- Tempo container fails to start: Verify tempo.yaml configuration file exists and has correct YAML syntax
- Traces not appearing in Grafana: Check Tempo data source configuration uses http://tempo:3200 as URL
- OTLP trace ingestion failing: Confirm applications are sending traces to correct ports 4317 (gRPC) or 4318 (HTTP)
- High memory usage in Tempo: Adjust ingestion rate limits and trace sampling configuration in tempo.yaml
- Grafana cannot connect to Tempo: Ensure both services are on the same Docker network and Tempo is healthy
- TraceQL queries returning no results: Verify trace retention period and check trace IDs are being indexed correctly

