# Open Source Datadog Alternative

> Full-featured observability platform with metrics, logs, traces, and dashboards.

- Category: Monitoring & Observability
- Difficulty: advanced
- Components: victoriametrics, grafana, loki, tempo, vector, alertmanager
- Tags: monitoring, observability, metrics, logs, traces
- Source: https://docker.recipes/monitoring/datadog-alternative

## Overview

VictoriaMetrics is a fast, cost-effective time-series database designed as a high-performance alternative to Prometheus, offering significantly better data compression and query performance while maintaining full PromQL compatibility. Originally developed by VictoriaMetrics team in 2019, it has become a popular choice for organizations seeking to reduce infrastructure costs while scaling their metrics storage beyond Prometheus limitations. This observability stack combines VictoriaMetrics with Grafana for visualization, Loki for log aggregation, Tempo for distributed tracing, Vector for log collection, and Alertmanager for notifications.

This particular combination creates a comprehensive observability platform that rivals commercial solutions like Datadog by providing the three pillars of observability: metrics through VictoriaMetrics, logs through Loki, and traces through Tempo. The stack leverages Vector as a high-performance data pipeline for log collection and transformation, while Grafana serves as the unified interface for querying and visualizing all telemetry data. Alertmanager handles notification routing and alert deduplication across the entire stack, creating a cohesive monitoring experience.

This configuration is ideal for engineering teams, DevOps professionals, and organizations looking to implement enterprise-grade observability without the recurring costs of SaaS solutions. Startups burning through observability budgets, enterprises with data sovereignty requirements, and infrastructure teams managing Kubernetes clusters will find this stack particularly valuable. The combination provides the functionality of premium monitoring platforms while maintaining complete control over data retention, storage costs, and customization.

## Features

- VictoriaMetrics high-compression storage with 90-day retention configured for long-term metrics analysis
- VMAgent automatic service discovery and metrics scraping with Prometheus configuration compatibility
- Loki label-based log indexing that dramatically reduces storage costs compared to full-text search solutions
- Tempo distributed tracing with OTLP receivers supporting both gRPC (4317) and HTTP (4318) protocols
- Vector high-performance log collection with Docker socket access for container log aggregation
- Grafana TraceQL editor integration for advanced trace querying and correlation with metrics
- Alertmanager notification routing with support for Slack, email, PagerDuty, and webhook integrations
- Cross-component correlation enabling jump from metrics to logs to traces within Grafana interface

## Use cases

- Kubernetes cluster monitoring with pod metrics, container logs, and distributed trace correlation
- Application performance monitoring for microservices architectures requiring trace analysis
- Infrastructure cost optimization by replacing expensive SaaS monitoring solutions
- Regulatory compliance environments requiring on-premises data retention and sovereignty
- Development teams needing full observability stack in local or staging environments
- High-volume metrics storage where Prometheus storage limitations become bottlenecks
- Multi-tenant environments requiring separate data isolation and retention policies

## Prerequisites

- Minimum 4GB RAM recommended (VictoriaMetrics 512MB, Grafana 512MB, Loki 1GB, Tempo 512MB, others 256MB each)
- Available ports 3000, 3100, 3200, 4317, 4318, 8428, 8429, 8686, 9093 on host system
- Docker socket access permissions for Vector container log collection functionality
- Configuration files: loki-config.yml, tempo.yml, vector.toml, alertmanager.yml, and prometheus.yml
- Understanding of PromQL query language for VictoriaMetrics metrics exploration
- Familiarity with LogQL for Loki log querying and TraceQL for Tempo trace analysis

## docker-compose.yml

```yaml
services:
  victoriametrics:
    image: victoriametrics/victoria-metrics:latest
    ports:
      - "8428:8428"
    volumes:
      - vm_data:/victoria-metrics-data
    command:
      - '--storageDataPath=/victoria-metrics-data'
      - '--retentionPeriod=90d'
      - '--httpListenAddr=:8428'
    networks:
      - obs-net
    restart: unless-stopped

  vmagent:
    image: victoriametrics/vmagent:latest
    ports:
      - "8429:8429"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - vmagent_data:/vmagent-data
    command:
      - '--promscrape.config=/etc/prometheus/prometheus.yml'
      - '--remoteWrite.url=http://victoriametrics:8428/api/v1/write'
    depends_on:
      - victoriametrics
    networks:
      - obs-net
    restart: unless-stopped

  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yml:/etc/loki/local-config.yaml:ro
      - loki_data:/loki
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - obs-net
    restart: unless-stopped

  tempo:
    image: grafana/tempo:latest
    ports:
      - "3200:3200"
      - "4317:4317"
      - "4318:4318"
    volumes:
      - ./tempo.yml:/etc/tempo/tempo.yaml:ro
      - tempo_data:/tmp/tempo
    command: -config.file=/etc/tempo/tempo.yaml
    networks:
      - obs-net
    restart: unless-stopped

  vector:
    image: timberio/vector:latest-alpine
    ports:
      - "8686:8686"
    volumes:
      - ./vector.toml:/etc/vector/vector.toml:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      - obs-net
    restart: unless-stopped

  alertmanager:
    image: prom/alertmanager:latest
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
      - alertmanager_data:/alertmanager
    command:
      - '--config.file=/etc/alertmanager/alertmanager.yml'
      - '--storage.path=/alertmanager'
    networks:
      - obs-net
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
      - GF_FEATURE_TOGGLES_ENABLE=traceqlEditor
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning
    depends_on:
      - victoriametrics
      - loki
      - tempo
    networks:
      - obs-net
    restart: unless-stopped

volumes:
  vm_data:
  vmagent_data:
  loki_data:
  tempo_data:
  alertmanager_data:
  grafana_data:

networks:
  obs-net:
    driver: bridge
```

## .env

```bash
# Grafana
GRAFANA_PASSWORD=secure_grafana_password

# VictoriaMetrics
VM_RETENTION_PERIOD=90d

# Alertmanager Slack webhook (optional)
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/xxx/xxx/xxx
```

## Usage notes

- Grafana at http://localhost:3000
- VictoriaMetrics at http://localhost:8428
- Loki at http://localhost:3100
- Tempo traces at http://localhost:3200
- Create config files for Loki, Tempo, and Vector

## Troubleshooting

- VictoriaMetrics 'no space left on device' error: Increase Docker volume size or adjust --retentionPeriod to reduce storage usage
- Loki 'entry out of order' ingestion failures: Configure Vector or log shippers with proper timestamp sorting and batch settings
- Tempo traces not appearing in Grafana: Verify OTLP endpoint connectivity and check tempo.yml receiver configuration matches application instrumentation
- Vector container logs not collected: Ensure Docker socket mount permissions and check vector.toml Docker source configuration
- Grafana data source connection timeouts: Verify inter-service network connectivity and check service health endpoints before configuring data sources
- Alertmanager notifications not sending: Validate alertmanager.yml routing configuration and test webhook/SMTP connectivity from container network

