# Vector Log Aggregator Pipeline

> High-performance observability pipeline for logs, metrics, and traces.

- Category: Monitoring & Observability
- Difficulty: intermediate
- Components: vector, clickhouse, grafana, prometheus
- Tags: vector, datadog, logs, metrics, pipeline
- Source: https://docker.recipes/monitoring/vector-aggregator

## Overview

Vector is a high-performance observability data pipeline built in Rust by Datadog that efficiently collects, transforms, and routes logs, metrics, and traces. Originally developed to address the performance limitations and memory overhead of traditional log aggregators like Fluentd and Logstash, Vector can process millions of events per second with minimal resource consumption. This stack combines Vector's powerful data processing capabilities with ClickHouse's columnar storage engine for analytics, Prometheus for metrics collection, and Grafana for visualization. The combination creates a complete observability pipeline where Vector acts as the central data router, ingesting logs from multiple sources, transforming and enriching the data, then distributing it to ClickHouse for long-term storage and analysis while exposing its own operational metrics to Prometheus. This architecture is particularly valuable for organizations that need to process large volumes of observability data while maintaining control over their data pipeline, costs, and performance. The stack offers enterprise-grade observability capabilities without the recurring costs of SaaS solutions, making it ideal for companies with significant log volumes, compliance requirements, or those seeking to build custom analytics on their observability data.

## Features

- Vector's memory-mapped buffer system ensures data durability and prevents data loss during restarts or failures
- ClickHouse's columnar storage provides sub-second query performance on billions of log entries with automatic data compression
- Vector's VRL (Vector Remap Language) enables complex log parsing, enrichment, and transformation without external dependencies
- Prometheus integration captures Vector's internal metrics including throughput, error rates, and buffer utilization
- ClickHouse materialized views automatically aggregate log data for faster dashboard queries and reporting
- Vector's adaptive concurrency automatically optimizes sink throughput based on downstream system performance
- Grafana's ClickHouse data source plugin enables SQL-based log exploration with time-series visualization
- Vector's topology health checks monitor sink connectivity and automatically buffer data during outages

## Use cases

- High-volume application log aggregation for e-commerce platforms processing millions of transactions daily
- Security operations center (SOC) log analysis pipeline for threat detection and incident response
- Multi-tenant SaaS platform centralized logging with per-tenant data isolation and cost tracking
- Kubernetes cluster observability with pod log collection, metric extraction, and distributed tracing correlation
- Compliance logging for financial services requiring tamper-proof audit trails and long-term retention
- Real-time fraud detection system processing payment logs with complex pattern matching and alerting
- DevOps pipeline monitoring aggregating CI/CD logs, deployment metrics, and infrastructure health data

## Prerequisites

- Minimum 8GB RAM recommended (2GB for ClickHouse, 1GB for Prometheus, 512MB for Grafana, remainder for Vector buffers)
- Basic understanding of TOML configuration syntax for Vector pipeline definitions
- Familiarity with PromQL for creating custom metrics dashboards and alerts
- Knowledge of ClickHouse SQL dialect differences from standard SQL for log analysis queries
- Available ports 3000, 8123, 8686, 9000, 9090, and 514/6514 for service communication
- Understanding of log formats (JSON, syslog, CEF) that will be ingested by Vector

## docker-compose.yml

```yaml
services:
  vector:
    image: timberio/vector:latest-alpine
    ports:
      - "8686:8686"
      - "9598:9598"
      - "514:514/udp"
      - "6514:6514"
    volumes:
      - ./vector.toml:/etc/vector/vector.toml:ro
      - vector_data:/var/lib/vector
    networks:
      - vector_net

  clickhouse:
    image: clickhouse/clickhouse-server:latest
    ports:
      - "8123:8123"
      - "9000:9000"
    environment:
      - CLICKHOUSE_USER=${CLICKHOUSE_USER}
      - CLICKHOUSE_PASSWORD=${CLICKHOUSE_PASSWORD}
      - CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1
    volumes:
      - clickhouse_data:/var/lib/clickhouse
    networks:
      - vector_net

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    networks:
      - vector_net

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
    volumes:
      - grafana_data:/var/lib/grafana
    depends_on:
      - clickhouse
      - prometheus
    networks:
      - vector_net

volumes:
  vector_data:
  clickhouse_data:
  prometheus_data:
  grafana_data:

networks:
  vector_net:
```

## .env

```bash
# Vector Pipeline
CLICKHOUSE_USER=default
CLICKHOUSE_PASSWORD=secure_clickhouse_password
GRAFANA_PASSWORD=secure_grafana_password

# Vector API at http://localhost:8686
# ClickHouse at http://localhost:8123
```

## Usage notes

- Vector API at http://localhost:8686
- Syslog input at port 514/UDP
- ClickHouse for log storage
- Configure vector.toml for pipelines
- Written in Rust for performance

## Troubleshooting

- Vector buffer full errors: Increase buffer size in vector.toml or check ClickHouse write performance with SHOW PROCESSLIST
- ClickHouse connection refused: Verify CLICKHOUSE_USER and CLICKHOUSE_PASSWORD environment variables match container settings
- Vector transformation parsing errors: Enable Vector debug logging and validate VRL syntax using vector validate command
- Grafana ClickHouse queries timing out: Add appropriate indexes on timestamp and frequently queried columns in ClickHouse
- High memory usage in Vector: Reduce buffer.max_events setting and enable compression in sink configurations
- Missing logs in ClickHouse: Check Vector logs for sink errors and verify ClickHouse table schema matches Vector output format

