# Vector Log Pipeline

> High-performance observability data pipeline for logs, metrics, and traces.

- Category: Monitoring & Observability
- Difficulty: intermediate
- Components: vector, clickhouse, grafana
- Tags: logging, vector, pipeline, observability, metrics
- Source: https://docker.recipes/monitoring/vector-logs

## Overview

Vector is a high-performance, Rust-based observability data pipeline that transforms, routes, and ships logs, metrics, and traces at scale. Originally developed by Timber.io (now part of Datadog), Vector addresses the complexity and performance limitations of traditional log processing tools by providing a unified platform that can handle millions of events per second with minimal resource overhead. Its vendor-neutral design and rich transformation capabilities make it ideal for organizations seeking to standardize their observability infrastructure without vendor lock-in.

This stack combines Vector's powerful data processing capabilities with ClickHouse's columnar analytics engine and Grafana's visualization platform to create a complete observability solution. Vector ingests logs from multiple sources, applies transformations and enrichments, then streams the processed data to ClickHouse for storage and analysis. ClickHouse's column-oriented architecture excels at analytical queries over large log datasets, providing sub-second query performance even with billions of events, while Grafana creates interactive dashboards and alerts from the stored data.

This combination is particularly valuable for engineering teams and SREs who need real-time log analytics without the cost and complexity of managed solutions like Splunk or Elastic Cloud. The stack supports high-throughput environments where traditional ELK stacks struggle with performance, and its SQL-based querying in ClickHouse makes it accessible to teams already familiar with relational databases. Organizations processing hundreds of GB to TB of logs daily will find this setup provides enterprise-grade capabilities at a fraction of the operational cost.

## Features

- Vector's topology-based configuration for complex data routing and transformation pipelines
- Real-time log parsing with Vector's built-in VRL (Vector Remap Language) for field extraction and enrichment
- ClickHouse's MergeTree engine optimized for time-series log data with automatic partitioning
- Sub-second analytical queries on billions of log events using ClickHouse's columnar storage
- Native ClickHouse data source integration in Grafana for log analytics dashboards
- Vector's built-in metrics and health endpoints for pipeline observability
- ClickHouse materialized views for pre-aggregated log metrics and faster dashboard queries
- Grafana Explore interface for ad-hoc log querying using SQL syntax

## Use cases

- High-volume application log analytics for microservices architectures generating 100GB+ daily
- Security information and event management (SIEM) with real-time threat detection queries
- Infrastructure monitoring for Kubernetes clusters with container log aggregation and analysis
- Cost-effective replacement for Splunk or Elastic Cloud in enterprise environments
- Real-time business intelligence dashboards based on application event logs
- Compliance logging with long-term retention and fast audit queries
- DevOps troubleshooting with correlated logs, metrics, and traces in a single interface

## Prerequisites

- Docker host with minimum 4GB RAM (8GB+ recommended for production workloads)
- Available ports 3000, 8123, 8686, 9000, and 9598 for service access
- Basic understanding of log formats and parsing requirements for your data sources
- Vector configuration file (vector.toml) defining sources, transforms, and sinks
- ClickHouse users configuration file for authentication and access control
- Sufficient disk space for log retention (ClickHouse achieves 10:1 compression ratios typically)

## docker-compose.yml

```yaml
services:
  vector:
    image: timberio/vector:latest-alpine
    container_name: vector
    volumes:
      - ./vector.toml:/etc/vector/vector.toml:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - vector-data:/var/lib/vector
    ports:
      - "8686:8686"
      - "9598:9598"
    networks:
      - vector-network
    restart: unless-stopped

  clickhouse:
    image: clickhouse/clickhouse-server:latest
    container_name: vector-clickhouse
    volumes:
      - clickhouse-data:/var/lib/clickhouse
      - ./clickhouse-users.xml:/etc/clickhouse-server/users.d/users.xml:ro
    ports:
      - "8123:8123"
      - "9000:9000"
    networks:
      - vector-network
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    container_name: vector-grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
      - GF_INSTALL_PLUGINS=grafana-clickhouse-datasource
    volumes:
      - grafana-data:/var/lib/grafana
    ports:
      - "3000:3000"
    networks:
      - vector-network
    restart: unless-stopped

volumes:
  vector-data:
  clickhouse-data:
  grafana-data:

networks:
  vector-network:
    driver: bridge
```

## .env

```bash
# Vector
GRAFANA_PASSWORD=secure_grafana_password

# Create vector.toml configuration file
```

## Usage notes

- Vector API at http://localhost:8686
- Metrics at :9598
- ClickHouse at :8123
- Grafana at http://localhost:3000
- High-performance Rust-based pipeline

## Troubleshooting

- Vector pipeline stalled with no data flowing: Check vector.toml syntax and verify source connectivity using vector validate command
- ClickHouse connection refused on port 9000: Ensure ClickHouse container is healthy and users.xml contains proper authentication settings
- Grafana ClickHouse plugin installation fails: Restart Grafana container after plugin installation or use pre-built image with plugin
- High memory usage in ClickHouse: Tune max_memory_usage settings and verify proper partitioning keys in table schemas
- Vector backpressure warnings: Increase buffer sizes in vector.toml or optimize ClickHouse insert performance with batch settings
- Dashboard queries timing out in Grafana: Add proper indexes to ClickHouse tables and use time-based query filters

