# Vector Log Pipeline

> High-performance observability pipeline with Vector, ClickHouse storage, and Grafana.

- Category: Monitoring & Observability
- Difficulty: advanced
- Components: vector, clickhouse, grafana, nginx
- Tags: vector, logging, clickhouse, pipeline, observability
- Source: https://docker.recipes/monitoring/vector-log-pipeline

## Overview

Vector is a high-performance observability data pipeline developed by Datadog that transforms, routes, and processes logs, metrics, and traces at scale. Originally created to address the limitations of traditional log processing tools like Logstash and Fluentd, Vector provides better performance, memory safety through Rust, and more flexible data transformation capabilities. This pipeline configuration creates a complete log analytics solution where Vector collects and processes log data from multiple sources, stores it in ClickHouse's columnar database for fast analytical queries, and visualizes the results through Grafana dashboards. ClickHouse serves as the storage backend, providing sub-second query performance on billions of log entries through its column-oriented architecture and aggressive compression. The combination eliminates the need for expensive proprietary logging solutions while delivering enterprise-grade performance for log analytics, real-time monitoring, and operational insights. This stack particularly excels for organizations processing high-volume log data who need both real-time ingestion capabilities and fast historical analysis. Vector's ability to parse, filter, and transform logs in-flight combined with ClickHouse's analytical performance creates a powerful alternative to ELK stack deployments, especially when SQL-based log analysis is preferred over Elasticsearch's query DSL.

## Features

- Vector's high-performance log ingestion with support for 50+ data sources including syslog, Docker, Kubernetes, and file tailing
- Real-time log transformation and filtering using Vector Remap Language (VRL) for field extraction and data enrichment
- ClickHouse columnar storage with 10-100x faster analytical queries compared to traditional row-based log storage
- Grafana ClickHouse data source integration for SQL-based dashboard creation and log exploration
- Vector's backpressure handling and automatic retries ensuring reliable log delivery to ClickHouse
- ClickHouse MergeTree engine optimized for time-series log data with automatic partitioning and TTL policies
- NGINX reverse proxy providing unified access point and load balancing for the entire observability stack
- Vector's built-in metrics and health endpoints for monitoring the pipeline performance itself

## Use cases

- Application log analytics for debugging and performance monitoring in microservices architectures
- Infrastructure monitoring for server logs, system metrics, and container orchestration platforms
- Security information and event management (SIEM) with fast log correlation and threat detection queries
- Business intelligence dashboards combining application logs with operational metrics for product insights
- Compliance logging with long-term retention and fast audit trail queries using ClickHouse's compression
- DevOps teams replacing expensive Splunk or DataDog deployments with open-source log analytics
- Real-time alerting based on log patterns and anomaly detection using Grafana's alerting rules

## Prerequisites

- Minimum 8GB RAM recommended for ClickHouse analytical workloads and log retention
- Docker host with sufficient disk I/O performance for high-volume log ingestion and ClickHouse storage
- Network access to log sources that Vector will collect from (ports 514 for syslog, Docker socket access)
- Basic understanding of SQL for creating ClickHouse schemas and Grafana dashboard queries
- Familiarity with Vector Remap Language (VRL) for log parsing and transformation configuration
- Available ports: 8686 (Vector API), 8123 (ClickHouse HTTP), 3000 (Grafana), 9000 (Vector sources)

## docker-compose.yml

```yaml
services:
  vector:
    image: timberio/vector:latest-alpine
    ports:
      - "8686:8686"
      - "9000:9000"
    volumes:
      - ./vector.toml:/etc/vector/vector.toml:ro
      - /var/log:/var/log:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    depends_on:
      - clickhouse
    networks:
      - vector-net
    restart: unless-stopped

  clickhouse:
    image: clickhouse/clickhouse-server:latest
    ports:
      - "8123:8123"
      - "9001:9000"
    volumes:
      - clickhouse_data:/var/lib/clickhouse
      - ./clickhouse-config.xml:/etc/clickhouse-server/config.d/config.xml:ro
    environment:
      CLICKHOUSE_DB: ${CLICKHOUSE_DB}
      CLICKHOUSE_USER: ${CLICKHOUSE_USER}
      CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD}
    ulimits:
      nofile:
        soft: 262144
        hard: 262144
    networks:
      - vector-net
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_PASSWORD}
      GF_INSTALL_PLUGINS: grafana-clickhouse-datasource
    volumes:
      - grafana_data:/var/lib/grafana
    depends_on:
      - clickhouse
    networks:
      - vector-net
    restart: unless-stopped

  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - grafana
    networks:
      - vector-net
    restart: unless-stopped

volumes:
  clickhouse_data:
  grafana_data:

networks:
  vector-net:
    driver: bridge
```

## .env

```bash
# ClickHouse
CLICKHOUSE_DB=logs
CLICKHOUSE_USER=vector
CLICKHOUSE_PASSWORD=secure_clickhouse_password

# Grafana
GRAFANA_PASSWORD=secure_grafana_password
```

## Usage notes

- Vector API at http://localhost:8686
- ClickHouse HTTP at http://localhost:8123
- Grafana dashboards at http://localhost:3000
- Configure vector.toml for sources and sinks

## Troubleshooting

- ClickHouse 'Memory limit exceeded' errors: Increase max_memory_usage setting or add more RAM to handle large analytical queries
- Vector pipeline lag or dropped logs: Check backpressure configuration and ClickHouse insert performance, tune batch sizes in vector.toml
- Grafana ClickHouse datasource connection failed: Verify ClickHouse HTTP interface is accessible on port 8123 and credentials are correct
- Vector sources not receiving logs: Check file permissions for /var/log access and Docker socket permissions for container logs
- ClickHouse out of disk space: Configure TTL policies for automatic log rotation and enable compression in MergeTree settings
- High Vector CPU usage during parsing: Optimize VRL transform functions and consider using native ClickHouse parsing functions instead

