# Apache Druid Analytics

> Druid for real-time analytics with ZooKeeper.

- Category: Database Stacks
- Difficulty: advanced
- Components: druid, zookeeper, postgresql
- Tags: druid, analytics, olap, real-time
- Source: https://docker.recipes/databases/druid-analytics-cluster

## Overview

Apache Druid is a high-performance, column-oriented, distributed data store designed for fast slice-and-dice analytics on large datasets. Originally developed at Metamarkets and later open-sourced, Druid excels at ingesting streaming data and providing sub-second query responses for OLAP workloads, making it ideal for real-time dashboards, user-facing analytics, and interactive data exploration. Unlike traditional databases, Druid pre-aggregates data at ingestion time and uses advanced indexing techniques to achieve lightning-fast query performance on time-series data.

This stack combines Druid's distributed architecture with PostgreSQL as the metadata store and ZooKeeper for service coordination. PostgreSQL maintains Druid's cluster metadata, segment information, and configuration data with ACID compliance, while ZooKeeper handles service discovery, leader election, and coordination between Druid's various node types. The configuration deploys three essential Druid services: the Coordinator manages data availability and balancing, the Broker handles query routing and merging results, and the Router provides a unified query endpoint for client applications.

Data engineers and analysts working with high-volume streaming data should consider this stack when traditional databases cannot meet sub-second query requirements. This combination particularly benefits organizations ingesting millions of events per day from sources like web analytics, IoT sensors, or application logs, where users need to drill down into data with complex filters and aggregations in real-time. The stack scales horizontally and provides the foundation for building responsive analytics applications that can handle both batch and streaming data ingestion patterns.

## Features

- Sub-second OLAP queries on billion-row datasets through columnar storage and bitmap indexing
- Real-time data ingestion from Kafka streams with automatic rollup and pre-aggregation
- Automatic data tiering between hot and cold storage based on time-based rules
- Built-in approximation algorithms for fast cardinality estimation and quantile calculations
- Native JSON support for nested data structures and flexible schema evolution
- Automatic query parallelization across Historical nodes with intelligent data pruning
- PostgreSQL-backed metadata storage ensuring ACID compliance for cluster state
- ZooKeeper coordination enabling automatic failover and service discovery across nodes

## Use cases

- Real-time web analytics dashboards showing user behavior and conversion funnels
- IoT sensor data analysis for manufacturing equipment monitoring and predictive maintenance
- Financial trading platforms requiring millisecond query responses on market data
- Gaming analytics tracking player behavior and in-game economy metrics
- Advertising technology platforms analyzing bid requests and campaign performance
- Network monitoring systems processing log data for security and performance insights
- E-commerce recommendation engines analyzing clickstream data for personalization

## Prerequisites

- Minimum 4GB RAM (Druid Broker and Coordinator require 1-2GB each, plus PostgreSQL and ZooKeeper overhead)
- Available ports 8081 (Coordinator), 8082 (Broker), 8888 (Router), 5432 (PostgreSQL), 2181 (ZooKeeper)
- Understanding of OLAP concepts, time-series data modeling, and data warehouse architectures
- Familiarity with JSON-based data ingestion specifications and Druid's segment structure
- Basic knowledge of JVM tuning for optimal memory allocation across Druid services
- Experience with PostgreSQL administration for metadata store maintenance and backups

## docker-compose.yml

```yaml
services:
  postgres:
    image: postgres:15-alpine
    container_name: druid-postgres
    restart: unless-stopped
    environment:
      - POSTGRES_USER=druid
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      - POSTGRES_DB=druid
    volumes:
      - postgres_data:/var/lib/postgresql/data

  zookeeper:
    image: zookeeper:3.9
    container_name: druid-zk
    restart: unless-stopped
    volumes:
      - zk_data:/data

  coordinator:
    image: apache/druid:latest
    container_name: druid-coordinator
    restart: unless-stopped
    ports:
      - "${COORDINATOR_PORT:-8081}:8081"
    environment:
      - druid_metadata_storage_type=postgresql
      - druid_zk_service_host=zookeeper
    command: coordinator
    depends_on:
      - postgres
      - zookeeper

  broker:
    image: apache/druid:latest
    container_name: druid-broker
    restart: unless-stopped
    ports:
      - "${BROKER_PORT:-8082}:8082"
    environment:
      - druid_zk_service_host=zookeeper
    command: broker

  router:
    image: apache/druid:latest
    container_name: druid-router
    restart: unless-stopped
    ports:
      - "${ROUTER_PORT:-8888}:8888"
    command: router

volumes:
  postgres_data:
  zk_data:
```

## .env

```bash
# Apache Druid
COORDINATOR_PORT=8081
BROKER_PORT=8082
ROUTER_PORT=8888
POSTGRES_PASSWORD=druid_password
```

## Usage notes

- Druid Console at http://localhost:8888
- Coordinator at http://localhost:8081
- Add historical nodes for production

## Troubleshooting

- Coordinator shows 'No segments available' error: Verify Historical nodes are running and connected to ZooKeeper, check segment loading rules in Coordinator console
- Queries timeout or return incomplete results: Increase Broker query timeout settings and verify all Historical nodes are healthy and segments are distributed
- PostgreSQL connection failures during startup: Ensure PostgreSQL is fully initialized before Druid services start, add healthcheck dependencies between containers
- ZooKeeper connection lost errors: Verify ZooKeeper has sufficient memory allocation and stable network connectivity, check ZooKeeper logs for session timeouts
- High memory usage and OOM crashes: Adjust JVM heap settings using DRUID_XMX and DRUID_XMS environment variables based on available container resources
- Ingestion tasks fail with metadata errors: Check PostgreSQL disk space and connection pool settings, verify Druid metadata tables are properly initialized

