Varnish HTTP Cache Stack

intermediate

High-performance HTTP cache with Varnish and monitoring.

Overview

Varnish is a high-performance HTTP accelerator and reverse proxy cache originally developed at the Norwegian newspaper Verdens Gang to handle their massive web traffic. Unlike traditional caches that store data on disk, Varnish stores cached content directly in RAM, delivering microsecond response times and handling thousands of requests per second. Its VCL (Varnish Configuration Language) provides granular control over caching policies, allowing developers to define complex rules for what gets cached, for how long, and under what conditions. This stack combines Varnish's caching prowess with NGINX as the backend web server, creating a two-tier architecture where Varnish handles cache hits at lightning speed while NGINX serves the origin content. Prometheus collects detailed metrics from both Varnish and the custom varnish-exporter, tracking cache hit ratios, backend response times, and memory usage patterns. Grafana transforms this telemetry into visual dashboards that reveal cache performance bottlenecks and optimization opportunities. Content delivery networks, high-traffic publishing platforms, and e-commerce sites benefit most from this configuration. The stack excels when serving content with predictable caching patterns - news articles, product catalogs, API responses, and static assets. With Varnish's ability to serve cached content even when backends fail, combined with comprehensive monitoring, this setup provides both performance gains and operational visibility that traditional web servers alone cannot match.

Key Features

VCL-based cache policies with custom purging rules and conditional caching logic
RAM-based storage delivering sub-millisecond response times for cached content
Grace mode serves stale content when NGINX backend becomes unavailable
Prometheus varnish-exporter provides cache hit ratios and backend health metrics
NGINX backend handles dynamic content generation and SSL termination
Grafana dashboards visualize cache efficiency and identify optimization opportunities
ESI (Edge Side Includes) support for dynamic content assembly at the cache layer
Built-in load balancing across multiple NGINX backends with health checks

Common Use Cases

1High-traffic news websites caching article pages and reducing database load
2E-commerce platforms accelerating product catalog and search result delivery
3API gateways caching frequently requested endpoints and reducing backend calls
4WordPress and CMS hosting providers improving site performance for multiple clients
5CDN edge nodes serving regional content with detailed performance monitoring
6Development environments testing cache invalidation strategies before production
7Content aggregation platforms serving millions of article views with minimal infrastructure

Prerequisites

Minimum 1GB RAM (Varnish cache: 256MB + Prometheus: 256MB + Grafana: 512MB)
Ports 80, 3000, 9090, and 9131 available on the host system
Understanding of HTTP caching headers (Cache-Control, ETag, Vary)
Basic VCL syntax knowledge for customizing Varnish cache behavior
Familiarity with Prometheus query language (PromQL) for custom metrics
Docker Compose 3.8+ with volume and network support

For development & testing. Review security settings, change default credentials, and test thoroughly before production use. See Terms

docker-compose.yml

1services: 
2  nginx: 
3    image: nginx:alpine
4    volumes: 
5      - ./html:/usr/share/nginx/html:ro
6      - ./nginx.conf:/etc/nginx/nginx.conf:ro
7    networks: 
8      - varnish_net
9
10  varnish: 
11    image: varnish:stable
12    ports: 
13      - "80:80"
14    environment: 
15      - VARNISH_SIZE=256M
16    volumes: 
17      - ./default.vcl:/etc/varnish/default.vcl:ro
18    command: -a :80 -f /etc/varnish/default.vcl -s malloc,256M
19    depends_on: 
20      - nginx
21    networks: 
22      - varnish_net
23
24  varnish-exporter: 
25    image: jonnenauha/prometheus_varnish_exporter:latest
26    ports: 
27      - "9131:9131"
28    command: 
29      - -varnish.health-url=http://varnish/health
30    depends_on: 
31      - varnish
32    networks: 
33      - varnish_net
34
35  prometheus: 
36    image: prom/prometheus:latest
37    ports: 
38      - "9090:9090"
39    volumes: 
40      - ./prometheus.yml:/etc/prometheus/prometheus.yml
41      - prometheus_data:/prometheus
42    networks: 
43      - varnish_net
44
45  grafana: 
46    image: grafana/grafana:latest
47    ports: 
48      - "3000:3000"
49    environment: 
50      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
51    volumes: 
52      - grafana_data:/var/lib/grafana
53    networks: 
54      - varnish_net
55
56volumes: 
57  prometheus_data: 
58  grafana_data: 
59
60networks: 
61  varnish_net:

.env Template

.env

1# Varnish Cache
2GRAFANA_PASSWORD=secure_grafana_password
3
4# Varnish at http://localhost
5# Grafana at http://localhost:3000

Usage Notes

1Varnish at http://localhost
2Configure VCL for cache rules
3Monitor with Grafana
4Varnish stats via varnishstat
5Great for static content caching

Individual Services(5 services)

Copy individual services to mix and match with your existing compose files.

nginx

nginx:
  image: nginx:alpine
  volumes:
    - ./html:/usr/share/nginx/html:ro
    - ./nginx.conf:/etc/nginx/nginx.conf:ro
  networks:
    - varnish_net

varnish

varnish:
  image: varnish:stable
  ports:
    - "80:80"
  environment:
    - VARNISH_SIZE=256M
  volumes:
    - ./default.vcl:/etc/varnish/default.vcl:ro
  command: "-a :80 -f /etc/varnish/default.vcl -s malloc,256M"
  depends_on:
    - nginx
  networks:
    - varnish_net

varnish-exporter

varnish-exporter:
  image: jonnenauha/prometheus_varnish_exporter:latest
  ports:
    - "9131:9131"
  command:
    - "-varnish.health-url=http://varnish/health"
  depends_on:
    - varnish
  networks:
    - varnish_net

prometheus

prometheus:
  image: prom/prometheus:latest
  ports:
    - "9090:9090"
  volumes:
    - ./prometheus.yml:/etc/prometheus/prometheus.yml
    - prometheus_data:/prometheus
  networks:
    - varnish_net

grafana

grafana:
  image: grafana/grafana:latest
  ports:
    - "3000:3000"
  environment:
    - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
  volumes:
    - grafana_data:/var/lib/grafana
  networks:
    - varnish_net

Quick Start

terminal

1# 1. Create the compose file
2cat > docker-compose.yml << 'EOF'
3services:
4  nginx:
5    image: nginx:alpine
6    volumes:
7      - ./html:/usr/share/nginx/html:ro
8      - ./nginx.conf:/etc/nginx/nginx.conf:ro
9    networks:
10      - varnish_net
11
12  varnish:
13    image: varnish:stable
14    ports:
15      - "80:80"
16    environment:
17      - VARNISH_SIZE=256M
18    volumes:
19      - ./default.vcl:/etc/varnish/default.vcl:ro
20    command: -a :80 -f /etc/varnish/default.vcl -s malloc,256M
21    depends_on:
22      - nginx
23    networks:
24      - varnish_net
25
26  varnish-exporter:
27    image: jonnenauha/prometheus_varnish_exporter:latest
28    ports:
29      - "9131:9131"
30    command:
31      - -varnish.health-url=http://varnish/health
32    depends_on:
33      - varnish
34    networks:
35      - varnish_net
36
37  prometheus:
38    image: prom/prometheus:latest
39    ports:
40      - "9090:9090"
41    volumes:
42      - ./prometheus.yml:/etc/prometheus/prometheus.yml
43      - prometheus_data:/prometheus
44    networks:
45      - varnish_net
46
47  grafana:
48    image: grafana/grafana:latest
49    ports:
50      - "3000:3000"
51    environment:
52      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
53    volumes:
54      - grafana_data:/var/lib/grafana
55    networks:
56      - varnish_net
57
58volumes:
59  prometheus_data:
60  grafana_data:
61
62networks:
63  varnish_net:
64EOF
65
66# 2. Create the .env file
67cat > .env << 'EOF'
68# Varnish Cache
69GRAFANA_PASSWORD=secure_grafana_password
70
71# Varnish at http://localhost
72# Grafana at http://localhost:3000
73EOF
74
75# 3. Start the services
76docker compose up -d
77
78# 4. View logs
79docker compose logs -f

One-Liner

Run this command to download and set up the recipe in one step:

terminal

1curl -fsSL https://docker.recipes/api/recipes/varnish-cache-complete/run | bash

Troubleshooting

Varnish returns 503 Backend fetch failed: Check NGINX container health and verify network connectivity between varnish and nginx services
Cache hit ratio remains low despite configuration: Review VCL rules in default.vcl and ensure backend responses include proper Cache-Control headers
Grafana dashboard shows no Varnish metrics: Verify varnish-exporter container is running and accessible at port 9131, check prometheus.yml scrape configuration
Memory usage grows continuously in Varnish: Reduce VARNISH_SIZE environment variable or implement cache object TTL limits in VCL configuration
Prometheus target appears down for varnish-exporter: Update varnish-exporter health URL to match actual Varnish service name and port in Docker network
NGINX backend connection refused: Ensure nginx service starts before Varnish and verify nginx.conf backend server configuration matches service names

Community Notes

Loading notes...

Download Recipe Kit

Get all files in a ready-to-deploy package

Includes docker-compose.yml, .env template, README, and license

Components

varnishnginxprometheusgrafana

Related Recipes

Ad Space

Shortcuts: C CopyF FavoriteD Download