# Grafana OnCall

> On-call management system integrated with Grafana alerting.

- Category: Monitoring & Observability
- Difficulty: intermediate
- Components: grafana, oncall-engine, oncall-celery, postgresql, redis, rabbitmq
- Tags: oncall, alerting, grafana, incident-management, pagerduty-alternative
- Source: https://docker.recipes/monitoring/grafana-oncall

## Overview

Grafana OnCall is an open-source incident response and on-call management system developed by Grafana Labs. It provides comprehensive alerting, escalation policies, and team collaboration features for managing critical incidents, serving as a cost-effective alternative to commercial solutions like PagerDuty and Opsgenie. The platform integrates natively with Grafana's monitoring ecosystem while supporting multiple notification channels including Slack, Telegram, SMS, and phone calls.

This deployment creates a complete on-call management infrastructure using six interconnected services: a Grafana instance with the OnCall plugin pre-installed, the main oncall-engine service running the web application via uWSGI, oncall-celery for background task processing and scheduled operations, PostgreSQL for persistent data storage, Redis for caching and session management, and RabbitMQ as the message broker for task distribution. The oncall-engine handles API requests and user interactions while oncall-celery processes notifications, escalations, and periodic maintenance tasks in the background.

This stack is ideal for DevOps teams, SREs, and organizations seeking enterprise-grade incident management capabilities without vendor lock-in or per-user pricing. The combination provides robust scheduling, intelligent alert grouping, escalation chains, and detailed incident tracking while maintaining complete control over sensitive operational data and notification workflows.

## Features

- Native Grafana integration with pre-configured OnCall plugin for unified monitoring and alerting workflows
- Intelligent alert grouping and deduplication to reduce notification fatigue during incident storms
- Flexible escalation policies with multiple notification channels including Slack, Telegram, SMS, and voice calls
- Advanced scheduling system supporting rotations, overrides, and time-zone aware on-call management
- Real-time incident collaboration with acknowledgments, resolution tracking, and team communication
- Background task processing via Celery for reliable notification delivery and scheduled operations
- PostgreSQL-backed incident history and analytics with full audit trails and reporting capabilities
- Multi-tenancy support with team-based access controls and customizable notification preferences

## Use cases

- 24/7 production monitoring for SaaS applications requiring immediate incident response
- DevOps teams managing microservices architectures with complex alert routing and escalation needs
- Organizations migrating from expensive commercial on-call solutions like PagerDuty or Opsgenie
- Enterprise environments requiring on-premises incident management with strict data sovereignty requirements
- Multi-team engineering organizations needing centralized on-call scheduling and rotation management
- Startups building scalable incident response processes without per-user licensing costs
- Healthcare or financial services requiring compliant incident tracking and audit capabilities

## Prerequisites

- Docker and Docker Compose installed with minimum 4GB RAM available for all services
- Port 3000 available for Grafana web interface access
- Valid SMTP configuration or external notification service credentials for alert delivery
- Environment variables configured: GRAFANA_PASSWORD, DB_PASSWORD, RABBITMQ_PASSWORD, SECRET_KEY, BASE_URL
- Understanding of Grafana dashboards and alerting rules for integration with existing monitoring
- Basic knowledge of on-call management concepts like escalation policies and rotation schedules

## docker-compose.yml

```yaml
services:
  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
      - GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=grafana-oncall-app
      - GF_INSTALL_PLUGINS=grafana-oncall-app
    volumes:
      - grafana-data:/var/lib/grafana
    ports:
      - "3000:3000"
    networks:
      - oncall-network
    restart: unless-stopped

  oncall-engine:
    image: grafana/oncall:latest
    container_name: oncall-engine
    command: uwsgi --ini uwsgi.ini
    environment:
      - DATABASE_TYPE=postgresql
      - DATABASE_HOST=db
      - DATABASE_NAME=oncall
      - DATABASE_USER=oncall
      - DATABASE_PASSWORD=${DB_PASSWORD}
      - BROKER_TYPE=rabbitmq
      - RABBITMQ_HOST=rabbitmq
      - RABBITMQ_USERNAME=oncall
      - RABBITMQ_PASSWORD=${RABBITMQ_PASSWORD}
      - REDIS_URI=redis://redis:6379
      - SECRET_KEY=${SECRET_KEY}
      - BASE_URL=${BASE_URL}
      - GRAFANA_API_URL=http://grafana:3000
    depends_on:
      - db
      - redis
      - rabbitmq
    networks:
      - oncall-network
    restart: unless-stopped

  oncall-celery:
    image: grafana/oncall:latest
    container_name: oncall-celery
    command: celery -A engine worker -l info -B --scheduler=django_celery_beat.schedulers:DatabaseScheduler
    environment:
      - DATABASE_TYPE=postgresql
      - DATABASE_HOST=db
      - DATABASE_NAME=oncall
      - DATABASE_USER=oncall
      - DATABASE_PASSWORD=${DB_PASSWORD}
      - BROKER_TYPE=rabbitmq
      - RABBITMQ_HOST=rabbitmq
      - RABBITMQ_USERNAME=oncall
      - RABBITMQ_PASSWORD=${RABBITMQ_PASSWORD}
      - REDIS_URI=redis://redis:6379
      - SECRET_KEY=${SECRET_KEY}
    depends_on:
      - oncall-engine
    networks:
      - oncall-network
    restart: unless-stopped

  db:
    image: postgres:15-alpine
    container_name: oncall-db
    environment:
      - POSTGRES_USER=oncall
      - POSTGRES_PASSWORD=${DB_PASSWORD}
      - POSTGRES_DB=oncall
    volumes:
      - postgres-data:/var/lib/postgresql/data
    networks:
      - oncall-network
    restart: unless-stopped

  redis:
    image: redis:7-alpine
    container_name: oncall-redis
    volumes:
      - redis-data:/data
    networks:
      - oncall-network
    restart: unless-stopped

  rabbitmq:
    image: rabbitmq:3-management-alpine
    container_name: oncall-rabbitmq
    environment:
      - RABBITMQ_DEFAULT_USER=oncall
      - RABBITMQ_DEFAULT_PASS=${RABBITMQ_PASSWORD}
    volumes:
      - rabbitmq-data:/var/lib/rabbitmq
    networks:
      - oncall-network
    restart: unless-stopped

volumes:
  grafana-data:
  postgres-data:
  redis-data:
  rabbitmq-data:

networks:
  oncall-network:
    driver: bridge
```

## .env

```bash
# Grafana OnCall
BASE_URL=http://localhost:8080
GRAFANA_PASSWORD=secure_grafana_password
DB_PASSWORD=secure_oncall_password
RABBITMQ_PASSWORD=secure_rabbitmq_password

# Generate with: openssl rand -hex 32
SECRET_KEY=your_secret_key_here
```

## Usage notes

- Grafana at http://localhost:3000
- Enable OnCall plugin in Grafana
- Create schedules and escalation policies
- Integrates with Slack, Telegram, etc.
- PagerDuty/Opsgenie alternative

## Troubleshooting

- OnCall plugin not visible in Grafana: Verify GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS is set and restart grafana container
- oncall-engine startup fails with database errors: Ensure db container is fully initialized before oncall-engine starts, check DATABASE_PASSWORD environment variable
- Celery workers not processing tasks: Verify rabbitmq container is running and RABBITMQ_PASSWORD matches between oncall-celery and rabbitmq services
- Notifications not being delivered: Check celery worker logs in oncall-celery container and verify external notification service credentials
- Redis connection timeouts: Increase Redis memory limits or check for memory pressure on host system affecting redis container performance
- Grafana API integration errors: Verify GRAFANA_API_URL points to correct internal service name (http://grafana:3000) and admin credentials are correct

