# Kestra

> Event-driven orchestration platform.

- Category: Message Queues & Brokers
- Difficulty: intermediate
- Components: kestra, postgres
- Tags: kestra, orchestration, workflows, events
- Source: https://docker.recipes/messaging/kestra

## Overview

Kestra is a modern event-driven orchestration platform designed to simplify complex workflow automation across diverse environments. Created to address the limitations of traditional workflow schedulers, Kestra combines the power of event-driven architecture with declarative YAML-based workflow definitions, supporting over 400 plugins for integrating with databases, cloud services, APIs, and messaging systems. Unlike legacy orchestrators that rely primarily on time-based scheduling, Kestra enables workflows to respond to real-world events like file changes, database updates, webhook calls, and Kafka messages. PostgreSQL serves as Kestra's robust metadata store, handling workflow definitions, execution history, task states, and audit logs with full ACID compliance. The database's advanced JSON support proves essential for storing Kestra's complex workflow metadata and execution artifacts, while its reliability ensures workflow state consistency even during high-concurrency scenarios. This combination delivers a production-grade orchestration platform capable of handling enterprise-scale automation while maintaining the flexibility needed for rapid workflow development. This stack appeals to DevOps teams transitioning from legacy schedulers like Airflow or Jenkins, data engineers building modern ELT pipelines, and platform teams implementing event-driven microservice orchestration. The PostgreSQL backend provides the reliability and performance characteristics required for mission-critical workflow automation, while Kestra's plugin ecosystem and event-driven capabilities enable sophisticated integration patterns that traditional cron-based schedulers cannot achieve.

## Features

- Event-driven workflow triggers supporting schedules, webhooks, file watchers, Kafka consumers, and database change streams
- Visual workflow topology with real-time execution monitoring and interactive dependency graphs
- Built-in secrets management with variable templating and environment-specific configuration inheritance
- PostgreSQL-backed execution history with queryable logs, metrics, and audit trails for compliance reporting
- Plugin ecosystem with 400+ pre-built connectors for databases, cloud platforms, APIs, and data processing tools
- Multi-tenant namespace isolation with role-based access control and team-based workflow organization
- Advanced workflow patterns including parallel execution, conditional branching, error handling, and retry policies
- RESTful API on port 8081 enabling programmatic workflow management and integration with external systems

## Use cases

- Data pipeline orchestration replacing Airflow with event-driven ETL processes triggered by file uploads or database changes
- Microservice workflow coordination responding to Kafka events for order processing, user onboarding, or payment workflows
- Infrastructure automation triggered by monitoring alerts, combining Terraform deployments with notification workflows
- CI/CD pipeline orchestration beyond traditional build tools, handling complex deployment workflows across multiple environments
- Business process automation connecting CRM webhooks to email campaigns, invoice generation, and customer notification sequences
- IoT data processing workflows triggered by sensor data streams, combining real-time analytics with automated responses
- Multi-cloud resource management coordinating deployments, backups, and cost optimization across AWS, GCP, and Azure

## Prerequisites

- Docker Engine 20.10+ with Docker Compose v2 for container orchestration support
- Minimum 2GB RAM allocated to Docker (1GB+ for PostgreSQL, additional overhead for Kestra workflows)
- Available ports 8080 (Kestra Web UI) and 8081 (Kestra API) on the Docker host
- Environment variables DB_USER and DB_PASSWORD configured for PostgreSQL authentication
- Basic understanding of YAML syntax for workflow definition and event-driven architecture concepts
- Familiarity with PostgreSQL administration for backup strategies and performance tuning in production environments

## docker-compose.yml

```yaml
services:
  kestra:
    image: kestra/kestra:latest
    container_name: kestra
    restart: unless-stopped
    command: server standalone
    environment:
      KESTRA_CONFIGURATION: |
        datasources:
          postgres:
            url: jdbc:postgresql://postgres:5432/kestra
            driverClassName: org.postgresql.Driver
            username: ${DB_USER}
            password: ${DB_PASSWORD}
    ports:
      - "8080:8080"
      - "8081:8081"
    depends_on:
      - postgres
    networks:
      - kestra

  postgres:
    image: postgres:16-alpine
    container_name: kestra-postgres
    environment:
      POSTGRES_DB: kestra
      POSTGRES_USER: ${DB_USER}
      POSTGRES_PASSWORD: ${DB_PASSWORD}
    volumes:
      - postgres_data:/var/lib/postgresql/data
    networks:
      - kestra

volumes:
  postgres_data:

networks:
  kestra:
    driver: bridge
```

## .env

```bash
DB_USER=kestra
DB_PASSWORD=changeme
```

## Usage notes

- Docs: https://kestra.io/docs/
- Web UI at http://localhost:8080, API on 8081
- Define workflows in YAML with 400+ plugins
- Event triggers: schedule, webhook, file, Kafka, database
- Built-in secrets management and variable templating
- Real-time execution logs and topology visualization

## Troubleshooting

- Kestra fails to connect to PostgreSQL with 'Connection refused': Verify postgres container is healthy and DB_USER/DB_PASSWORD environment variables match between services
- Web UI shows 'Database migration failed' error: Check PostgreSQL logs for schema creation issues and ensure POSTGRES_DB=kestra matches Kestra's datasource URL
- Workflows fail with 'Plugin not found' errors: Update to specific Kestra image tags rather than 'latest' to ensure plugin compatibility, or check plugin documentation for version requirements
- High memory usage during workflow execution: Increase Docker memory limits and consider PostgreSQL connection pooling settings, monitor postgres_data volume disk usage
- API calls to port 8081 return connection timeouts: Verify both ports 8080 and 8081 are properly exposed and no firewall rules block internal container communication
- Workflow executions appear stuck in 'RUNNING' state: Check Kestra logs for task executor issues and verify PostgreSQL lock tables aren't showing long-running transactions blocking workflow state updates

