# Dagster Data Platform

> Dagster data orchestration platform with PostgreSQL and web UI.

- Category: Database Stacks
- Difficulty: intermediate
- Components: dagster, dagster-webserver, dagster-daemon, postgresql
- Tags: dagster, orchestration, data-pipeline, etl, python
- Source: https://docker.recipes/databases/dagster-data-platform

## Overview

Dagster is a modern data orchestration platform developed by Elementl that transforms how organizations build, test, and monitor data pipelines. Unlike traditional workflow engines that focus solely on task execution, Dagster provides a comprehensive framework for data asset management with strong typing, automatic lineage tracking, and powerful observability features. It emerged in 2019 as a response to the limitations of existing tools like Airflow, offering a code-first approach that treats data pipelines as software engineering artifacts with proper testing, versioning, and development practices.

This stack combines Dagster's multi-component architecture with PostgreSQL as the persistent storage backend. The dagster-webserver provides the intuitive Dagit web interface for pipeline visualization and monitoring, while dagster-daemon handles background processes like scheduling, sensor evaluation, and run queue management. PostgreSQL stores all metadata including asset definitions, run history, event logs, and schedule states, providing ACID compliance and robust querying capabilities that Dagster leverages for complex lineage analysis and performance optimization.

Data engineers and analytics teams building modern data platforms will find this combination particularly valuable for replacing fragmented toolchains. Organizations migrating from legacy ETL tools, teams implementing DataOps practices, and companies requiring strong data governance and observability should consider this stack. The combination excels in environments where data quality, testing, and collaboration between data teams are critical, making it ideal for financial services, healthcare, and technology companies with complex data transformation requirements.

## Features

- Asset-based data modeling with automatic dependency resolution and lineage tracking across complex data transformations
- Software-defined assets (SDAs) that treat data tables, ML models, and reports as first-class versioned objects
- Built-in data quality testing framework with configurable asset checks and automated anomaly detection
- Multi-tenancy support through workspaces allowing isolated development and production pipeline environments
- Native integration with dbt, Spark, Pandas, and cloud data platforms through Dagster's extensive library ecosystem
- Backfill capabilities for historical data processing with intelligent partitioning and incremental updates
- Real-time monitoring with Slack, email, and webhook notifications for pipeline failures and SLA violations
- PostgreSQL-backed metadata store providing fast queries for asset lineage, run history, and performance analytics

## Use cases

- Modern data warehouse orchestration replacing legacy ETL tools like Informatica or DataStage with code-first pipelines
- MLOps workflows for training, validating, and deploying machine learning models with feature store integration
- Data lake processing for ingesting and transforming large volumes of raw data from multiple sources
- Business intelligence pipeline automation connecting data sources to reporting tools like Tableau or Looker
- Real-time data processing for streaming analytics and event-driven data transformations
- Data migration projects requiring complex validation, transformation, and quality assurance processes
- Multi-environment data platform management for development, staging, and production data pipeline deployment

## Prerequisites

- Minimum 4GB RAM recommended for running Dagster webserver, daemon, and PostgreSQL with moderate workloads
- Docker Engine 20.10+ and Docker Compose V2 for proper container networking and volume management
- Port 3000 available for Dagster web interface access and port 5432 free for PostgreSQL connections
- Python development knowledge for writing Dagster pipelines, assets, and custom resource configurations
- Understanding of data pipeline concepts including ETL/ELT patterns, data lineage, and workflow orchestration
- Basic SQL knowledge for database troubleshooting and custom metadata queries against PostgreSQL backend

## docker-compose.yml

```yaml
services:
  postgres:
    image: postgres:16-alpine
    container_name: dagster-postgres
    restart: unless-stopped
    environment:
      POSTGRES_USER: ${POSTGRES_USER:-dagster}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-dagster}
      POSTGRES_DB: ${POSTGRES_DB:-dagster}
    volumes:
      - postgres_data:/var/lib/postgresql/data
    networks:
      - dagster-network

  dagster-webserver:
    image: dagster/dagster-k8s:latest
    container_name: dagster-webserver
    restart: unless-stopped
    entrypoint:
      - dagster-webserver
      - -h
      - "0.0.0.0"
      - -p
      - "3000"
    ports:
      - "${DAGSTER_PORT:-3000}:3000"
    environment:
      DAGSTER_POSTGRES_USER: ${POSTGRES_USER:-dagster}
      DAGSTER_POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-dagster}
      DAGSTER_POSTGRES_DB: ${POSTGRES_DB:-dagster}
      DAGSTER_POSTGRES_HOST: postgres
    depends_on:
      - postgres
    networks:
      - dagster-network

  dagster-daemon:
    image: dagster/dagster-k8s:latest
    container_name: dagster-daemon
    restart: unless-stopped
    entrypoint:
      - dagster-daemon
      - run
    environment:
      DAGSTER_POSTGRES_USER: ${POSTGRES_USER:-dagster}
      DAGSTER_POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-dagster}
      DAGSTER_POSTGRES_DB: ${POSTGRES_DB:-dagster}
      DAGSTER_POSTGRES_HOST: postgres
    depends_on:
      - postgres
    networks:
      - dagster-network

volumes:
  postgres_data:

networks:
  dagster-network:
    driver: bridge
```

## .env

```bash
# Dagster
DAGSTER_PORT=3000
POSTGRES_USER=dagster
POSTGRES_PASSWORD=dagster
POSTGRES_DB=dagster
```

## Usage notes

- Dagster UI at http://localhost:3000
- Define pipelines in Python
- Daemon handles schedules and sensors
- Great for modern data pipelines

## Troubleshooting

- Dagster webserver fails to start with database connection errors: Verify PostgreSQL container is fully initialized before webserver startup and check DAGSTER_POSTGRES_* environment variables match database credentials
- Pipeline runs stuck in QUEUED status indefinitely: Restart dagster-daemon container as the daemon may have stopped processing the run queue due to resource constraints or configuration issues
- Asset materialization failures with 'ImportError' messages: Mount your Dagster project code directory into both webserver and daemon containers using additional volume mounts
- High memory usage causing container crashes: Increase Docker memory limits and consider adding PostgreSQL connection pooling configuration to prevent connection exhaustion
- Dagit UI shows 'No repositories found' error: Ensure your dagster.yaml workspace configuration file is properly mounted and contains valid repository definitions
- Slow query performance in PostgreSQL affecting pipeline execution: Add database indexes on frequently queried event log tables and consider PostgreSQL performance tuning parameters

