# Concourse CI

> Container-based continuous integration and delivery.

- Category: DevOps & CI/CD
- Difficulty: advanced
- Components: concourse-web, concourse-worker, postgresql
- Tags: concourse, ci, cd, pipelines
- Source: https://docker.recipes/devops/concourse-ci

## Overview

Concourse CI is a container-native continuous integration and delivery platform that treats pipelines as first-class resources defined in YAML. Unlike traditional CI/CD systems that rely on plugins and complex configurations, Concourse embraces immutable infrastructure principles where every build runs in isolated containers, ensuring reproducible and reliable automation workflows. The platform separates compute resources (workers) from coordination logic (web interface), enabling horizontal scaling and distributed build execution.

This deployment consists of three interconnected services: concourse-db provides PostgreSQL database storage for pipeline definitions, build history, and authentication data; concourse-web serves the web interface, API, and pipeline orchestration on port 8080; and concourse-worker handles actual job execution with privileged container access for Docker-in-Docker operations. The web component communicates with workers through an internal TSA (SSH-based) connection on port 2222, while shared volumes ensure cryptographic keys are available across services.

This configuration targets development teams and DevOps engineers who need a robust, scalable CI/CD solution that can handle complex pipeline dependencies, parallel job execution, and resource management. The container-based approach makes it particularly valuable for organizations already invested in containerized infrastructure, offering native Docker support and the ability to run any toolchain within isolated build environments.

## Features

- Pipeline-as-Code with YAML definitions stored in version control alongside application code
- Resource-based pipeline modeling where inputs/outputs are explicit dependencies between jobs
- Container-native job execution with automatic Docker image pulling and cleanup
- Distributed worker architecture supporting horizontal scaling across multiple nodes
- Built-in authentication with local users and external provider integration
- Fly CLI for pipeline management, job triggering, and build log streaming from command line
- Automatic build triggering based on resource changes like Git commits or Docker image updates
- Visual pipeline interface showing real-time job status and resource flow dependencies

## Use cases

- Multi-stage application delivery pipelines with testing, building, and deployment phases
- Infrastructure-as-Code validation and deployment using Terraform, Ansible, or Kubernetes manifests
- Microservices CI/CD with parallel builds and coordinated deployment orchestration
- Security scanning workflows integrating vulnerability assessment and compliance checking
- Automated testing environments with database seeding, integration tests, and cleanup
- Release management with approval gates, rollback capabilities, and environment promotion
- Development team onboarding with standardized build environments and consistent tooling

## Prerequisites

- Minimum 2GB RAM recommended for PostgreSQL database and concurrent pipeline execution
- Docker host with privileged container support for concourse-worker build isolation
- Port 8080 available for web interface access and API communication
- Basic YAML knowledge for pipeline definition and job configuration
- Understanding of CI/CD concepts including resources, jobs, and task dependencies
- Fly CLI installation for pipeline management and build interaction from development machines

## docker-compose.yml

```yaml
services:
  concourse-db:
    image: postgres:15-alpine
    container_name: concourse-db
    restart: unless-stopped
    environment:
      POSTGRES_DB: concourse
      POSTGRES_USER: concourse
      POSTGRES_PASSWORD: ${DB_PASSWORD}
    volumes:
      - concourse_db:/var/lib/postgresql/data

  concourse-web:
    image: concourse/concourse:latest
    container_name: concourse-web
    command: web
    restart: unless-stopped
    depends_on:
      - concourse-db
    environment:
      CONCOURSE_POSTGRES_HOST: concourse-db
      CONCOURSE_POSTGRES_DATABASE: concourse
      CONCOURSE_POSTGRES_USER: concourse
      CONCOURSE_POSTGRES_PASSWORD: ${DB_PASSWORD}
      CONCOURSE_EXTERNAL_URL: http://localhost:8080
      CONCOURSE_ADD_LOCAL_USER: admin:${ADMIN_PASSWORD}
      CONCOURSE_MAIN_TEAM_LOCAL_USER: admin
      CONCOURSE_CLUSTER_NAME: dev
    ports:
      - "8080:8080"
    volumes:
      - concourse_keys:/concourse-keys

  concourse-worker:
    image: concourse/concourse:latest
    container_name: concourse-worker
    command: worker
    privileged: true
    restart: unless-stopped
    depends_on:
      - concourse-web
    environment:
      CONCOURSE_TSA_HOST: concourse-web:2222
    volumes:
      - concourse_keys:/concourse-keys

volumes:
  concourse_db:
  concourse_keys:
```

## .env

```bash
DB_PASSWORD=changeme
ADMIN_PASSWORD=changeme
```

## Usage notes

- Docs: https://concourse-ci.org/docs.html
- Access at http://localhost:8080 - login with admin/ADMIN_PASSWORD
- Install fly CLI: download from http://localhost:8080/api/v1/cli
- Login: fly -t main login -c http://localhost:8080 -u admin -p PASSWORD
- Set pipeline: fly -t main set-pipeline -p my-pipeline -c pipeline.yml
- Resources, jobs, and tasks define pipelines in YAML

## Troubleshooting

- concourse-web fails to start with database connection errors: Verify DB_PASSWORD environment variable matches between concourse-db and concourse-web services
- concourse-worker shows 'failed to register with ATC' errors: Check that concourse-web is fully started and accessible before worker attempts registration
- Pipeline jobs fail with 'no workers available' message: Ensure concourse-worker container has privileged: true and can communicate with concourse-web on port 2222
- Web interface shows 503 errors during startup: Wait for PostgreSQL database initialization to complete before concourse-web attempts schema creation
- Fly CLI login fails with certificate errors: Use -k flag to skip SSL verification when connecting to http://localhost:8080 in development environments
- Build containers fail with permission errors: Verify Docker socket access and privileged mode settings for concourse-worker container

