# Concourse CI Pipeline

> Concourse CI for container-native CI.

- Category: DevOps & CI/CD
- Difficulty: intermediate
- Components: concourse, postgresql
- Tags: concourse, ci, pipelines, containers
- Source: https://docker.recipes/devops/concourse-ci-stack

## Overview

Concourse CI is a container-native continuous integration and continuous delivery (CI/CD) platform that treats pipelines as first-class citizens. Originally developed by Pivotal (now part of VMware Tanzu), Concourse emphasizes simplicity, reproducibility, and transparency in build processes. Unlike traditional CI systems that rely on persistent build agents, Concourse executes every task in ephemeral containers, ensuring clean, reproducible builds without accumulated state or configuration drift. This deployment creates a fully functional Concourse CI environment using two services: a PostgreSQL 15 database for metadata storage and the Concourse application server running in quickstart mode. The PostgreSQL instance stores pipeline configurations, build history, resource metadata, and team permissions, while the Concourse container handles both web UI and worker responsibilities in a single process. This configuration is ideal for development teams, small organizations, or proof-of-concept deployments that need a lightweight yet powerful CI/CD solution. The container-native approach makes this stack particularly valuable for teams already invested in containerized workflows, as pipelines can seamlessly integrate with existing Docker registries, Kubernetes clusters, and container-based deployment strategies.

## Features

- Container-native pipeline execution with ephemeral task containers ensuring clean, reproducible builds
- Pipeline-as-code with YAML configuration stored in version control for full auditability
- Resource-based pipeline modeling with built-in support for Git, Docker images, S3, and time triggers
- Real-time pipeline visualization with interactive web interface showing resource flow and build status
- Multi-team support with role-based access control and team-specific pipeline isolation
- Built-in secret management with credential interpolation from various backends
- Fly CLI integration for pipeline management, job triggering, and direct task execution
- Automatic resource checking and version tracking across pipeline dependencies

## Use cases

- Containerized application CI/CD with Docker image building, testing, and registry publishing
- Microservices deployment pipelines with service-specific build stages and integration testing
- Infrastructure-as-code validation and deployment using Terraform, Ansible, or Kubernetes manifests
- Multi-environment promotion workflows with automated testing gates between development, staging, and production
- Open source project automation including pull request validation, release packaging, and documentation generation
- Development team onboarding and CI/CD learning environment with visual pipeline feedback
- Compliance-focused deployments requiring full build traceability and approval workflows

## Prerequisites

- Docker Engine 20.10+ with privileged container support for Concourse worker functionality
- Minimum 2GB RAM available (1GB for Concourse, 512MB for PostgreSQL, plus container overhead)
- Port 8080 available for Concourse web interface access
- Fly CLI installed locally for pipeline management and task execution
- Basic understanding of YAML syntax for pipeline configuration
- Familiarity with container registries and Git repositories for resource integration

## docker-compose.yml

```yaml
services:
  concourse-db:
    image: postgres:15-alpine
    container_name: concourse-db
    restart: unless-stopped
    environment:
      - POSTGRES_USER=concourse
      - POSTGRES_PASSWORD=${DB_PASSWORD}
      - POSTGRES_DB=concourse
    volumes:
      - concourse_db_data:/var/lib/postgresql/data

  concourse:
    image: concourse/concourse:latest
    container_name: concourse
    restart: unless-stopped
    privileged: true
    ports:
      - "${CONCOURSE_PORT:-8080}:8080"
    environment:
      - CONCOURSE_POSTGRES_HOST=concourse-db
      - CONCOURSE_POSTGRES_USER=concourse
      - CONCOURSE_POSTGRES_PASSWORD=${DB_PASSWORD}
      - CONCOURSE_EXTERNAL_URL=http://localhost:${CONCOURSE_PORT:-8080}
      - CONCOURSE_ADD_LOCAL_USER=admin:${ADMIN_PASSWORD}
      - CONCOURSE_MAIN_TEAM_LOCAL_USER=admin
    command: quickstart
    depends_on:
      - concourse-db

volumes:
  concourse_db_data:
```

## .env

```bash
# Concourse CI
CONCOURSE_PORT=8080
DB_PASSWORD=concourse_password
ADMIN_PASSWORD=admin_password
```

## Usage notes

- Concourse at http://localhost:8080
- Login with admin credentials
- Install fly CLI for pipelines
- Define pipelines in YAML

## Troubleshooting

- Concourse container fails to start with permission errors: Ensure Docker daemon supports privileged containers and the host allows container privilege escalation
- Pipeline tasks fail with 'no workers available': Restart the concourse container as quickstart mode combines web and worker in single process
- Fly CLI connection refused: Verify CONCOURSE_EXTERNAL_URL matches your actual access URL and port mapping is correct
- Resource check failures with Git repositories: Ensure Concourse container has network access to external Git servers and registries
- Database connection errors on startup: Check that POSTGRES_PASSWORD and DB_PASSWORD environment variables match exactly
- Pipeline upload fails with authentication errors: Login with fly CLI using admin credentials defined in ADMIN_PASSWORD environment variable

