docker.recipes

Concourse CI

advanced

Container-based continuous integration and delivery.

Overview

Concourse CI is a container-native continuous integration and delivery platform that treats pipelines as first-class resources defined in YAML. Unlike traditional CI/CD systems that rely on plugins and complex configurations, Concourse embraces immutable infrastructure principles where every build runs in isolated containers, ensuring reproducible and reliable automation workflows. The platform separates compute resources (workers) from coordination logic (web interface), enabling horizontal scaling and distributed build execution. This deployment consists of three interconnected services: concourse-db provides PostgreSQL database storage for pipeline definitions, build history, and authentication data; concourse-web serves the web interface, API, and pipeline orchestration on port 8080; and concourse-worker handles actual job execution with privileged container access for Docker-in-Docker operations. The web component communicates with workers through an internal TSA (SSH-based) connection on port 2222, while shared volumes ensure cryptographic keys are available across services. This configuration targets development teams and DevOps engineers who need a robust, scalable CI/CD solution that can handle complex pipeline dependencies, parallel job execution, and resource management. The container-based approach makes it particularly valuable for organizations already invested in containerized infrastructure, offering native Docker support and the ability to run any toolchain within isolated build environments.

Key Features

  • Pipeline-as-Code with YAML definitions stored in version control alongside application code
  • Resource-based pipeline modeling where inputs/outputs are explicit dependencies between jobs
  • Container-native job execution with automatic Docker image pulling and cleanup
  • Distributed worker architecture supporting horizontal scaling across multiple nodes
  • Built-in authentication with local users and external provider integration
  • Fly CLI for pipeline management, job triggering, and build log streaming from command line
  • Automatic build triggering based on resource changes like Git commits or Docker image updates
  • Visual pipeline interface showing real-time job status and resource flow dependencies

Common Use Cases

  • 1Multi-stage application delivery pipelines with testing, building, and deployment phases
  • 2Infrastructure-as-Code validation and deployment using Terraform, Ansible, or Kubernetes manifests
  • 3Microservices CI/CD with parallel builds and coordinated deployment orchestration
  • 4Security scanning workflows integrating vulnerability assessment and compliance checking
  • 5Automated testing environments with database seeding, integration tests, and cleanup
  • 6Release management with approval gates, rollback capabilities, and environment promotion
  • 7Development team onboarding with standardized build environments and consistent tooling

Prerequisites

  • Minimum 2GB RAM recommended for PostgreSQL database and concurrent pipeline execution
  • Docker host with privileged container support for concourse-worker build isolation
  • Port 8080 available for web interface access and API communication
  • Basic YAML knowledge for pipeline definition and job configuration
  • Understanding of CI/CD concepts including resources, jobs, and task dependencies
  • Fly CLI installation for pipeline management and build interaction from development machines

For development & testing. Review security settings, change default credentials, and test thoroughly before production use. See Terms

docker-compose.yml

docker-compose.yml
1services: 
2  concourse-db: 
3    image: postgres:15-alpine
4    container_name: concourse-db
5    restart: unless-stopped
6    environment: 
7      POSTGRES_DB: concourse
8      POSTGRES_USER: concourse
9      POSTGRES_PASSWORD: ${DB_PASSWORD}
10    volumes: 
11      - concourse_db:/var/lib/postgresql/data
12
13  concourse-web: 
14    image: concourse/concourse:latest
15    container_name: concourse-web
16    command: web
17    restart: unless-stopped
18    depends_on: 
19      - concourse-db
20    environment: 
21      CONCOURSE_POSTGRES_HOST: concourse-db
22      CONCOURSE_POSTGRES_DATABASE: concourse
23      CONCOURSE_POSTGRES_USER: concourse
24      CONCOURSE_POSTGRES_PASSWORD: ${DB_PASSWORD}
25      CONCOURSE_EXTERNAL_URL: http://localhost:8080
26      CONCOURSE_ADD_LOCAL_USER: admin:${ADMIN_PASSWORD}
27      CONCOURSE_MAIN_TEAM_LOCAL_USER: admin
28      CONCOURSE_CLUSTER_NAME: dev
29    ports: 
30      - "8080:8080"
31    volumes: 
32      - concourse_keys:/concourse-keys
33
34  concourse-worker: 
35    image: concourse/concourse:latest
36    container_name: concourse-worker
37    command: worker
38    privileged: true
39    restart: unless-stopped
40    depends_on: 
41      - concourse-web
42    environment: 
43      CONCOURSE_TSA_HOST: concourse-web:2222
44    volumes: 
45      - concourse_keys:/concourse-keys
46
47volumes: 
48  concourse_db: 
49  concourse_keys:

.env Template

.env
1DB_PASSWORD=changeme
2ADMIN_PASSWORD=changeme

Usage Notes

  1. 1Docs: https://concourse-ci.org/docs.html
  2. 2Access at http://localhost:8080 - login with admin/ADMIN_PASSWORD
  3. 3Install fly CLI: download from http://localhost:8080/api/v1/cli
  4. 4Login: fly -t main login -c http://localhost:8080 -u admin -p PASSWORD
  5. 5Set pipeline: fly -t main set-pipeline -p my-pipeline -c pipeline.yml
  6. 6Resources, jobs, and tasks define pipelines in YAML

Individual Services(3 services)

Copy individual services to mix and match with your existing compose files.

concourse-db
concourse-db:
  image: postgres:15-alpine
  container_name: concourse-db
  restart: unless-stopped
  environment:
    POSTGRES_DB: concourse
    POSTGRES_USER: concourse
    POSTGRES_PASSWORD: ${DB_PASSWORD}
  volumes:
    - concourse_db:/var/lib/postgresql/data
concourse-web
concourse-web:
  image: concourse/concourse:latest
  container_name: concourse-web
  command: web
  restart: unless-stopped
  depends_on:
    - concourse-db
  environment:
    CONCOURSE_POSTGRES_HOST: concourse-db
    CONCOURSE_POSTGRES_DATABASE: concourse
    CONCOURSE_POSTGRES_USER: concourse
    CONCOURSE_POSTGRES_PASSWORD: ${DB_PASSWORD}
    CONCOURSE_EXTERNAL_URL: http://localhost:8080
    CONCOURSE_ADD_LOCAL_USER: admin:${ADMIN_PASSWORD}
    CONCOURSE_MAIN_TEAM_LOCAL_USER: admin
    CONCOURSE_CLUSTER_NAME: dev
  ports:
    - "8080:8080"
  volumes:
    - concourse_keys:/concourse-keys
concourse-worker
concourse-worker:
  image: concourse/concourse:latest
  container_name: concourse-worker
  command: worker
  privileged: true
  restart: unless-stopped
  depends_on:
    - concourse-web
  environment:
    CONCOURSE_TSA_HOST: concourse-web:2222
  volumes:
    - concourse_keys:/concourse-keys

Quick Start

terminal
1# 1. Create the compose file
2cat > docker-compose.yml << 'EOF'
3services:
4  concourse-db:
5    image: postgres:15-alpine
6    container_name: concourse-db
7    restart: unless-stopped
8    environment:
9      POSTGRES_DB: concourse
10      POSTGRES_USER: concourse
11      POSTGRES_PASSWORD: ${DB_PASSWORD}
12    volumes:
13      - concourse_db:/var/lib/postgresql/data
14
15  concourse-web:
16    image: concourse/concourse:latest
17    container_name: concourse-web
18    command: web
19    restart: unless-stopped
20    depends_on:
21      - concourse-db
22    environment:
23      CONCOURSE_POSTGRES_HOST: concourse-db
24      CONCOURSE_POSTGRES_DATABASE: concourse
25      CONCOURSE_POSTGRES_USER: concourse
26      CONCOURSE_POSTGRES_PASSWORD: ${DB_PASSWORD}
27      CONCOURSE_EXTERNAL_URL: http://localhost:8080
28      CONCOURSE_ADD_LOCAL_USER: admin:${ADMIN_PASSWORD}
29      CONCOURSE_MAIN_TEAM_LOCAL_USER: admin
30      CONCOURSE_CLUSTER_NAME: dev
31    ports:
32      - "8080:8080"
33    volumes:
34      - concourse_keys:/concourse-keys
35
36  concourse-worker:
37    image: concourse/concourse:latest
38    container_name: concourse-worker
39    command: worker
40    privileged: true
41    restart: unless-stopped
42    depends_on:
43      - concourse-web
44    environment:
45      CONCOURSE_TSA_HOST: concourse-web:2222
46    volumes:
47      - concourse_keys:/concourse-keys
48
49volumes:
50  concourse_db:
51  concourse_keys:
52EOF
53
54# 2. Create the .env file
55cat > .env << 'EOF'
56DB_PASSWORD=changeme
57ADMIN_PASSWORD=changeme
58EOF
59
60# 3. Start the services
61docker compose up -d
62
63# 4. View logs
64docker compose logs -f

One-Liner

Run this command to download and set up the recipe in one step:

terminal
1curl -fsSL https://docker.recipes/api/recipes/concourse-ci/run | bash

Troubleshooting

  • concourse-web fails to start with database connection errors: Verify DB_PASSWORD environment variable matches between concourse-db and concourse-web services
  • concourse-worker shows 'failed to register with ATC' errors: Check that concourse-web is fully started and accessible before worker attempts registration
  • Pipeline jobs fail with 'no workers available' message: Ensure concourse-worker container has privileged: true and can communicate with concourse-web on port 2222
  • Web interface shows 503 errors during startup: Wait for PostgreSQL database initialization to complete before concourse-web attempts schema creation
  • Fly CLI login fails with certificate errors: Use -k flag to skip SSL verification when connecting to http://localhost:8080 in development environments
  • Build containers fail with permission errors: Verify Docker socket access and privileged mode settings for concourse-worker container

Community Notes

Loading...
Loading notes...

Download Recipe Kit

Get all files in a ready-to-deploy package

Includes docker-compose.yml, .env template, README, and license

Components

concourse-webconcourse-workerpostgresql

Tags

#concourse#ci#cd#pipelines

Category

DevOps & CI/CD

Related Recipes

Ad Space