Buildkite Agent Stack

intermediate

Buildkite agents with Docker-in-Docker.

Overview

Buildkite Agent is a self-hosted build runner that connects to Buildkite's hosted platform to execute CI/CD pipelines. Originally developed by Buildkite as an alternative to fully cloud-hosted CI solutions, it allows organizations to run builds on their own infrastructure while leveraging Buildkite's powerful pipeline management interface. The agent polls for jobs from Buildkite's servers and executes them locally, providing complete control over the build environment while maintaining the convenience of a managed CI/CD platform. This stack combines Buildkite Agent with Docker-in-Docker (DinD) to create an isolated containerized CI environment capable of building Docker images and running container-based workloads. The Docker daemon runs in a separate privileged container, while Buildkite agents connect to it via TCP, enabling secure Docker operations without exposing the host's Docker socket. A local Docker registry completes the stack, providing immediate storage for built images during multi-stage builds or testing scenarios. Development teams and DevOps engineers who need to run Docker-based builds in Buildkite will find this configuration particularly valuable. Unlike cloud-hosted CI runners with limited customization, this self-hosted approach provides full control over the build environment, custom tooling installation, and network access while maintaining isolation between builds through containerization.

Key Features

Buildkite Agent with automatic job polling and pipeline execution
Isolated Docker-in-Docker daemon with TLS certificate authentication
Local Docker registry for immediate image storage and caching
Privileged DinD container enabling full Docker functionality including volume mounts
Agent auto-registration with configurable names and metadata
Shared certificate volume for secure agent-to-daemon communication
Horizontal scaling through additional agent containers
Build artifact persistence through dedicated Docker volumes

Common Use Cases

1Running Docker image builds and multi-stage builds in Buildkite pipelines
2Testing containerized applications with full Docker Compose support
3Creating CI environments for teams migrating from Jenkins to Buildkite
4Executing integration tests requiring Docker containers and networking
5Building and pushing images to private registries from Buildkite jobs
6Running security scans and vulnerability assessments on Docker images
7Supporting development teams needing custom build tools and environments

Prerequisites

Buildkite organization with agent registration token
Docker host with at least 4GB RAM for DinD operations
Available port 5000 for local Docker registry access
Privileged container support on the Docker host
Network connectivity to api.buildkite.com for agent communication
Understanding of Buildkite pipeline YAML configuration

For development & testing. Review security settings, change default credentials, and test thoroughly before production use. See Terms

docker-compose.yml

1services: 
2  dind: 
3    image: docker:24-dind
4    container_name: buildkite-dind
5    restart: unless-stopped
6    privileged: true
7    environment: 
8      - DOCKER_TLS_CERTDIR=/certs
9    volumes: 
10      - dind_certs:/certs
11      - dind_data:/var/lib/docker
12
13  agent-1: 
14    image: buildkite/agent:latest
15    container_name: buildkite-agent-1
16    restart: unless-stopped
17    environment: 
18      - BUILDKITE_AGENT_TOKEN=${AGENT_TOKEN}
19      - BUILDKITE_AGENT_NAME=agent-1
20      - DOCKER_HOST=tcp://dind:2376
21    volumes: 
22      - dind_certs:/certs/client:ro
23    depends_on: 
24      - dind
25
26  registry: 
27    image: registry:2
28    container_name: buildkite-registry
29    restart: unless-stopped
30    ports: 
31      - "${REGISTRY_PORT:-5000}:5000"
32    volumes: 
33      - registry_data:/var/lib/registry
34
35volumes: 
36  dind_certs: 
37  dind_data: 
38  registry_data:

.env Template

.env

1# Buildkite Agents
2AGENT_TOKEN=your_buildkite_agent_token
3REGISTRY_PORT=5000

Usage Notes

1Get agent token from Buildkite
2Agents auto-register
3Local registry at localhost:5000
4Scale by adding more agents

Individual Services(3 services)

Copy individual services to mix and match with your existing compose files.

dind

dind:
  image: docker:24-dind
  container_name: buildkite-dind
  restart: unless-stopped
  privileged: true
  environment:
    - DOCKER_TLS_CERTDIR=/certs
  volumes:
    - dind_certs:/certs
    - dind_data:/var/lib/docker

agent-1

agent-1:
  image: buildkite/agent:latest
  container_name: buildkite-agent-1
  restart: unless-stopped
  environment:
    - BUILDKITE_AGENT_TOKEN=${AGENT_TOKEN}
    - BUILDKITE_AGENT_NAME=agent-1
    - DOCKER_HOST=tcp://dind:2376
  volumes:
    - dind_certs:/certs/client:ro
  depends_on:
    - dind

registry

registry:
  image: registry:2
  container_name: buildkite-registry
  restart: unless-stopped
  ports:
    - ${REGISTRY_PORT:-5000}:5000
  volumes:
    - registry_data:/var/lib/registry

Quick Start

terminal

1# 1. Create the compose file
2cat > docker-compose.yml << 'EOF'
3services:
4  dind:
5    image: docker:24-dind
6    container_name: buildkite-dind
7    restart: unless-stopped
8    privileged: true
9    environment:
10      - DOCKER_TLS_CERTDIR=/certs
11    volumes:
12      - dind_certs:/certs
13      - dind_data:/var/lib/docker
14
15  agent-1:
16    image: buildkite/agent:latest
17    container_name: buildkite-agent-1
18    restart: unless-stopped
19    environment:
20      - BUILDKITE_AGENT_TOKEN=${AGENT_TOKEN}
21      - BUILDKITE_AGENT_NAME=agent-1
22      - DOCKER_HOST=tcp://dind:2376
23    volumes:
24      - dind_certs:/certs/client:ro
25    depends_on:
26      - dind
27
28  registry:
29    image: registry:2
30    container_name: buildkite-registry
31    restart: unless-stopped
32    ports:
33      - "${REGISTRY_PORT:-5000}:5000"
34    volumes:
35      - registry_data:/var/lib/registry
36
37volumes:
38  dind_certs:
39  dind_data:
40  registry_data:
41EOF
42
43# 2. Create the .env file
44cat > .env << 'EOF'
45# Buildkite Agents
46AGENT_TOKEN=your_buildkite_agent_token
47REGISTRY_PORT=5000
48EOF
49
50# 3. Start the services
51docker compose up -d
52
53# 4. View logs
54docker compose logs -f

One-Liner

Run this command to download and set up the recipe in one step:

terminal

1curl -fsSL https://docker.recipes/api/recipes/buildkite-agent-stack/run | bash

Troubleshooting

Agent shows 'connection refused' errors: Verify BUILDKITE_AGENT_TOKEN is valid and network allows HTTPS to api.buildkite.com
Docker commands fail with 'Cannot connect to Docker daemon': Check dind container is running and DOCKER_HOST points to tcp://dind:2376
Permission denied accessing /certs directory: Ensure dind_certs volume is properly mounted and containers have started in correct order
Registry push fails with 'server gave HTTP response to HTTPS client': Configure Docker daemon in DinD with --insecure-registry flag for local registry
Agent not appearing in Buildkite dashboard: Verify agent token permissions and check agent container logs for registration errors

Community Notes

Loading notes...

Download Recipe Kit

Get all files in a ready-to-deploy package

Includes docker-compose.yml, .env template, README, and license

Components

buildkite-agentdocker-dindregistry

Related Recipes

Ad Space

Shortcuts: C CopyF FavoriteD Download