# K3s Development Cluster

> Lightweight Kubernetes with k3s, Traefik ingress, and local registry for development.

- Category: Development Tools
- Difficulty: advanced
- Components: k3s, registry, traefik
- Tags: k3s, kubernetes, development, traefik, registry
- Source: https://docker.recipes/development/k3s-development-cluster

## Overview

K3s is Rancher's production-ready, CNCF-certified Kubernetes distribution designed for resource-constrained environments and edge computing. Built from the ground up to be lightweight, K3s removes optional and legacy features from standard Kubernetes, packages everything into a single binary under 100MB, and replaces etcd with SQLite by default. This makes it perfect for development environments, IoT deployments, ARM devices, and situations where full Kubernetes would be overkill.

This deployment creates a complete K3s development cluster using three specialized containers: a k3s-server running the full Kubernetes control plane and worker node in a single process, a Docker Registry v2 for storing and distributing container images locally, and a web-based registry UI for browsing and managing those images. The k3s-server includes Traefik as the default ingress controller, providing automatic load balancing and SSL termination capabilities built into the cluster. The configuration automatically sets up kubeconfig file generation and configures the cluster to use the local registry for pulling images.

This stack is ideal for developers who need a full-featured Kubernetes environment for testing applications, CI/CD pipelines that require isolated Kubernetes clusters, and teams wanting to prototype cloud-native applications without the complexity and resource overhead of full Kubernetes distributions like kubeadm or managed services. The local registry integration eliminates the need to push images to external registries during development, significantly speeding up the container build-test-deploy cycle.

## Features

- Complete Kubernetes API compatibility in a single lightweight container under 100MB
- Built-in Traefik ingress controller with automatic service discovery and load balancing
- Local Docker Registry v2 with pre-configured integration for seamless image management
- Web-based registry UI for browsing, searching, and managing container images and tags
- Automatic kubeconfig generation with external access via port 6443
- SQLite-backed storage eliminating etcd complexity and resource requirements
- Privileged container mode enabling full Kubernetes networking and storage features
- Pre-configured registries.yaml for automatic local registry authentication

## Use cases

- Local Kubernetes development environment for testing microservices and cloud-native applications
- CI/CD pipeline testing requiring isolated, disposable Kubernetes clusters
- Kubernetes training and certification labs with realistic cluster behavior
- Edge computing prototypes requiring lightweight Kubernetes at remote locations
- Container image development workflow with local registry for rapid iteration
- Helm chart development and testing without external cluster dependencies
- GitOps and ArgoCD testing in controlled local environment

## Prerequisites

- Docker Engine with privileged container support enabled
- Minimum 2GB RAM available (4GB+ recommended for running workloads)
- Ports 6443, 80, 443, 5000, and 8080 available on host system
- Basic Kubernetes and kubectl knowledge for cluster interaction
- Understanding of container registries and image tagging conventions
- K3S_TOKEN environment variable set for cluster security

## docker-compose.yml

```yaml
services:
  k3s-server:
    image: rancher/k3s:latest
    command: server
    tmpfs:
      - /run
      - /var/run
    ulimits:
      nproc: 65535
      nofile:
        soft: 65535
        hard: 65535
    privileged: true
    environment:
      K3S_TOKEN: ${K3S_TOKEN}
      K3S_KUBECONFIG_OUTPUT: /output/kubeconfig.yaml
      K3S_KUBECONFIG_MODE: 666
    volumes:
      - k3s_server:/var/lib/rancher/k3s
      - ./output:/output
      - ./registries.yaml:/etc/rancher/k3s/registries.yaml:ro
    ports:
      - "6443:6443"
      - "80:80"
      - "443:443"
    networks:
      - k3s-net
    restart: unless-stopped

  registry:
    image: registry:2
    ports:
      - "5000:5000"
    volumes:
      - registry_data:/var/lib/registry
    networks:
      - k3s-net
    restart: unless-stopped

  registry-ui:
    image: joxit/docker-registry-ui:latest
    ports:
      - "8080:80"
    environment:
      REGISTRY_TITLE: Local Registry
      REGISTRY_URL: http://registry:5000
      SINGLE_REGISTRY: "true"
    depends_on:
      - registry
    networks:
      - k3s-net
    restart: unless-stopped

volumes:
  k3s_server:
  registry_data:

networks:
  k3s-net:
    driver: bridge
```

## .env

```bash
# K3s Token
K3S_TOKEN=your_secret_token_here

# Generate with: openssl rand -hex 32
```

## Usage notes

- Kubeconfig at ./output/kubeconfig.yaml
- Local registry at localhost:5000
- Registry UI at http://localhost:8080
- Tag images: localhost:5000/myapp:latest

## Troubleshooting

- k3s-server fails to start with 'permission denied': Ensure Docker daemon has privileged container support and the container is running in privileged mode
- kubectl connection refused on port 6443: Check that ./output directory exists with proper permissions and kubeconfig.yaml is generated
- Images fail to pull from localhost:5000: Verify registries.yaml configuration is mounted correctly and registry service is running
- registry-ui shows 'registry not found' error: Confirm registry container is healthy and accessible on the k3s-net network
- Pods stuck in Pending state: Check if k3s-server has sufficient resources and tmpfs mounts are properly configured
- Traefik ingress not routing traffic: Ensure ports 80/443 are not blocked by host firewall and IngressRoute resources are properly configured

