# HashiCorp Nomad

> Simple and flexible workload orchestrator for containers and non-containerized apps.

- Category: DevOps & CI/CD
- Difficulty: advanced
- Components: nomad-server, nomad-client, consul
- Tags: orchestration, nomad, hashicorp, workload, scheduler
- Source: https://docker.recipes/devops/hashicorp-nomad

## Overview

HashiCorp Nomad is a simple and flexible workload orchestrator designed to schedule and manage containers, legacy applications, virtual machines, and other workloads across on-premises and cloud environments. Unlike complex orchestration platforms, Nomad focuses on simplicity with a single binary deployment while supporting diverse workload types including Docker containers, Java applications, and system processes. It provides native integration with other HashiCorp tools and offers powerful scheduling capabilities without the operational overhead of more complex systems.

This deployment creates a complete Nomad cluster with three essential services: a Nomad server node that handles scheduling decisions and maintains cluster state, a Nomad client node that executes workloads and reports resource availability, and a Consul instance for service discovery and coordination. The server runs with bootstrap mode for single-node operation, while the client connects with privileged access to manage Docker containers and system resources. Consul provides the service mesh and key-value store capabilities that enhance Nomad's scheduling and service discovery features.

This configuration is ideal for organizations wanting to explore container orchestration beyond Kubernetes, teams managing mixed workloads that include both containerized and traditional applications, and HashiCorp stack users seeking integrated tooling. The setup provides a production-like environment for testing Nomad's unique capabilities like batch job scheduling, multi-datacenter federation, and support for non-containerized workloads while maintaining the simplicity that makes Nomad attractive for smaller teams and specific use cases.

## Features

- Multi-workload scheduling supporting Docker containers, Java applications, and raw executables
- Single binary architecture with minimal operational complexity compared to Kubernetes
- Native Consul integration for service discovery and health checking
- Batch job scheduling with automatic retry and constraint-based placement
- Blue-green deployment support with built-in rolling update strategies
- Privileged client access enabling GPU workloads and system-level task execution
- Federation capabilities for multi-datacenter workload distribution
- Built-in service discovery and load balancing through Consul integration

## Use cases

- Container orchestration for teams seeking simpler alternatives to Kubernetes
- Mixed environment management with both containerized and legacy applications
- Batch processing workflows requiring scheduled job execution and resource management
- Development and testing environments for HashiCorp stack integration
- Edge computing deployments where lightweight orchestration is preferred
- CI/CD pipeline execution with dynamic resource allocation
- Machine learning workloads requiring GPU scheduling and batch processing

## Prerequisites

- Docker Engine with privileged container support for client operations
- Minimum 1GB RAM (512MB+ recommended per Nomad instance, 256MB+ for Consul)
- Available ports 4646-4648 for Nomad and 8500 for Consul web interfaces
- Basic understanding of job specification files and HCL configuration syntax
- Knowledge of container networking concepts for service discovery setup
- Familiarity with HashiCorp tools ecosystem for optimal configuration

## docker-compose.yml

```yaml
services:
  nomad-server:
    image: hashicorp/nomad:latest
    container_name: nomad-server
    command: agent -server -bootstrap-expect=1 -bind=0.0.0.0
    cap_add:
      - SYS_ADMIN
    volumes:
      - ./nomad-server.hcl:/etc/nomad.d/nomad.hcl:ro
      - nomad-server-data:/opt/nomad/data
    ports:
      - "4646:4646"
      - "4647:4647"
      - "4648:4648"
    networks:
      - nomad-network
    restart: unless-stopped

  nomad-client:
    image: hashicorp/nomad:latest
    container_name: nomad-client
    command: agent -client -servers=nomad-server:4647
    privileged: true
    cap_add:
      - SYS_ADMIN
    volumes:
      - ./nomad-client.hcl:/etc/nomad.d/nomad.hcl:ro
      - nomad-client-data:/opt/nomad/data
      - /var/run/docker.sock:/var/run/docker.sock
    depends_on:
      - nomad-server
    networks:
      - nomad-network
    restart: unless-stopped

  consul:
    image: hashicorp/consul:latest
    container_name: nomad-consul
    command: agent -server -bootstrap-expect=1 -ui -client=0.0.0.0
    environment:
      - CONSUL_BIND_INTERFACE=eth0
    volumes:
      - consul-data:/consul/data
    ports:
      - "8500:8500"
    networks:
      - nomad-network
    restart: unless-stopped

volumes:
  nomad-server-data:
  nomad-client-data:
  consul-data:

networks:
  nomad-network:
    driver: bridge
```

## .env

```bash
# HashiCorp Nomad
# Configure nomad-server.hcl and nomad-client.hcl
```

## Usage notes

- Nomad UI at http://localhost:4646
- Consul UI at http://localhost:8500
- Deploy jobs: nomad job run job.nomad
- Supports Docker, exec, raw_exec
- Integrates with Vault and Consul

## Troubleshooting

- Nomad client fails to connect: Verify nomad-server is fully started and port 4647 is accessible within the Docker network
- Jobs fail with 'docker driver not found': Ensure nomad-client has privileged: true and access to /var/run/docker.sock
- Consul UI shows no services: Check that Nomad agents are properly configured to register with Consul and network connectivity exists
- Resource allocation errors: Verify nomad-client has sufficient CPU and memory resources available and SYS_ADMIN capability
- Bootstrap failures on restart: Remove nomad-server-data volume to re-bootstrap or configure proper cluster persistence
- Permission denied errors: Confirm container capabilities (SYS_ADMIN) are properly configured and Docker daemon is accessible

