# Determined AI

> Deep learning training platform with experiment tracking.

- Category: AI & Machine Learning
- Difficulty: advanced
- Components: determined, postgres
- Tags: determined, training, deep-learning, experiments
- Source: https://docker.recipes/ai-ml/determined-ai

## Overview

Determined AI is an open-source machine learning training platform designed to simplify and scale deep learning workflows. Built by former Google researchers, it addresses the complexities of distributed training, experiment management, and hyperparameter optimization that plague modern ML teams. The platform automatically handles checkpointing, fault tolerance, and resource allocation while providing sophisticated experiment tracking capabilities that rival commercial solutions like Weights & Biases or Neptune.

This stack combines Determined's master-agent architecture with PostgreSQL as the metadata store. The determined-master service orchestrates experiments, manages the web interface, and stores all experiment metadata, metrics, and configurations in PostgreSQL. The determined-agent handles actual workload execution, automatically pulling training code and managing containerized experiments. PostgreSQL's ACID compliance and JSON support make it ideal for storing complex experiment configurations, hyperparameter spaces, and time-series metrics data that Determined generates.

This configuration targets ML engineering teams and research organizations running serious deep learning workloads who need reproducible experiments and efficient resource utilization. Unlike notebook-based workflows or simple training scripts, Determined enforces best practices around experiment versioning, data loading, and model checkpointing. The PostgreSQL backend ensures experiment history survives system restarts and provides robust querying capabilities for analyzing training runs across multiple projects.

## Features

- Adaptive hyperparameter search algorithms (ASHA, Population Based Training) that intelligently terminate poor-performing trials early
- Automatic distributed training with built-in support for data parallelism and model parallelism across multiple GPUs
- Fault-tolerant training with automatic checkpointing and resume capabilities when nodes fail
- Fair-share cluster scheduling with preemption support for multi-user environments
- Web-based experiment comparison interface with real-time metrics visualization and hyperparameter analysis
- PostgreSQL-backed experiment metadata storage with full JSON support for complex configuration tracking
- Docker-based experiment isolation ensuring reproducible training environments
- CLI-driven workflow supporting both interactive development and CI/CD integration

## Use cases

- Computer vision teams training large models (ResNet, EfficientNet, Vision Transformers) requiring distributed GPU training
- NLP research groups running transformer fine-tuning experiments with extensive hyperparameter sweeps
- Autonomous vehicle companies managing hundreds of simultaneous model training jobs across GPU clusters
- Pharmaceutical research using deep learning for drug discovery with complex molecular property prediction models
- Financial institutions training fraud detection models requiring rigorous experiment tracking for regulatory compliance
- Academic research labs needing shared GPU resources with fair scheduling among multiple PhD students and projects
- MLOps teams transitioning from ad-hoc training scripts to production-grade experiment management platforms

## Prerequisites

- Minimum 4GB RAM for PostgreSQL and Determined master services combined
- Docker daemon with access to GPU runtime (nvidia-docker2) if training GPU-accelerated models
- Understanding of machine learning training loops and familiarity with PyTorch or TensorFlow
- Network access to pull training data and Docker images during experiment execution
- Basic knowledge of YAML configuration files for defining Determined experiment specifications
- Port 8080 available for the Determined web interface and API access

## docker-compose.yml

```yaml
services:
  determined-master:
    image: determinedai/determined-master:latest
    container_name: determined-master
    restart: unless-stopped
    environment:
      DET_DB_HOST: postgres
      DET_DB_NAME: determined
      DET_DB_USER: determined
      DET_DB_PASSWORD: determined
    ports:
      - "8080:8080"
    depends_on:
      - postgres
    networks:
      - determined

  determined-agent:
    image: determinedai/determined-agent:latest
    container_name: determined-agent
    restart: unless-stopped
    environment:
      DET_MASTER_HOST: determined-master
      DET_MASTER_PORT: 8080
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    depends_on:
      - determined-master
    networks:
      - determined

  postgres:
    image: postgres:16-alpine
    container_name: determined-postgres
    environment:
      POSTGRES_DB: determined
      POSTGRES_USER: determined
      POSTGRES_PASSWORD: determined
    volumes:
      - postgres_data:/var/lib/postgresql/data
    networks:
      - determined

volumes:
  postgres_data:

networks:
  determined:
    driver: bridge
```

## .env

```bash
# Configure via web UI
```

## Usage notes

- Docs: https://docs.determined.ai/
- Web UI at http://localhost:8080 - default login: admin (no password)
- CLI: pip install determined, then det experiment create config.yaml .
- Distributed training with automatic checkpointing and fault tolerance
- Hyperparameter search: grid, random, adaptive (ASHA, PBT)
- GPU cluster management with fair-share scheduling

## Troubleshooting

- Agent shows 'failed to connect to master' errors: Verify determined-master container is running and port 8080 is accessible from agent container
- Experiments stuck in 'QUEUED' state indefinitely: Check agent logs for Docker socket permissions and ensure /var/run/docker.sock is properly mounted
- PostgreSQL connection failures during master startup: Wait for postgres container to fully initialize before master starts, or add healthcheck dependencies
- Web UI shows 'Internal Server Error' on experiment pages: Check postgres container disk space and connection limits in PostgreSQL configuration
- Training experiments fail with 'image pull' errors: Ensure determined-agent has access to pull experiment Docker images from configured registries
- Hyperparameter search creates too many concurrent trials: Adjust 'max_concurrent_trials' in experiment configuration to match available resources

