docker.recipes

Triton Inference Server

advanced

NVIDIA Triton for scalable ML model serving.

Overview

NVIDIA Triton Inference Server is an open-source inference serving software that enables teams to deploy trained AI models from any major machine learning framework at scale. Originally developed by NVIDIA as TensorRT Inference Server and later renamed Triton, it provides a standardized inference platform that can serve models from TensorFlow, PyTorch, ONNX Runtime, Python, and custom backends with optimized performance for both CPU and GPU workloads. The server handles model lifecycle management, dynamic batching, and concurrent model execution while exposing both HTTP/REST and gRPC APIs for client applications. This Triton deployment creates a production-grade inference service that automatically discovers and loads models from a structured repository directory. The configuration exposes three distinct service endpoints: port 8000 for HTTP inference requests, port 8001 for high-performance gRPC communication, and port 8002 for Prometheus metrics collection. The server continuously monitors the model repository for changes, enabling hot-swapping of model versions without service interruption, while supporting advanced features like ensemble pipelines and multi-model serving from a single instance. Data scientists and MLOps teams building production inference pipelines will find this stack particularly valuable for standardizing model deployment across different frameworks and hardware configurations. Organizations serving multiple models or requiring high-throughput inference with dynamic batching capabilities can leverage Triton's optimization features to maximize resource utilization. The unified API surface and comprehensive monitoring make it ideal for teams transitioning from research prototypes to production services, especially when deploying models that need to scale from thousands to millions of requests per day.

Key Features

  • Multi-framework model serving supporting TensorFlow SavedModel, TorchScript, ONNX, TensorRT, Python, and custom backend formats
  • Dynamic request batching that automatically combines individual inference requests to maximize GPU utilization and throughput
  • Model versioning and hot-swapping allowing updates to models without service downtime or connection interruption
  • Concurrent model execution enabling multiple different models to run simultaneously on the same server instance
  • Model ensemble pipelines for chaining multiple models together with pre- and post-processing steps defined in configuration
  • Built-in A/B testing support through model routing and traffic splitting capabilities for canary deployments
  • Advanced memory management with model loading/unloading based on demand and configurable memory pools
  • Comprehensive metrics export including request latency, throughput, GPU utilization, and model-specific performance data

Common Use Cases

  • 1Real-time recommendation systems serving personalized content with sub-100ms latency requirements for e-commerce platforms
  • 2Computer vision pipelines processing high-volume image or video streams for autonomous vehicles or surveillance systems
  • 3Natural language processing APIs serving multiple transformer models for chatbots, translation, or content analysis
  • 4Financial fraud detection systems requiring ensemble models with strict latency and throughput SLAs
  • 5Healthcare diagnostic tools serving medical imaging models with regulatory compliance and audit trail requirements
  • 6Edge AI deployments where multiple optimized models need centralized serving across distributed locations
  • 7Research environments requiring rapid experimentation with different model versions and A/B testing capabilities

Prerequisites

  • NVIDIA Container Toolkit installed for GPU acceleration support and CUDA runtime access
  • Minimum 8GB RAM for basic model serving, 16GB+ recommended for multiple large models or high concurrency
  • Properly structured model repository with models organized in {model_name}/{version}/model.{ext} directory format
  • Understanding of model configuration files and Triton's config.pbtxt format for advanced model settings
  • Network ports 8000, 8001, and 8002 available and not conflicting with existing services
  • Basic familiarity with inference model formats (ONNX, TensorRT, SavedModel) and their performance characteristics

For development & testing. Review security settings, change default credentials, and test thoroughly before production use. See Terms

docker-compose.yml

docker-compose.yml
1services: 
2  triton: 
3    image: nvcr.io/nvidia/tritonserver:latest-py3
4    container_name: triton
5    restart: unless-stopped
6    ports: 
7      - "${HTTP_PORT:-8000}:8000"
8      - "${GRPC_PORT:-8001}:8001"
9      - "${METRICS_PORT:-8002}:8002"
10    volumes: 
11      - ./model_repository:/models
12    command: tritonserver --model-repository=/models
13    networks: 
14      - triton-network
15
16networks: 
17  triton-network: 
18    driver: bridge

.env Template

.env
1# Triton Inference Server
2HTTP_PORT=8000
3GRPC_PORT=8001
4METRICS_PORT=8002

Usage Notes

  1. 1Docs: https://docs.nvidia.com/deeplearning/triton-inference-server/
  2. 2HTTP API at http://localhost:8000/v2/models/{model}/infer
  3. 3gRPC at localhost:8001 for high-throughput production use
  4. 4Prometheus metrics at http://localhost:8002/metrics
  5. 5Model repo structure: model_repository/{model}/{version}/model.{format}
  6. 6Supports TensorRT, ONNX, PyTorch, TensorFlow, Python backends

Quick Start

terminal
1# 1. Create the compose file
2cat > docker-compose.yml << 'EOF'
3services:
4  triton:
5    image: nvcr.io/nvidia/tritonserver:latest-py3
6    container_name: triton
7    restart: unless-stopped
8    ports:
9      - "${HTTP_PORT:-8000}:8000"
10      - "${GRPC_PORT:-8001}:8001"
11      - "${METRICS_PORT:-8002}:8002"
12    volumes:
13      - ./model_repository:/models
14    command: tritonserver --model-repository=/models
15    networks:
16      - triton-network
17
18networks:
19  triton-network:
20    driver: bridge
21EOF
22
23# 2. Create the .env file
24cat > .env << 'EOF'
25# Triton Inference Server
26HTTP_PORT=8000
27GRPC_PORT=8001
28METRICS_PORT=8002
29EOF
30
31# 3. Start the services
32docker compose up -d
33
34# 4. View logs
35docker compose logs -f

One-Liner

Run this command to download and set up the recipe in one step:

terminal
1curl -fsSL https://docker.recipes/api/recipes/triton-inference-server/run | bash

Troubleshooting

  • Model loading failed - Invalid model repository structure: Ensure models follow the {model_name}/{version}/model.{ext} directory structure and version directories contain numeric names like '1' or '2'
  • CUDA out of memory errors during model loading: Reduce model instance counts in config.pbtxt, enable model unloading policies, or configure smaller memory pools for concurrent models
  • gRPC connection refused on port 8001: Verify the container is running with --gpcs-port=8001 and check firewall rules allowing gRPC traffic on that port
  • Models not auto-loading when added to repository: Enable model repository polling with --repository-poll-secs=30 command line argument or restart the container to rescan
  • High inference latency despite GPU availability: Enable dynamic batching in model configuration and tune max_batch_size and batch_timeout_microseconds parameters
  • Permission denied accessing model files: Ensure the model_repository directory has proper ownership (docker user) and read permissions for all model files and directories

Community Notes

Loading...
Loading notes...

Download Recipe Kit

Get all files in a ready-to-deploy package

Includes docker-compose.yml, .env template, README, and license

Components

triton

Tags

#triton#inference#nvidia#model-serving#ml

Category

AI & Machine Learning

Related Recipes

Ad Space