$docker.recipes

NVIDIA Triton Inference Server

advanced

High-performance inference server for ML models.

[i]Overview

NVIDIA Triton Inference Server is a high-performance, open-source inference serving platform developed by NVIDIA for deploying machine learning models at scale in production environments. Originally built to handle NVIDIA's internal ML workloads, Triton supports multiple deep learning frameworks including TensorRT, ONNX Runtime, TensorFlow, PyTorch, and custom backends, making it framework-agnostic while optimizing performance across different model types. The server provides concurrent model execution, dynamic batching, and model ensemble capabilities that maximize GPU utilization and throughput. This Triton deployment configuration leverages GPU acceleration through Docker's NVIDIA runtime integration, exposing three distinct service endpoints for different use cases. The HTTP REST API on port 8000 handles standard web-based inference requests, while the gRPC interface on port 8001 provides low-latency, high-throughput communication for performance-critical applications. Port 8002 serves Prometheus metrics for comprehensive monitoring and observability of inference performance, model statistics, and resource utilization. ML engineers, AI platform teams, and organizations deploying production inference workloads will benefit from this configuration's ability to serve multiple models simultaneously with automatic scaling and load balancing. The model repository structure supports versioning and A/B testing scenarios, while the multi-protocol support enables integration with diverse client applications ranging from web services to real-time streaming applications requiring microsecond-level response times.

[*]Key Features

  • [+]Multi-framework model serving supporting TensorRT, ONNX, TensorFlow, PyTorch, and custom backends
  • [+]Dynamic request batching that automatically groups individual requests to maximize GPU throughput
  • [+]Model ensemble capabilities allowing complex inference pipelines with preprocessing and postprocessing
  • [+]Concurrent model execution enabling multiple models to run simultaneously on shared GPU resources
  • [+]Model versioning and A/B testing support through structured repository organization
  • [+]Built-in performance optimization including CUDA kernel fusion and memory pool management
  • [+]Real-time inference metrics and health monitoring through Prometheus integration
  • [+]Multi-protocol support with HTTP REST, gRPC, and C API for diverse client integration patterns

[#]Common Use Cases

  • [1]Computer vision applications requiring real-time object detection, classification, and segmentation
  • [2]Natural language processing services for text classification, sentiment analysis, and language translation
  • [3]Recommendation engines serving personalized content with sub-100ms response times
  • [4]Edge AI deployments where multiple models need to share limited GPU resources efficiently
  • [5]MLOps pipelines requiring model versioning, canary deployments, and performance monitoring
  • [6]High-frequency trading systems using ML models for market prediction and risk assessment
  • [7]Autonomous vehicle inference stacks processing multiple sensor inputs through different neural networks

[!]Prerequisites

  • [!]NVIDIA GPU with CUDA Compute Capability 6.0 or higher and minimum 4GB VRAM
  • [!]NVIDIA Docker runtime (nvidia-docker2) installed and configured on the host system
  • [!]Docker Compose version 3.8+ with GPU device support enabled
  • [!]Minimum 8GB system RAM for model loading and request queuing buffers
  • [!]Pre-trained models in supported formats (TensorRT .plan, ONNX .onnx, TensorFlow SavedModel, PyTorch .pt)
  • [!]Understanding of model repository structure and Triton configuration files (config.pbtxt)
[!]

WARNING: For development & testing. Review security settings, change default credentials, and test thoroughly before production use. See Terms

[$]docker-compose.yml

[docker-compose.yml]
1services: 
2  triton: 
3    image: nvcr.io/nvidia/tritonserver:23.10-py3
4    container_name: triton
5    restart: unless-stopped
6    command: tritonserver --model-repository=/models
7    volumes: 
8      - ./models:/models
9    ports: 
10      - "8000:8000"
11      - "8001:8001"
12      - "8002:8002"
13    deploy: 
14      resources: 
15        reservations: 
16          devices: 
17            - driver: nvidia
18              count: all
19              capabilities: [gpu]

[$].env Template

[.env]
1# Place models in ./models directory

[i]Usage Notes

  1. [1]Docs: https://docs.nvidia.com/deeplearning/triton-inference-server/
  2. [2]HTTP REST API at http://localhost:8000/v2/models/{model}/infer
  3. [3]gRPC endpoint at localhost:8001 for high-performance inference
  4. [4]Prometheus metrics at http://localhost:8002/metrics
  5. [5]Model repository structure: models/{model_name}/{version}/model.plan
  6. [6]Supports TensorRT, ONNX, TensorFlow, PyTorch backends

[>]Quick Start

[terminal]
1# 1. Create the compose file
2cat > docker-compose.yml << 'EOF'
3services:
4  triton:
5    image: nvcr.io/nvidia/tritonserver:23.10-py3
6    container_name: triton
7    restart: unless-stopped
8    command: tritonserver --model-repository=/models
9    volumes:
10      - ./models:/models
11    ports:
12      - "8000:8000"
13      - "8001:8001"
14      - "8002:8002"
15    deploy:
16      resources:
17        reservations:
18          devices:
19            - driver: nvidia
20              count: all
21              capabilities: [gpu]
22EOF
23
24# 2. Create the .env file
25cat > .env << 'EOF'
26# Place models in ./models directory
27EOF
28
29# 3. Start the services
30docker compose up -d
31
32# 4. View logs
33docker compose logs -f

[>]One-Liner

Run this command to download and set up the recipe in one step:

[terminal]
1curl -fsSL https://docker.recipes/api/recipes/nvidia-triton/run | bash

[?]Troubleshooting

  • [!]Error 'nvidia-smi has failed because it couldn't communicate with the NVIDIA driver': Install NVIDIA drivers and verify nvidia-docker2 runtime configuration
  • [!]Model fails to load with 'Invalid argument: model configuration': Create proper config.pbtxt file with correct input/output tensor specifications for your model
  • [!]Out of memory errors during inference: Reduce max_batch_size in model configuration or implement request queuing with smaller batch sizes
  • [!]gRPC client connection refused on port 8001: Verify firewall settings allow gRPC traffic and client is using correct protobuf definitions
  • [!]Prometheus metrics showing high queue times: Enable dynamic batching or increase instance_group count in model configuration
  • [!]TensorRT models fail to load with 'engine incompatible': Rebuild TensorRT engines on target GPU architecture or use ONNX format for portability

Community Notes

Loading...
Loading notes...

## Download Recipe Kit

Get all files in a ready-to-deploy package

Includes docker-compose.yml, .env template, README, and license

## Components

triton

## Tags

#triton#inference#nvidia#serving

## Category

AI & Machine Learning

## Related