vLLM

Key Features

• PagedAttention algorithm delivering up to 24x higher throughput than standard transformer implementations
• OpenAI-compatible API endpoints supporting chat completions, completions, and embeddings
• Tensor parallelism support for distributing large models across multiple GPUs using --tensor-parallel-size
• Integrated quantization support for AWQ, GPTQ, and SqueezeLLM to reduce VRAM requirements
• Dynamic batching with continuous batching optimization for improved request handling efficiency
• Native Hugging Face Hub integration with automatic model downloading and caching
• GPU memory optimization through non-contiguous KV cache management eliminating fragmentation
• Support for popular model architectures including LLaMA, Mistral, CodeLlama, and Vicuna

Common Use Cases

• 1High-throughput chatbot backends requiring concurrent handling of hundreds of user requests
• 2Code generation services needing fast response times for developer IDE integrations
• 3Research environments comparing multiple LLM outputs with consistent serving infrastructure
• 4Production AI applications requiring OpenAI API compatibility without vendor lock-in
• 5Multi-tenant SaaS platforms serving different models to various customer segments
• 6Real-time content generation systems for marketing, writing assistance, or creative applications
• 7Edge AI deployments where memory efficiency is critical for running large models on limited hardware

Prerequisites

• NVIDIA GPU with CUDA support and sufficient VRAM for your target model (minimum 8GB for 7B models)
• Docker with NVIDIA Container Runtime installed and configured for GPU access
• Hugging Face account and token for accessing gated models like LLaMA-2 or Code Llama
• At least 32GB system RAM recommended for optimal performance with larger models
• Understanding of transformer model architectures and parameter counts for appropriate resource allocation
• Familiarity with OpenAI API format for integration with existing applications

docker-compose.yml

1services: 
2  vllm: 
3    image: vllm/vllm-openai:latest
4    container_name: vllm
5    restart: unless-stopped
6    command: --model ${MODEL_NAME} --port 8000
7    environment: 
8      HUGGING_FACE_HUB_TOKEN: ${HF_TOKEN}
9    volumes: 
10      - vllm_cache:/root/.cache/huggingface
11    ports: 
12      - "8000:8000"
13    deploy: 
14      resources: 
15        reservations: 
16          devices: 
17            - driver: nvidia
18              count: all
19              capabilities: [gpu]
20
21volumes: 
22  vllm_cache: 

.env Template

1MODEL_NAME=meta-llama/Llama-2-7b-chat-hf
2HF_TOKEN=your_huggingface_token

Usage Notes

Quick Start

1# 1. Create the compose file
2cat > docker-compose.yml << 'EOF'
3services:
4  vllm:
5    image: vllm/vllm-openai:latest
6    container_name: vllm
7    restart: unless-stopped
8    command: --model ${MODEL_NAME} --port 8000
9    environment:
10      HUGGING_FACE_HUB_TOKEN: ${HF_TOKEN}
11    volumes:
12      - vllm_cache:/root/.cache/huggingface
13    ports:
14      - "8000:8000"
15    deploy:
16      resources:
17        reservations:
18          devices:
19            - driver: nvidia
20              count: all
21              capabilities: [gpu]
22
23volumes:
24  vllm_cache:
25EOF
26
27# 2. Create the .env file
28cat > .env << 'EOF'
29MODEL_NAME=meta-llama/Llama-2-7b-chat-hf
30HF_TOKEN=your_huggingface_token
31EOF
32
33# 3. Start the services
34docker compose up -d
35
36# 4. View logs
37docker compose logs -f

One-Liner

Run this command to download and set up the recipe in one step:

1curl -fsSL https://docker.recipes/api/recipes/vllm/run | bash

Troubleshooting