# vLLM

> High-throughput LLM serving with PagedAttention.

- Category: AI & Machine Learning
- Difficulty: intermediate
- Components: vllm
- Tags: vllm, llm, inference, serving
- Source: https://docker.recipes/ai-ml/vllm

## Overview

vLLM (versatile Large Language Model) is a high-performance inference engine developed by UC Berkeley that revolutionizes LLM serving through its innovative PagedAttention algorithm. Originally created to address the memory bottlenecks in transformer-based language model serving, vLLM dynamically manages attention key-value tensors using virtual memory techniques similar to operating system paging. This breakthrough enables up to 24x higher serving throughput compared to traditional methods like Hugging Face Transformers while maintaining identical output quality.

vLLM operates as an OpenAI-compatible API server that can serve virtually any transformer-based language model from Hugging Face Hub with minimal configuration changes. The system efficiently manages GPU memory by storing attention keys and values in non-contiguous memory blocks, eliminating the need for pre-allocation of large contiguous memory chunks. This approach dramatically reduces memory fragmentation and enables higher batch sizes, leading to unprecedented throughput improvements for LLM inference workloads.

This configuration is ideal for AI researchers, ML engineers, and organizations requiring high-performance LLM serving infrastructure. The setup supports everything from small 7B parameter models on single GPUs to massive models requiring tensor parallelism across multiple GPUs. With built-in support for quantization techniques like AWQ, GPTQ, and SqueezeLLM, vLLM enables deployment of large models even on hardware with limited VRAM while maintaining excellent performance characteristics.

## Features

- PagedAttention algorithm delivering up to 24x higher throughput than standard transformer implementations
- OpenAI-compatible API endpoints supporting chat completions, completions, and embeddings
- Tensor parallelism support for distributing large models across multiple GPUs using --tensor-parallel-size
- Integrated quantization support for AWQ, GPTQ, and SqueezeLLM to reduce VRAM requirements
- Dynamic batching with continuous batching optimization for improved request handling efficiency
- Native Hugging Face Hub integration with automatic model downloading and caching
- GPU memory optimization through non-contiguous KV cache management eliminating fragmentation
- Support for popular model architectures including LLaMA, Mistral, CodeLlama, and Vicuna

## Use cases

- High-throughput chatbot backends requiring concurrent handling of hundreds of user requests
- Code generation services needing fast response times for developer IDE integrations
- Research environments comparing multiple LLM outputs with consistent serving infrastructure
- Production AI applications requiring OpenAI API compatibility without vendor lock-in
- Multi-tenant SaaS platforms serving different models to various customer segments
- Real-time content generation systems for marketing, writing assistance, or creative applications
- Edge AI deployments where memory efficiency is critical for running large models on limited hardware

## Prerequisites

- NVIDIA GPU with CUDA support and sufficient VRAM for your target model (minimum 8GB for 7B models)
- Docker with NVIDIA Container Runtime installed and configured for GPU access
- Hugging Face account and token for accessing gated models like LLaMA-2 or Code Llama
- At least 32GB system RAM recommended for optimal performance with larger models
- Understanding of transformer model architectures and parameter counts for appropriate resource allocation
- Familiarity with OpenAI API format for integration with existing applications

## docker-compose.yml

```yaml
services:
  vllm:
    image: vllm/vllm-openai:latest
    container_name: vllm
    restart: unless-stopped
    command: --model ${MODEL_NAME} --port 8000
    environment:
      HUGGING_FACE_HUB_TOKEN: ${HF_TOKEN}
    volumes:
      - vllm_cache:/root/.cache/huggingface
    ports:
      - "8000:8000"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

volumes:
  vllm_cache:
```

## .env

```bash
MODEL_NAME=meta-llama/Llama-2-7b-chat-hf
HF_TOKEN=your_huggingface_token
```

## Usage notes

- Docs: https://docs.vllm.ai/
- OpenAI-compatible API at http://localhost:8000/v1/chat/completions
- PagedAttention enables 24x higher throughput than HF Transformers
- Tensor parallelism: --tensor-parallel-size N for multi-GPU
- Supports AWQ, GPTQ, SqueezeLLM quantization for reduced VRAM
- HF token required for gated models like Llama-2

## Troubleshooting

- CUDA out of memory errors: Reduce model size, enable quantization with --quantization awq, or increase tensor parallelism across multiple GPUs
- Model download failures: Verify HF_TOKEN environment variable is set correctly and has access to the requested model repository
- Slow inference despite high-end hardware: Check GPU utilization and consider adjusting --max-num-batched-tokens or --max-num-seqs parameters
- Container fails to start with GPU detection errors: Ensure nvidia-container-runtime is properly installed and Docker daemon restarted
- OpenAI API compatibility issues: Verify client libraries support the /v1/chat/completions endpoint format and adjust request formatting
- High memory usage during model loading: Use --load-format pt for faster loading or --download-dir to specify custom model cache location

