# LLM Development Stack

> Local LLM development with Ollama, Open WebUI, ChromaDB vector store, and LangServe

- Category: AI & Machine Learning
- Difficulty: intermediate
- Components: ollama, open-webui, chromadb, langserve
- Tags: llm, ollama, rag, chromadb, langchain, ai
- Source: https://docker.recipes/ai-ml/llm-development-stack

## Overview

Ollama transforms how developers work with large language models by providing a simple way to run models like Llama 2, Mistral, and Code Llama locally without complex setup or cloud dependencies. Rather than managing CUDA drivers, model weights, and inference servers separately, Ollama packages everything into a streamlined interface that handles model downloading, quantization, and serving through an OpenAI-compatible API. This eliminates the traditional barriers of LLM deployment while maintaining full control over data privacy and inference costs. This development stack combines Ollama's local model serving with Open WebUI's ChatGPT-like interface, ChromaDB's vector storage capabilities, and LangServe's API framework to create a complete RAG-enabled AI development environment. The integration allows developers to experiment with retrieval-augmented generation, build custom LLM applications, and prototype AI features without sending data to external services or managing complex model infrastructure. ChromaDB provides persistent vector embeddings for document retrieval, while LangServe exposes LangChain applications as production-ready APIs. AI startups, privacy-conscious enterprises, and developers building LLM-powered applications will find this stack invaluable for rapid prototyping and production deployment. The combination delivers the full ChatGPT experience locally while providing the vector storage and API infrastructure needed for sophisticated RAG applications, making it ideal for teams that need to iterate quickly on AI features without cloud dependencies or data privacy concerns.

## Features

- Local LLM serving with Ollama's simple model management and GPU acceleration
- ChatGPT-like web interface through Open WebUI with chat history and model switching
- Persistent vector storage via ChromaDB for document embeddings and similarity search
- LangChain application serving through LangServe's FastAPI-based endpoints
- OpenAI-compatible API endpoints enabling drop-in replacement for existing applications
- Multi-model support allowing switching between Llama 2, Mistral, Code Llama and other models
- RAG document processing with automatic embedding generation and retrieval
- Multi-user support with individual chat histories and prompt template sharing

## Use cases

- AI startups prototyping RAG applications with proprietary documents and local model inference
- Enterprise teams building internal AI assistants without sending sensitive data to external APIs
- Developers creating LLM-powered applications that need offline functionality and data privacy
- Research teams experimenting with different models and RAG configurations for academic projects
- Companies migrating from OpenAI API to self-hosted solutions for cost control and compliance
- Development teams building chatbots that combine company knowledge bases with conversational AI
- Organizations creating internal documentation assistants using ChromaDB for vector search

## Prerequisites

- Minimum 16GB RAM for running 7B parameter models, 32GB+ recommended for 13B models
- NVIDIA GPU with CUDA support for optimal inference performance (CPU fallback available)
- Available ports 3000, 8000, 8080, and 11434 for Open WebUI, ChromaDB, LangServe, and Ollama
- Docker Compose with GPU support configured for NVIDIA container runtime
- Basic understanding of LangChain concepts for customizing RAG pipelines
- Familiarity with vector databases and embedding models for ChromaDB configuration

## docker-compose.yml

```yaml
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "${OLLAMA_PORT:-11434}:11434"
    volumes:
      - ollama_data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    ports:
      - "${WEBUI_PORT:-3000}:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - WEBUI_SECRET_KEY=${SECRET_KEY}
    volumes:
      - open_webui_data:/app/backend/data
    depends_on:
      - ollama

  chromadb:
    image: chromadb/chroma:latest
    container_name: chromadb
    restart: unless-stopped
    ports:
      - "${CHROMA_PORT:-8000}:8000"
    volumes:
      - chroma_data:/chroma/chroma
    environment:
      - IS_PERSISTENT=TRUE
      - ANONYMIZED_TELEMETRY=FALSE

  langserve:
    build:
      context: ./langserve
      dockerfile: Dockerfile
    container_name: langserve
    restart: unless-stopped
    ports:
      - "${LANGSERVE_PORT:-8080}:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - CHROMA_HOST=chromadb
      - CHROMA_PORT=8000
    depends_on:
      - ollama
      - chromadb
    profiles:
      - dev

volumes:
  ollama_data:
  open_webui_data:
  chroma_data:
```

## .env

```bash
# LLM Development Stack
OLLAMA_PORT=11434
WEBUI_PORT=3000
CHROMA_PORT=8000
LANGSERVE_PORT=8080

# Security
SECRET_KEY=your-secret-key
```

## Usage notes

- Open WebUI at http://localhost:3000 (create account)
- Ollama API at http://localhost:11434
- ChromaDB at http://localhost:8000
- Pull models: docker exec ollama ollama pull llama2
- ChromaDB provides vector storage for RAG
- LangServe exposes LangChain apps as APIs

## Troubleshooting

- Ollama fails to start with GPU errors: Ensure NVIDIA container toolkit is installed and docker-compose includes GPU runtime configuration
- Open WebUI shows 'Cannot connect to Ollama': Verify Ollama container is running and accessible at http://ollama:11434 within Docker network
- ChromaDB connection refused: Check if port 8000 is available and container started successfully with persistent volume mounted
- Models fail to load with OOM errors: Reduce model size or increase Docker memory limits, consider using quantized versions
- LangServe API returns embedding errors: Ensure ChromaDB is running before starting LangServe container and environment variables are correctly set
- Chat responses extremely slow: Enable GPU acceleration in Ollama or switch to smaller quantized models for better performance

