Vitess

advanced

Database clustering system for horizontal scaling of MySQL.

Overview

Vitess is YouTube's battle-tested database clustering system designed to horizontally scale MySQL workloads across thousands of servers. Originally developed to handle YouTube's massive growth, Vitess combines transparent sharding, connection pooling, and query routing to transform MySQL from a single-server database into a distributed system capable of serving millions of queries per second. The system acts as a proxy layer between applications and MySQL instances, automatically distributing data and queries across multiple shards while maintaining MySQL compatibility. This Docker configuration deploys vttestserver, which provides a complete Vitess environment in a single container for development and testing purposes. The stack includes VTGate as the SQL-aware proxy layer on port 15001, which handles query parsing, routing, and connection pooling, while maintaining full MySQL protocol compatibility on port 33577. VTGate automatically determines which shards need to be queried based on the VSchema configuration and can perform cross-shard joins and transactions when necessary. This setup is ideal for developers building applications that need to scale beyond single MySQL instances, platform engineers evaluating sharding strategies, and teams preparing to migrate existing MySQL workloads to a horizontally scaled architecture. Companies like Slack, Square, and GitHub rely on Vitess in production to handle billions of queries daily, making this an excellent choice for applications expecting rapid growth or already hitting MySQL scaling limits.

Key Features

Automatic horizontal sharding with configurable shard counts and distribution strategies
VTGate proxy layer providing MySQL wire protocol compatibility with intelligent query routing
Cross-shard query execution including distributed joins and transactions
Built-in connection pooling and query optimization to reduce database load
VSchema-based data distribution allowing fine-grained control over shard placement
Online schema migration tools for zero-downtime database changes across all shards
Comprehensive monitoring and observability with built-in metrics and query analytics
Backup and restore capabilities designed for distributed MySQL environments

Common Use Cases

1E-commerce platforms needing to partition customer and order data across multiple MySQL instances
2SaaS applications implementing tenant-based sharding for multi-customer database isolation
3Gaming backends requiring horizontal scaling for player data and leaderboard systems
4Financial services applications needing distributed transaction processing across sharded datasets
5Social media platforms managing user profiles, feeds, and relationships across geographic regions
6IoT data collection systems storing time-series data across date-based or device-based shards
7Development environments for testing sharding strategies before production MySQL migrations

Prerequisites

At least 2GB RAM available for Vitess components and internal MySQL instances
Understanding of MySQL query optimization and database sharding concepts
Familiarity with VSchema syntax for defining keyspace and table distribution rules
Knowledge of MySQL client tools and connection string formats for testing
Ports 33577, 15000, and 15001 available for Vitess services and web interfaces
Basic understanding of horizontal partitioning strategies and shard key selection

For development & testing. Review security settings, change default credentials, and test thoroughly before production use. See Terms

docker-compose.yml

1services: 
2  vitess: 
3    image: vitess/vttestserver:latest
4    container_name: vitess
5    restart: unless-stopped
6    environment: 
7      PORT: 33577
8      KEYSPACES: commerce
9      NUM_SHARDS: 1
10    ports: 
11      - "33577:33577"
12      - "15000:15000"
13      - "15001:15001"
14    networks: 
15      - vitess-network
16
17networks: 
18  vitess-network: 
19    driver: bridge

.env Template

.env

1# Vitess test server configuration

Usage Notes

1Docs: https://vitess.io/docs/
2MySQL-compatible on port 33577 - use mysql client
3VTGate on port 15001 - SQL-aware proxy layer
4Automatic sharding - scales MySQL horizontally
5Production-proven at YouTube, Slack, Square, GitHub
6VSchema defines how data is distributed across shards

Quick Start

terminal

1# 1. Create the compose file
2cat > docker-compose.yml << 'EOF'
3services:
4  vitess:
5    image: vitess/vttestserver:latest
6    container_name: vitess
7    restart: unless-stopped
8    environment:
9      PORT: 33577
10      KEYSPACES: commerce
11      NUM_SHARDS: 1
12    ports:
13      - "33577:33577"
14      - "15000:15000"
15      - "15001:15001"
16    networks:
17      - vitess-network
18
19networks:
20  vitess-network:
21    driver: bridge
22EOF
23
24# 2. Create the .env file
25cat > .env << 'EOF'
26# Vitess test server configuration
27EOF
28
29# 3. Start the services
30docker compose up -d
31
32# 4. View logs
33docker compose logs -f

One-Liner

Run this command to download and set up the recipe in one step:

terminal

1curl -fsSL https://docker.recipes/api/recipes/vitess/run | bash

Troubleshooting

Connection refused on port 33577: Wait for complete startup as Vitess initializes multiple internal components including vtgate, vttablet, and MySQL instances
Query fails with 'table not found' error: Verify VSchema configuration includes proper table definitions and routing rules for the target keyspace
Cross-shard queries returning incomplete results: Check that all shards are healthy and accessible through VTGate's shard discovery mechanism
High memory usage during startup: vttestserver creates multiple processes including MySQL instances, vtgate, and vttablet - ensure sufficient container memory limits
VTGate web interface (port 15001) shows no tablets: Restart container as tablet registration with topology service may have failed during initialization
Slow query performance on sharded tables: Ensure queries include shard key in WHERE clauses to avoid expensive scatter-gather operations across all shards

Community Notes

Loading notes...

Download Recipe Kit

Get all files in a ready-to-deploy package

Includes docker-compose.yml, .env template, README, and license

Components

vitess

Related Recipes

Ad Space

Shortcuts: C CopyF FavoriteD Download