# Vitess MySQL Sharding Cluster

> Horizontal MySQL scaling with Vitess, VTGate routing, and tablet management.

- Category: Database Stacks
- Difficulty: advanced
- Components: vitess, vtgate, vttablet, etcd, mysql
- Tags: vitess, mysql, sharding, scaling, youtube
- Source: https://docker.recipes/databases/vitess-mysql-cluster

## Overview

Vitess is YouTube's battle-tested database clustering system that transforms MySQL into a horizontally scalable platform capable of handling massive workloads. Originally developed to manage YouTube's explosive growth, Vitess acts as a transparent proxy layer that provides MySQL sharding, connection pooling, and query routing without requiring application changes. The system uses VTGate as the stateless frontend that accepts MySQL connections and routes queries to appropriate shards, while VTTablet manages individual MySQL instances and handles tablet lifecycle operations.

This clustering configuration combines Vitess components with etcd for distributed coordination, creating a production-ready sharded MySQL environment. VTGate serves as the MySQL-compatible entry point at port 3306, routing queries across multiple VTTablet instances that each manage their own MySQL data. The etcd cluster maintains topology information and coordinates tablet assignments, while VTCtld provides administrative control over the entire cluster through both web UI and command-line tools.

This stack targets organizations experiencing MySQL scaling bottlenecks that single-server configurations cannot resolve. Companies with read-heavy workloads, large datasets requiring horizontal partitioning, or applications needing transparent sharding will benefit from Vitess's proven architecture. The included Prometheus and Grafana monitoring provides visibility into query performance, tablet health, and sharding effectiveness, making it suitable for production environments where observability is crucial.

## Features

- Transparent MySQL sharding with automatic query routing across multiple tablet instances
- VTGate connection pooling and query optimization reduces database connection overhead
- VTTablet automated failover and primary election for high availability within shards
- VTCtld web interface for visual cluster management and topology monitoring
- Etcd-based topology service maintains consistent cluster state and tablet assignments
- MySQL-compatible protocol support allows existing applications to connect without modification
- Built-in query rewriting and cross-shard join optimization
- Integrated Prometheus metrics export for detailed performance monitoring and alerting

## Use cases

- E-commerce platforms with growing product catalogs and user bases requiring horizontal MySQL scaling
- SaaS applications implementing tenant-based sharding for multi-customer data isolation
- Gaming companies managing player data and statistics across geographically distributed shards
- Financial services platforms requiring high-availability transactional systems with read replica scaling
- Content management systems with large media libraries needing transparent database partitioning
- Analytics platforms processing time-series data with date-based sharding strategies
- Social media applications managing user-generated content across multiple database instances

## Prerequisites

- Minimum 4GB RAM recommended for running multiple VTTablet instances with embedded MySQL
- Basic understanding of MySQL sharding concepts and query routing principles
- Familiarity with etcd cluster operations and distributed consensus systems
- Available ports 3306, 15000, 15001, 15991, 2379, 9090, and 3000 for service access
- Understanding of Vitess topology concepts including cells, keyspaces, and tablet types
- Experience with Prometheus and Grafana for monitoring distributed database systems

## docker-compose.yml

```yaml
services:
  etcd:
    image: quay.io/coreos/etcd:v3.5.0
    command:
      - etcd
      - --data-dir=/etcd-data
      - --listen-client-urls=http://0.0.0.0:2379
      - --advertise-client-urls=http://etcd:2379
    volumes:
      - etcd_data:/etcd-data
    networks:
      - vitess_net

  vtctld:
    image: vitess/lite:latest
    command:
      - vtctld
      - --topo_implementation=etcd2
      - --topo_global_server_address=etcd:2379
      - --topo_global_root=/vitess/global
      - --service_map=grpc-vtctl
    ports:
      - "15000:15000"
      - "15999:15999"
    depends_on:
      - etcd
    networks:
      - vitess_net

  vttablet-zone1-1:
    image: vitess/lite:latest
    command:
      - vttablet
      - --topo_implementation=etcd2
      - --topo_global_server_address=etcd:2379
      - --topo_global_root=/vitess/global
      - --tablet-path=zone1-0000000001
      - --init_keyspace=commerce
      - --init_shard=0
      - --init_tablet_type=replica
      - --port=15001
      - --grpc_port=16001
      - --service_map=grpc-queryservice,grpc-tabletmanager,grpc-updatestream
    volumes:
      - vttablet1_data:/vt/vtdataroot
    depends_on:
      - vtctld
    networks:
      - vitess_net

  vttablet-zone1-2:
    image: vitess/lite:latest
    command:
      - vttablet
      - --topo_implementation=etcd2
      - --topo_global_server_address=etcd:2379
      - --topo_global_root=/vitess/global
      - --tablet-path=zone1-0000000002
      - --init_keyspace=commerce
      - --init_shard=0
      - --init_tablet_type=replica
      - --port=15002
      - --grpc_port=16002
      - --service_map=grpc-queryservice,grpc-tabletmanager,grpc-updatestream
    volumes:
      - vttablet2_data:/vt/vtdataroot
    depends_on:
      - vtctld
    networks:
      - vitess_net

  vtgate:
    image: vitess/lite:latest
    command:
      - vtgate
      - --topo_implementation=etcd2
      - --topo_global_server_address=etcd:2379
      - --topo_global_root=/vitess/global
      - --port=15001
      - --grpc_port=15991
      - --mysql_server_port=3306
      - --mysql_auth_server_impl=none
      - --cell=zone1
      - --cells_to_watch=zone1
      - --tablet_types_to_wait=PRIMARY,REPLICA
      - --service_map=grpc-vtgateservice
    ports:
      - "3306:3306"
      - "15001:15001"
      - "15991:15991"
    depends_on:
      - vttablet-zone1-1
      - vttablet-zone1-2
    networks:
      - vitess_net

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - prometheus_data:/prometheus
    networks:
      - vitess_net

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
    volumes:
      - grafana_data:/var/lib/grafana
    networks:
      - vitess_net

volumes:
  etcd_data:
  vttablet1_data:
  vttablet2_data:
  prometheus_data:
  grafana_data:

networks:
  vitess_net:
```

## .env

```bash
# Vitess MySQL Cluster
GRAFANA_PASSWORD=secure_grafana_password

# MySQL at localhost:3306
# VTCtld at http://localhost:15000
```

## Usage notes

- MySQL-compatible at port 3306
- VTCtld admin at http://localhost:15000
- Horizontal sharding support
- Used by YouTube for scale
- Configure sharding via VTCtl

## Troubleshooting

- VTGate connection refused at port 3306: Ensure VTTablet instances are healthy and registered in etcd topology before VTGate starts accepting connections
- Tablets stuck in 'NOT_SERVING' state: Check VTCtld logs for keyspace initialization errors and run 'vtctlclient InitShardPrimary' to assign primary tablets
- Etcd cluster unhealthy warnings: Verify etcd data directory permissions and ensure no port conflicts with existing etcd instances on the host
- VTCtld web interface shows empty topology: Confirm etcd connectivity and verify --topo_global_root path matches across all Vitess components
- Query routing errors for cross-shard operations: Review VTGate logs for unsupported query patterns and consider query rewriting for complex joins
- High memory usage in VTTablet containers: Adjust MySQL buffer pool settings or increase container memory limits based on dataset size

