# Kubeflow Pipelines

> ML workflow orchestration on Kubernetes.

- Category: AI & Machine Learning
- Difficulty: advanced
- Components: kubeflow
- Tags: kubeflow, kubernetes, ml-ops, pipelines
- Source: https://docker.recipes/ai-ml/kubeflow

## Overview

Kubeflow Pipelines is a comprehensive platform for building and deploying portable, scalable machine learning workflows based on Docker containers. Originally developed by Google and now part of the Cloud Native Computing Foundation, Kubeflow enables data scientists and ML engineers to orchestrate complex machine learning experiments and production workflows on Kubernetes. The platform provides a web-based UI for managing pipelines, experiments, and runs, along with a Python SDK for programmatic pipeline creation.

This configuration establishes the essential storage infrastructure required for Kubeflow Pipelines development and testing. MinIO serves as the S3-compatible object storage backend for artifacts, models, and intermediate data, while MySQL provides the metadata store for pipeline definitions, experiment tracking, and run histories. These components form the foundational data layer that Kubeflow Pipelines requires for persistent storage of ML workflows and their associated artifacts.

This stack is ideal for ML engineers setting up local development environments, data science teams prototyping pipeline architectures, and organizations preparing infrastructure for Kubeflow deployment. While full Kubeflow requires Kubernetes orchestration, this foundation enables developers to build and test pipelines using the Kubeflow Pipelines SDK before deploying to production Kubernetes clusters. The combination provides a lightweight development environment that mirrors the storage patterns of full Kubeflow deployments.

## Features

- S3-compatible object storage via MinIO for ML artifacts and model persistence
- MySQL metadata store for pipeline definitions and experiment tracking
- Compatible with Kubeflow Pipelines SDK for local pipeline development
- MinIO web interface for browsing and managing ML artifacts and datasets
- Persistent volume storage ensuring data survives container restarts
- Network isolation between storage components for development security
- Configurable MySQL database schema supporting MLPipeline metadata
- MinIO bucket structure supporting Kubeflow's artifact organization patterns

## Use cases

- Local development environment for Kubeflow pipeline authors and data scientists
- ML artifact storage and versioning during model experimentation phases
- Prototype testing of pipeline storage patterns before Kubernetes deployment
- Educational environments for learning Kubeflow Pipelines concepts and workflows
- CI/CD pipeline testing for ML workflows requiring persistent storage backends
- Data science team collaboration with shared artifact storage and experiment tracking
- Migration testing when moving ML workflows from other platforms to Kubeflow

## Prerequisites

- Docker and Docker Compose installed with at least 4GB available memory
- Port 9000 available for MinIO web interface and API access
- Python 3.7+ with kfp (Kubeflow Pipelines SDK) installed for pipeline development
- Basic understanding of ML workflows and pipeline orchestration concepts
- Familiarity with S3-compatible storage APIs for artifact management
- 8GB+ disk space for MySQL data and MinIO object storage

## docker-compose.yml

```yaml
# Kubeflow requires Kubernetes
# Use kind or minikube for local development
# This is a simplified standalone example
services:
  minio:
    image: minio/minio:latest
    container_name: kubeflow-minio
    command: server /data
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: minioadmin
    volumes:
      - minio_data:/data
    ports:
      - "9000:9000"
    networks:
      - kubeflow

  mysql:
    image: mysql:8.0
    container_name: kubeflow-mysql
    environment:
      MYSQL_ROOT_PASSWORD: root
      MYSQL_DATABASE: mlpipeline
    volumes:
      - mysql_data:/var/lib/mysql
    networks:
      - kubeflow

volumes:
  minio_data:
  mysql_data:

networks:
  kubeflow:
    driver: bridge
```

## .env

```bash
# Full Kubeflow requires Kubernetes
```

## Usage notes

- Docs: https://www.kubeflow.org/docs/
- Full Kubeflow requires Kubernetes - this provides storage backend only
- Author pipelines with kfp SDK: pip install kfp
- MinIO at http://localhost:9000 - credentials: minioadmin/minioadmin
- For local dev, use kind or minikube with Kubeflow manifests
- Alternative: use standalone Kubeflow Pipelines v2 on K8s

## Troubleshooting

- MinIO 'Access Denied' errors: Verify credentials are minioadmin/minioadmin and check bucket policies in the web interface
- MySQL connection refused: Ensure MySQL container is fully started (check logs for 'ready for connections') before connecting applications
- KFP SDK artifact upload failures: Verify MinIO endpoint configuration and ensure bucket exists for the pipeline namespace
- MySQL 'table doesn't exist' errors: Run pipeline metadata initialization scripts or ensure MLPipeline database schema is properly created
- MinIO console not accessible: Check that port 9000 is not blocked by firewall and container is running with correct port mapping
- Pipeline metadata persistence issues: Verify MySQL volume mounts are correctly configured and database has sufficient disk space

