# Kubeflow Pipelines Standalone

> Kubeflow Pipelines for ML workflow orchestration without full Kubernetes.

- Category: AI & Machine Learning
- Difficulty: advanced
- Components: kubeflow-pipelines, mysql, minio, workflow-controller
- Tags: kubeflow, pipelines, ml-ops, workflow, orchestration
- Source: https://docker.recipes/ai-ml/kubeflow-pipelines

## Overview

Kubeflow Pipelines is Google's open-source machine learning workflow orchestration platform, originally designed to run on Kubernetes clusters. It enables data scientists and ML engineers to build, deploy, and manage end-to-end ML workflows as code, with features like experiment tracking, pipeline versioning, and automated artifact management. This standalone deployment brings Kubeflow Pipelines capabilities to environments without full Kubernetes infrastructure.

This stack combines the Kubeflow Pipelines API server and frontend with MySQL for metadata storage and MinIO for artifact storage. MySQL handles pipeline definitions, execution history, and experiment metadata, while MinIO provides S3-compatible object storage for datasets, models, and pipeline artifacts. The API server orchestrates workflow execution and the frontend provides a web interface for pipeline management and visualization.

This configuration targets ML teams, data scientists, and organizations wanting to implement MLOps practices without the complexity of managing a full Kubernetes cluster. It's particularly valuable for smaller teams, development environments, edge deployments, or organizations with limited infrastructure resources who still need robust ML pipeline orchestration capabilities.

## Features

- Visual pipeline designer with drag-and-drop interface for building ML workflows
- Experiment tracking with metric visualization and comparison across pipeline runs
- Artifact lineage tracking showing data flow from raw datasets to trained models
- Pipeline versioning and rollback capabilities with Git-like workflow management
- S3-compatible artifact storage through MinIO for seamless cloud migration
- RESTful API for programmatic pipeline creation and execution using KFP SDK
- MySQL-backed metadata store for reliable pipeline state and execution history
- Component sharing and reuse across different pipeline implementations

## Use cases

- ML model training pipelines with automated data preprocessing and validation
- Batch inference workflows for processing large datasets on scheduled intervals
- A/B testing frameworks for comparing model performance across different algorithms
- Data pipeline orchestration for ETL processes in analytics workflows
- Model retraining automation triggered by data drift detection or performance degradation
- Multi-stage deployment pipelines with staging and production model validation
- Research experimentation environments for data science teams to track experiments

## Prerequisites

- Docker and Docker Compose installed with minimum 4GB RAM available for containers
- Python environment with Kubeflow Pipelines SDK (kfp) installed for pipeline development
- Basic understanding of machine learning workflows and pipeline concepts
- Ports 80, 8888, 9000, and 9001 available for UI, API server, and MinIO access
- Environment variables configured for MySQL credentials and MinIO access keys
- Familiarity with containerized ML workloads and artifact management practices

## docker-compose.yml

```yaml
services:
  mysql:
    image: mysql:8.0
    environment:
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
      - MYSQL_DATABASE=mlpipeline
      - MYSQL_USER=${MYSQL_USER}
      - MYSQL_PASSWORD=${MYSQL_PASSWORD}
    volumes:
      - mysql-data:/var/lib/mysql
    networks:
      - kubeflow-network
    restart: unless-stopped

  minio:
    image: minio/minio:latest
    command: server /data --console-address ":9001"
    environment:
      - MINIO_ROOT_USER=${MINIO_ACCESS_KEY}
      - MINIO_ROOT_PASSWORD=${MINIO_SECRET_KEY}
    volumes:
      - minio-data:/data
    ports:
      - "9000:9000"
      - "9001:9001"
    networks:
      - kubeflow-network
    restart: unless-stopped

  api-server:
    image: gcr.io/ml-pipeline/api-server:latest
    environment:
      - OBJECTSTORECONFIG_BUCKETNAME=mlpipeline
      - OBJECTSTORECONFIG_ACCESSKEY=${MINIO_ACCESS_KEY}
      - OBJECTSTORECONFIG_SECRETACCESSKEY=${MINIO_SECRET_KEY}
      - OBJECTSTORECONFIG_HOST=minio
      - OBJECTSTORECONFIG_PORT=9000
      - DBCONFIG_HOST=mysql
      - DBCONFIG_USER=${MYSQL_USER}
      - DBCONFIG_PASSWORD=${MYSQL_PASSWORD}
    ports:
      - "8888:8888"
    depends_on:
      - mysql
      - minio
    networks:
      - kubeflow-network
    restart: unless-stopped

  ui:
    image: gcr.io/ml-pipeline/frontend:latest
    environment:
      - ML_PIPELINE_SERVICE_HOST=api-server
      - ML_PIPELINE_SERVICE_PORT=8888
    ports:
      - "80:3000"
    depends_on:
      - api-server
    networks:
      - kubeflow-network
    restart: unless-stopped

volumes:
  mysql-data:
  minio-data:

networks:
  kubeflow-network:
    driver: bridge
```

## .env

```bash
# Kubeflow Pipelines
MYSQL_ROOT_PASSWORD=secure_root_password
MYSQL_USER=mlpipeline
MYSQL_PASSWORD=secure_mysql_password

# MinIO
MINIO_ACCESS_KEY=minioaccesskey
MINIO_SECRET_KEY=secure_minio_secret
```

## Usage notes

- Pipelines UI at http://localhost
- API server at http://localhost:8888
- MinIO console at http://localhost:9001
- Standalone mode without Kubernetes
- Use KFP SDK to create pipelines

## Troubleshooting

- Pipeline execution fails with 'ObjectStore connection error': Verify MINIO_ACCESS_KEY and MINIO_SECRET_KEY match between api-server and minio services
- UI shows 'Failed to fetch pipelines' error: Check that api-server container is running and accessible on port 8888
- MySQL connection refused during startup: Ensure mysql container has sufficient time to initialize before api-server starts, add healthcheck or sleep
- MinIO console inaccessible at localhost:9001: Verify MinIO container started with console-address parameter and port 9001 is not blocked by firewall
- Pipeline artifacts not persisting between restarts: Confirm minio-data and mysql-data volumes are properly mounted and have write permissions
- KFP SDK cannot connect to API server: Verify kfp.Client() points to http://localhost:8888 and api-server container network configuration

