# Jupyter Data Science Lab

> Complete data science environment with JupyterLab, Spark, and databases.

- Category: Development Tools
- Difficulty: intermediate
- Components: jupyterlab, spark, postgresql, minio, mlflow
- Tags: jupyter, datascience, spark, python, ml
- Source: https://docker.recipes/development/jupyter-datascience

## Overview

JupyterLab is an interactive computing platform that revolutionized data science workflows through web-based notebooks combining live code, equations, visualizations, and narrative text. Born from Project Jupyter in 2014, it has become the de facto standard for data exploration, machine learning prototyping, and research reproducibility, supporting multiple programming languages through kernels while providing rich output capabilities for plots, tables, and HTML content. This comprehensive data science stack integrates JupyterLab with Apache Spark for distributed computing, PostgreSQL for robust data warehousing, MinIO for S3-compatible object storage, and MLflow for machine learning lifecycle management. The combination creates a complete analytics pipeline where data scientists can ingest raw data into MinIO, process it with Spark's distributed computing engine, store structured results in PostgreSQL, experiment within JupyterLab notebooks, and track ML experiments through MLflow's versioning system. This stack serves data science teams, ML engineers, and research organizations who need a self-hosted alternative to cloud platforms like Databricks or AWS SageMaker, providing enterprise-grade capabilities for experiment tracking, model versioning, and large-scale data processing while maintaining complete control over sensitive datasets and proprietary algorithms.

## Features

- Interactive JupyterLab notebooks with Python, R, and Scala kernels pre-configured for Spark integration
- Distributed Apache Spark processing with master-worker architecture for big data analytics
- PostgreSQL data warehouse with ACID compliance and JSON support for structured analytics
- S3-compatible MinIO object storage with erasure coding and versioning for data lake operations
- MLflow experiment tracking with automatic logging of parameters, metrics, and model artifacts
- Pre-built Spark connectivity in notebooks using the all-spark-notebook Docker image
- Integrated storage backend linking MLflow artifacts to MinIO and metadata to PostgreSQL
- Spark Web UI monitoring and cluster management through dedicated master node interface

## Use cases

- Machine learning model development with automated experiment tracking and version control
- Big data analytics processing terabyte-scale datasets across distributed Spark clusters
- Financial modeling and risk analysis requiring ACID-compliant transaction processing
- Research data pipelines combining structured PostgreSQL data with unstructured MinIO objects
- Data science education environments providing students with industry-standard toolchains
- Regulatory compliance scenarios requiring on-premises ML model governance and auditing
- Multi-team data science organizations needing centralized experiment management and collaboration

## Prerequisites

- Minimum 6GB RAM for full stack operation (2GB JupyterLab, 2GB Spark, 1GB PostgreSQL, 1GB MinIO/MLflow)
- Docker Engine 20.10+ and Docker Compose V2 for modern compose file syntax support
- Available ports 5000, 7077, 8081, 8888, 9000, 9001 for service web interfaces
- Basic understanding of Spark DataFrames and SQL for effective data processing workflows
- Familiarity with Python data science libraries (pandas, scikit-learn, matplotlib) for notebook development
- Knowledge of S3 API concepts for MinIO bucket and object management operations

## docker-compose.yml

```yaml
services:
  jupyterlab:
    image: jupyter/all-spark-notebook:latest
    ports:
      - "8888:8888"
    environment:
      - JUPYTER_TOKEN=${JUPYTER_TOKEN}
    volumes:
      - ./notebooks:/home/jovyan/work
      - jupyter_data:/home/jovyan/.local
    networks:
      - datascience_net

  spark-master:
    image: bitnami/spark:latest
    ports:
      - "7077:7077"
      - "8081:8080"
    environment:
      - SPARK_MODE=master
    networks:
      - datascience_net

  spark-worker:
    image: bitnami/spark:latest
    environment:
      - SPARK_MODE=worker
      - SPARK_MASTER_URL=spark://spark-master:7077
    depends_on:
      - spark-master
    networks:
      - datascience_net

  postgres:
    image: postgres:15-alpine
    environment:
      - POSTGRES_USER=datascience
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      - POSTGRES_DB=datawarehouse
    volumes:
      - postgres_data:/var/lib/postgresql/data
    networks:
      - datascience_net

  minio:
    image: minio/minio:latest
    command: server /data --console-address ":9001"
    ports:
      - "9000:9000"
      - "9001:9001"
    environment:
      - MINIO_ROOT_USER=${MINIO_USER}
      - MINIO_ROOT_PASSWORD=${MINIO_PASSWORD}
    volumes:
      - minio_data:/data
    networks:
      - datascience_net

  mlflow:
    image: ghcr.io/mlflow/mlflow:latest
    ports:
      - "5000:5000"
    environment:
      - MLFLOW_S3_ENDPOINT_URL=http://minio:9000
      - AWS_ACCESS_KEY_ID=${MINIO_USER}
      - AWS_SECRET_ACCESS_KEY=${MINIO_PASSWORD}
    command: mlflow server --host 0.0.0.0 --backend-store-uri postgresql://datascience:${POSTGRES_PASSWORD}@postgres:5432/datawarehouse
    depends_on:
      - postgres
      - minio
    networks:
      - datascience_net

volumes:
  jupyter_data:
  postgres_data:
  minio_data:

networks:
  datascience_net:
```

## .env

```bash
# Data Science Lab
JUPYTER_TOKEN=secure_jupyter_token
POSTGRES_PASSWORD=secure_postgres_password
MINIO_USER=minioadmin
MINIO_PASSWORD=secure_minio_password

# JupyterLab at http://localhost:8888
# Spark UI at http://localhost:8081
# MLflow at http://localhost:5000
```

## Usage notes

- JupyterLab at http://localhost:8888
- Spark Master UI at http://localhost:8081
- MLflow tracking at http://localhost:5000
- MinIO for S3-compatible storage
- All spark notebooks included

## Troubleshooting

- JupyterLab shows 'Kernel not ready' for Spark: Increase Docker memory allocation and verify spark-master container is running
- MLflow experiments not saving: Check PostgreSQL connection string format and ensure database 'datawarehouse' exists
- Spark jobs failing with memory errors: Reduce spark.executor.memory in notebook configuration or add more worker containers
- MinIO console inaccessible: Verify MINIO_ROOT_USER and MINIO_ROOT_PASSWORD environment variables are set correctly
- PostgreSQL connection refused: Wait 30-60 seconds after startup for database initialization to complete
- MLflow artifacts not uploading to MinIO: Confirm AWS_ACCESS_KEY_ID matches MINIO_ROOT_USER in environment variables

