# Apache Spark

> Distributed computing engine for big data processing.

- Category: AI & Machine Learning
- Difficulty: advanced
- Components: spark
- Tags: spark, big-data, distributed, analytics
- Source: https://docker.recipes/ai-ml/spark

## Overview

Apache Spark is an open-source unified analytics engine designed for large-scale data processing and distributed computing. Originally developed at UC Berkeley's AMPLab in 2009, Spark has become the de facto standard for big data processing, offering in-memory computing capabilities that can be up to 100x faster than traditional Hadoop MapReduce. Spark provides high-level APIs in Java, Scala, Python, and R, along with an optimized engine that supports general computation graphs for data analysis. This configuration deploys a complete Spark cluster using the master-worker architecture with Bitnami's optimized Spark containers. The setup includes one Spark master node responsible for cluster management, resource allocation, and job scheduling, paired with multiple worker nodes that execute the actual data processing tasks. The master node exposes both the cluster management interface and the Spark context endpoint, while workers automatically register with the master and contribute their configured CPU cores and memory resources to the cluster. This distributed setup enables horizontal scaling and fault tolerance for processing large datasets across multiple nodes. Data engineers, machine learning practitioners, and analytics teams working with big data workloads will find this stack invaluable for batch processing, real-time streaming analytics, and distributed machine learning tasks. The combination of containerized deployment with Spark's unified analytics engine makes it ideal for organizations needing to process terabytes of data, perform complex ETL operations, or train machine learning models at scale without the complexity of managing bare-metal Spark installations.

## Features

- Distributed in-memory computing with automatic data caching across worker nodes
- Unified analytics engine supporting SQL queries, streaming data, machine learning (MLlib), and graph processing (GraphX)
- Fault-tolerant RDD (Resilient Distributed Dataset) computation with automatic recovery from node failures
- Dynamic resource allocation with configurable worker memory (2GB) and CPU cores (2 cores) per worker
- Built-in Catalyst SQL optimizer for query performance optimization and code generation
- Support for multiple data sources including HDFS, S3, Cassandra, HBase, and structured streaming
- Web-based Spark UI for real-time monitoring of jobs, stages, tasks, and cluster resources
- Horizontal scaling through Docker Compose replicas with automatic worker registration to master

## Use cases

- Large-scale ETL pipelines processing terabytes of log data, customer records, or financial transactions
- Real-time streaming analytics for IoT sensor data, clickstream analysis, or fraud detection systems
- Distributed machine learning model training using MLlib for recommendation engines or predictive analytics
- Interactive data exploration and analytics using Spark SQL for business intelligence and reporting
- Graph processing and social network analysis using GraphX for relationship mapping and community detection
- Data lake processing for transforming raw data into structured formats for downstream analytics
- High-performance computing clusters for research institutions processing scientific datasets

## Prerequisites

- Minimum 8GB RAM (4GB+ for Spark master and workers, plus host OS overhead)
- Multi-core CPU (4+ cores recommended for meaningful parallel processing)
- Available ports 8080 (Spark Master UI) and 7077 (Spark cluster communication)
- Understanding of distributed computing concepts and Spark programming model (RDDs, DataFrames, Datasets)
- Familiarity with at least one Spark-supported language (Python/PySpark, Scala, Java, or R)
- Knowledge of data formats commonly used with Spark (Parquet, JSON, CSV, Avro)

## docker-compose.yml

```yaml
services:
  spark-master:
    image: bitnami/spark:latest
    container_name: spark-master
    environment:
      SPARK_MODE: master
      SPARK_MASTER_HOST: spark-master
    ports:
      - "8080:8080"
      - "7077:7077"
    networks:
      - spark

  spark-worker:
    image: bitnami/spark:latest
    environment:
      SPARK_MODE: worker
      SPARK_MASTER_URL: spark://spark-master:7077
      SPARK_WORKER_MEMORY: 2G
      SPARK_WORKER_CORES: 2
    deploy:
      replicas: 2
    depends_on:
      - spark-master
    networks:
      - spark

networks:
  spark:
    driver: bridge
```

## .env

```bash
# Adjust SPARK_WORKER_MEMORY and SPARK_WORKER_CORES
```

## Usage notes

- Docs: https://spark.apache.org/docs/latest/
- Master UI at http://localhost:8080 - cluster and job monitoring
- Submit jobs: spark-submit --master spark://localhost:7077 app.py
- Scale workers via deploy.replicas, adjust memory/cores per worker
- PySpark: SparkSession.builder.master('spark://localhost:7077').getOrCreate()
- Supports SQL, streaming, ML (MLlib), and graph processing

## Troubleshooting

- Workers not appearing in Master UI: Check network connectivity and ensure SPARK_MASTER_URL points to correct master hostname
- OutOfMemoryError during job execution: Increase SPARK_WORKER_MEMORY environment variable or reduce data partition sizes
- Jobs failing with 'Task not serializable' error: Ensure all functions and variables used in Spark transformations are serializable
- Slow performance on small datasets: Spark overhead makes it inefficient for small data; consider increasing data size or reducing parallelism
- Connection refused on port 7077: Verify spark-master container is running and port 7077 is properly exposed
- Executor lost errors during long-running jobs: Increase spark.network.timeout and spark.sql.adaptive.coalescePartitions.enabled for stability

