# Apache Spark Standalone Cluster

> Apache Spark cluster with master and worker nodes for distributed data processing.

- Category: Database Stacks
- Difficulty: advanced
- Components: spark-master, spark-worker, spark-history
- Tags: spark, big-data, distributed, analytics, etl
- Source: https://docker.recipes/databases/apache-spark-standalone

## Overview

Apache Spark is an open-source unified analytics engine developed at UC Berkeley's AMPLab, designed for large-scale data processing with built-in modules for streaming, SQL, machine learning, and graph processing. Unlike traditional MapReduce frameworks, Spark performs in-memory computing, making it up to 100 times faster for certain applications by keeping data in RAM between operations rather than writing to disk after each step.

This standalone cluster deployment combines a Spark master node with multiple worker nodes to create a distributed computing environment capable of processing massive datasets across multiple machines. The master node coordinates job scheduling and resource allocation while worker nodes execute the actual data processing tasks, with the cluster automatically handling data partitioning, fault tolerance, and load balancing across available resources.

Data engineers, data scientists, and organizations processing large volumes of structured or unstructured data will benefit from this setup, particularly those running ETL pipelines, machine learning workflows, or real-time analytics. This configuration is ideal for companies transitioning from single-machine data processing to distributed computing, or those needing a development environment that mirrors production Spark clusters without the complexity of full cluster management platforms like YARN or Kubernetes.

## Features

- Resilient Distributed Datasets (RDDs) with automatic fault recovery and data lineage tracking
- In-memory computing capabilities reducing disk I/O by up to 100x compared to MapReduce
- Built-in Catalyst SQL optimizer for automatic query optimization and code generation
- Dynamic resource allocation allowing workers to scale compute resources based on workload demands
- Spark Streaming for processing live data streams with micro-batch processing architecture
- MLlib machine learning library with distributed algorithms for classification, regression, and clustering
- GraphX graph processing framework for social network analysis and graph algorithms
- Multi-language support with native APIs for Scala, Java, Python (PySpark), and R (SparkR)

## Use cases

- ETL pipeline processing for transforming terabytes of log data from multiple sources into analytical formats
- Real-time fraud detection systems analyzing credit card transactions and user behavior patterns
- Machine learning model training on large datasets for recommendation engines and predictive analytics
- Financial risk modeling and Monte Carlo simulations requiring distributed parallel processing
- Genomics research processing DNA sequencing data and performing large-scale bioinformatics analysis
- IoT sensor data aggregation and analysis for smart city infrastructure monitoring
- E-commerce clickstream analysis for customer journey mapping and conversion optimization

## Prerequisites

- Minimum 8GB RAM allocated to Docker with at least 4GB available for Spark worker processes
- Docker Engine 20.10+ and Docker Compose 2.0+ for proper container networking and resource management
- Understanding of distributed computing concepts including data partitioning and parallel processing
- Basic knowledge of Spark DataFrame/Dataset APIs and SQL for job submission and data manipulation
- Familiarity with JVM memory management and garbage collection tuning for optimal performance
- Network ports 7077 and 8080 available on host system for Spark master and web UI access

## docker-compose.yml

```yaml
services:
  spark-master:
    image: bitnami/spark:latest
    container_name: spark-master
    restart: unless-stopped
    environment:
      - SPARK_MODE=master
      - SPARK_MASTER_HOST=spark-master
      - SPARK_MASTER_PORT=${SPARK_MASTER_PORT:-7077}
      - SPARK_MASTER_WEBUI_PORT=${SPARK_WEBUI_PORT:-8080}
    ports:
      - "${SPARK_MASTER_PORT:-7077}:7077"
      - "${SPARK_WEBUI_PORT:-8080}:8080"
    volumes:
      - spark_data:/opt/bitnami/spark
    networks:
      - spark-network

  spark-worker-1:
    image: bitnami/spark:latest
    container_name: spark-worker-1
    restart: unless-stopped
    environment:
      - SPARK_MODE=worker
      - SPARK_MASTER_URL=spark://spark-master:7077
      - SPARK_WORKER_MEMORY=${SPARK_WORKER_MEMORY:-2g}
      - SPARK_WORKER_CORES=${SPARK_WORKER_CORES:-2}
    depends_on:
      - spark-master
    networks:
      - spark-network

  spark-worker-2:
    image: bitnami/spark:latest
    container_name: spark-worker-2
    restart: unless-stopped
    environment:
      - SPARK_MODE=worker
      - SPARK_MASTER_URL=spark://spark-master:7077
      - SPARK_WORKER_MEMORY=${SPARK_WORKER_MEMORY:-2g}
      - SPARK_WORKER_CORES=${SPARK_WORKER_CORES:-2}
    depends_on:
      - spark-master
    networks:
      - spark-network

volumes:
  spark_data:

networks:
  spark-network:
    driver: bridge
```

## .env

```bash
# Apache Spark Cluster
SPARK_MASTER_PORT=7077
SPARK_WEBUI_PORT=8080
SPARK_WORKER_MEMORY=2g
SPARK_WORKER_CORES=2
```

## Usage notes

- Spark Master UI at http://localhost:8080
- Submit jobs to spark://localhost:7077
- Scale workers by adding more services
- Great for big data processing

## Troubleshooting

- java.lang.OutOfMemoryError in worker containers: Increase SPARK_WORKER_MEMORY environment variable or reduce SPARK_WORKER_CORES to balance memory per core allocation
- Workers failing to connect to master with 'Connection refused' errors: Verify spark-master container is fully started before workers attempt connection, add health checks or startup delays
- Spark jobs hanging in RUNNING state indefinitely: Check executor memory settings and ensure sufficient resources are available, may need to increase driver memory or reduce executor instances
- ClassNotFoundException when submitting custom applications: Mount application JAR files to /opt/bitnami/spark/jars directory in all containers or use spark-submit with --jars parameter
- Web UI showing 'Application Not Found' errors: Ensure Spark History Server is configured with shared storage, or check that application completed successfully without exceptions
- Serialization errors with custom objects: Verify all custom classes implement Serializable interface and are available on classpath of all worker nodes

