Apache Spark on ARC

Overview

This guide gives an overview of running Apache Spark clusters under the existing scheduling system of the ARC cluster at the University of Calgary.

If you want to use Spark with Jupyter via the OOD (Open On Demand) system skip to the section titled "Jupyter Notebook".

When activated by a Spark module, a python module is added to the path which allows provisioning of an Apache Spark cluster as a job on a Slurm cluster. This class is instantiated right inside of the Python code and submits a job to the cluster to gather resources during execution. Although this "Driver" process can be run on the login node for smaller examples, using a node in the "single" partition is recommended.

How to get a node in the single partition on ARC for 3 hours:

$ salloc -p single -N 1 -n 8 -c 1 --mem=0 -t 3:00:00

The first time that you start up spark on ARC, you may need to install some additional packages locally, depending on which libraries you are including. At a minimum, graphframes:graphframes will be needed to use basic spark data structures. This can be obtained by running the following command from the command line in an interactive job:

$ pyspark --packages=graphframes:graphframes:0.3.0-spark2.0-s_2.11 --repositories=https://repos.spark-packages.org

You may need to install multiple packages to resolve all of the missing 3rd party spark modules, depending on what you are doing. You can also correct the repositories list by modifying the PYSPARK_SUBMIT_ARGS environment variable to include the relevant string. After the first time that you run this, the jar files and configuration files are downloaded to your ~/.ivy2 directory, and you shouldn't have to pass these arguments again in the future.

Procedure

Jupyter Notebook

1. Point your browser to ood-arc.rcs.ucalgary.ca and login using the IT portal. If you have recently signed in to your email this may not even happen.
2. On the Home screen click the "Jupyter + Spark Cluster" App
3. Fill out the form presented with the number of nodes/workers per node/memory per cpu. These define the size of your Spark cluster.
4. If you like jupyterlab go ahead and check the box to use it.
5. Create a new Python 3 notebook
6. Paste the below code block into a cell in your new notebook
7. Once you run the cell you will have an sc object in your environment. This is the "Spark Context". Executing methods on this object allows you to interact with the spark cluster you have just created.

From the Command Line

1. Login to arc.ucalgary.ca using your IT username and password
2. Load the spark module with "module load spark/jupyterhub" BEFORE starting your Python interpreter.
3. For interactive work, simply start your preferred Python. Examples: Jupyter Notebook, ipython, python.

Instantiate Spark Cluster

In your Python file or terminal load the appropriate python modules and instantiate the cluster:

import os
import atexit
import sys
import re

import pyspark
from pyspark.conf import SparkConf
from pyspark.context import SparkContext
from pyspark.sql import SQLContext
conflines=[tuple(a.rstrip().split(" ")) for a in open(os.environ['SPARK_CONFIG_FILE']).readlines()]
conf=SparkConf()
conf.setAll(conflines)
conf.setMaster("spark://%s:%s"% (os.environ['SPARK_MASTER_HOST'],os.environ['SPARK_MASTER_PORT']))
sc=pyspark.SparkContext(conf=conf)

#You need this line if you want to use SparkSQL
sqlCtx=SQLContext(sc)

#YOUR CODE GOES HERE

You now have a sc (Spark Context) and sqlCtx (SQL Context) objects to operate on. Please remember to return to the OOD screen and terminate the Jupyter + Spark app when you are finished.

There are many Spark tutorials out there. Here are some good places to look:

• https://spark.apache.org/docs/latest/quick-start.html
• https://spark.apache.org/docs/latest/rdd-programming-guide.html
• https://spark.apache.org/docs/latest/sql-programming-guide.html

HINT: It helps to google "pyspark" as that returns Python results instead of Scala which is another common language used to interact with Spark.