Apache Spark on ARC: Difference between revisions

From RCSWiki
Jump to navigation Jump to search
(Created page with "= Overview = This guide gives an overview of running Apache Spark clusters under the existing scheduling system of the ARC cluster at the University of Calgary. When activat...")
 
(Show ARC navbox)
 
(13 intermediate revisions by 4 users not shown)
Line 3: Line 3:
This guide gives an overview of running Apache Spark clusters under the existing scheduling system of the ARC cluster at the University of Calgary.
This guide gives an overview of running Apache Spark clusters under the existing scheduling system of the ARC cluster at the University of Calgary.


When activated by a Spark module, a python module is added to the path which allows provisioning of an Apache Spark cluster as a job on a Slurm cluster. This class is instantiated right inside of the Python code and submits a job to the cluster to gather resources during execution. Although this "Driver" process can be run on the login node for smaller examples, using a node in the "single" partition is recommended (salloc -p single -N 1 --ntasks-per-node=8 --mem=0).
If you want to use Spark with Jupyter via the OOD (Open On Demand) system skip to the section titled "Jupyter Notebook".
 
When activated by a Spark module, a python module is added to the path which allows provisioning of an Apache Spark cluster as a job on a Slurm cluster. This class is instantiated right inside of the Python code and submits a job to the cluster to gather resources during execution. Although this "Driver" process can be run on the login node for smaller examples, using a node in the "single" partition is recommended.
 
How to get a node in the '''single''' partition on ARC for '''3 hours''':
$ salloc -p single -N 1 -n 8 -c 1 --mem=0 -t 3:00:00
The first time that you start up spark on ARC, you may need to install some additional packages locally, depending on which libraries you are including. At a minimum, graphframes:graphframes will be needed to use basic spark data structures. This can be obtained by running the following command from the command line in an interactive job:
$ pyspark --packages=graphframes:graphframes:0.3.0-spark2.0-s_2.11 --repositories=<nowiki>https://repos.spark-packages.org</nowiki>
You may need to install multiple packages to resolve all of the missing 3rd party spark modules, depending on what you are doing. You can also correct the repositories list by modifying the PYSPARK_SUBMIT_ARGS environment variable to include the relevant string. After the first time that you run this, the jar files and configuration files are downloaded to your ~/.ivy2 directory, and you shouldn't have to pass these arguments again in the future.


= Procedure =
= Procedure =
== Jupyter Notebook ==
== Jupyter Notebook ==


# Point your browser to jupyter.ucalgary.ca and login using your IT username and password
# Point your browser to ood-arc.rcs.ucalgary.ca and login using the IT portal.  If you have recently signed in to your email this may not even happen.
# On the Home screen click the "Jupyter + Spark Cluster" App
# Fill out the form presented with the number of nodes/workers per node/memory per cpu.  These define the size of your Spark cluster.
# If you like jupyterlab go ahead and check the box to use it.
# Create a new Python 3 notebook
# Create a new Python 3 notebook
# Paste the below code block into a cell in your new notebook
# Paste the below code block into a cell in your new notebook
Line 17: Line 27:


# Login to arc.ucalgary.ca using your IT username and password
# Login to arc.ucalgary.ca using your IT username and password
# Load the spark module with "module load spark/2.2.0" BEFORE starting your Python interpreter.
# Load the spark module with "module load spark/jupyterhub" BEFORE starting your Python interpreter.
# For interactive work, simply start your preferred Python. Examples: Jupyter Notebook, ipython, python.
# For interactive work, simply start your preferred Python. Examples: Jupyter Notebook, ipython, python.


Line 23: Line 33:


In your Python file or terminal load the appropriate python modules and instantiate the cluster:
In your Python file or terminal load the appropriate python modules and instantiate the cluster:
<pre>
<syntaxhighlight lang="python">
 
import os
import os
import atexit
import atexit
import sys
import sys
import re


import pyspark
import pyspark
from pyspark.conf import SparkConf
from pyspark.context import SparkContext
from pyspark.context import SparkContext
from pyspark.sql import SQLContext
from pyspark.sql import SQLContext
import findspark
conflines=[tuple(a.rstrip().split(" ")) for a in open(os.environ['SPARK_CONFIG_FILE']).readlines()]
from sparkhpc import sparkjob
conf=SparkConf()
conf.setAll(conflines)
conf.setMaster("spark://%s:%s"% (os.environ['SPARK_MASTER_HOST'],os.environ['SPARK_MASTER_PORT']))
sc=pyspark.SparkContext(conf=conf)


#Exit handler to clean up the Spark cluster if the script exits or crashes
#You need this line if you want to use SparkSQL
def exitHandler(sj,sc):
sqlCtx=SQLContext(sc)
    try:
        print('Trapped Exit cleaning up Spark Context')
        sc.stop()
    except:
        pass
    try:
        print('Trapped Exit cleaning up Spark Job')
        sj.stop()
    except:
        pass


findspark.init()
#YOUR CODE GOES HERE


#Parameters for the Spark cluster
</syntaxhighlight>
nodes=3
tasks_per_node=8
memory_per_task=1024 #1 gig per process, adjust accordingly
# Please estimate walltime carefully to keep unused Spark clusters from sitting
# idle so that others may use the resources.
walltime="1:00" #1 hour
os.environ['SBATCH_PARTITION']='lattice' #Set the appropriate ARC partition


sj = sparkjob.sparkjob(
You now have a sc (Spark Context) and sqlCtx (SQL Context) objects to operate on. Please remember to return to the OOD screen and terminate the Jupyter + Spark app when you are finished.
    ncores=nodes*tasks_per_node,
    cores_per_executor=tasks_per_node,
    memory_per_core=memory_per_task,
    walltime=walltime
    )
 
sj.wait_to_start()
sc = sj.start_spark()
 
#Register the exit handler                                                                                                   
atexit.register(exitHandler,sj,sc)
 
#You need this line if you want to use SparkSQL
sqlCtx=SQLContext(sc)
</pre>
 
You now have a sc (Spark Context) and sqlCtx (SQL Context) objects to operate on. Please remember to cal sc.stop() and sj.stop() when you are finished.


There are many Spark tutorials out there.  Here are some good places to look:
There are many Spark tutorials out there.  Here are some good places to look:
Line 84: Line 66:


'''HINT''': It helps to google "pyspark" as that returns Python results instead of Scala which is another common language used to interact with Spark.
'''HINT''': It helps to google "pyspark" as that returns Python results instead of Scala which is another common language used to interact with Spark.
[[Category:Software]]
[[Category:ARC]]
{{Navbox ARC}}

Latest revision as of 18:11, 21 September 2023

Overview

This guide gives an overview of running Apache Spark clusters under the existing scheduling system of the ARC cluster at the University of Calgary.

If you want to use Spark with Jupyter via the OOD (Open On Demand) system skip to the section titled "Jupyter Notebook".

When activated by a Spark module, a python module is added to the path which allows provisioning of an Apache Spark cluster as a job on a Slurm cluster. This class is instantiated right inside of the Python code and submits a job to the cluster to gather resources during execution. Although this "Driver" process can be run on the login node for smaller examples, using a node in the "single" partition is recommended.

How to get a node in the single partition on ARC for 3 hours:

$ salloc -p single -N 1 -n 8 -c 1 --mem=0 -t 3:00:00

The first time that you start up spark on ARC, you may need to install some additional packages locally, depending on which libraries you are including. At a minimum, graphframes:graphframes will be needed to use basic spark data structures. This can be obtained by running the following command from the command line in an interactive job:

$ pyspark --packages=graphframes:graphframes:0.3.0-spark2.0-s_2.11 --repositories=https://repos.spark-packages.org

You may need to install multiple packages to resolve all of the missing 3rd party spark modules, depending on what you are doing. You can also correct the repositories list by modifying the PYSPARK_SUBMIT_ARGS environment variable to include the relevant string. After the first time that you run this, the jar files and configuration files are downloaded to your ~/.ivy2 directory, and you shouldn't have to pass these arguments again in the future.

Procedure

Jupyter Notebook

  1. Point your browser to ood-arc.rcs.ucalgary.ca and login using the IT portal. If you have recently signed in to your email this may not even happen.
  2. On the Home screen click the "Jupyter + Spark Cluster" App
  3. Fill out the form presented with the number of nodes/workers per node/memory per cpu. These define the size of your Spark cluster.
  4. If you like jupyterlab go ahead and check the box to use it.
  5. Create a new Python 3 notebook
  6. Paste the below code block into a cell in your new notebook
  7. Once you run the cell you will have an sc object in your environment. This is the "Spark Context". Executing methods on this object allows you to interact with the spark cluster you have just created.

From the Command Line

  1. Login to arc.ucalgary.ca using your IT username and password
  2. Load the spark module with "module load spark/jupyterhub" BEFORE starting your Python interpreter.
  3. For interactive work, simply start your preferred Python. Examples: Jupyter Notebook, ipython, python.

Instantiate Spark Cluster

In your Python file or terminal load the appropriate python modules and instantiate the cluster:

import os
import atexit
import sys
import re

import pyspark
from pyspark.conf import SparkConf
from pyspark.context import SparkContext
from pyspark.sql import SQLContext
conflines=[tuple(a.rstrip().split(" ")) for a in open(os.environ['SPARK_CONFIG_FILE']).readlines()]
conf=SparkConf()
conf.setAll(conflines)
conf.setMaster("spark://%s:%s"% (os.environ['SPARK_MASTER_HOST'],os.environ['SPARK_MASTER_PORT']))
sc=pyspark.SparkContext(conf=conf)

#You need this line if you want to use SparkSQL
sqlCtx=SQLContext(sc)

#YOUR CODE GOES HERE

You now have a sc (Spark Context) and sqlCtx (SQL Context) objects to operate on. Please remember to return to the OOD screen and terminate the Jupyter + Spark app when you are finished.

There are many Spark tutorials out there. Here are some good places to look:

HINT: It helps to google "pyspark" as that returns Python results instead of Scala which is another common language used to interact with Spark.