Apache Spark on ARC - Revision history

Lleung: Show ARC navbox

2023-09-21T18:11:12Z

Show ARC navbox

← Older revision		Revision as of 18:11, 21 September 2023
Line 67:		Line 67:
	'''HINT''': It helps to google "pyspark" as that returns Python results instead of Scala which is another common language used to interact with Spark.		'''HINT''': It helps to google "pyspark" as that returns Python results instead of Scala which is another common language used to interact with Spark.

			[[Category:Software]]
	[[Category:ARC]]		[[Category:ARC]]
	~~[[Category:Software]]~~		{{Navbox ARC}}

	{{Navbox ~~Software~~}}

Lleung: Added navbox

2023-09-20T22:25:53Z

Added navbox

← Older revision		Revision as of 22:25, 20 September 2023
Line 69:		Line 69:
	[[Category:ARC]]		[[Category:ARC]]
	[[Category:Software]]		[[Category:Software]]

			{{Navbox Software}}

Dschulz: /* Procedure */

2023-01-23T19:52:27Z

Procedure

← Older revision		Revision as of 19:52, 23 January 2023
Line 14:		Line 14:

	= Procedure =		= Procedure =
	~~'''Note:'''The jupyter.ucalgary.ca system is being decommissioned and will soon redirect to the new system.~~

	== Jupyter Notebook ==		== Jupyter Notebook ==

Dschulz at 20:30, 8 March 2022

2022-03-08T20:30:24Z

← Older revision		Revision as of 20:30, 8 March 2022
Line 2:		Line 2:

	This guide gives an overview of running Apache Spark clusters under the existing scheduling system of the ARC cluster at the University of Calgary.		This guide gives an overview of running Apache Spark clusters under the existing scheduling system of the ARC cluster at the University of Calgary.

			If you want to use Spark with Jupyter via the OOD (Open On Demand) system skip to the section titled "Jupyter Notebook".

	When activated by a Spark module, a python module is added to the path which allows provisioning of an Apache Spark cluster as a job on a Slurm cluster. This class is instantiated right inside of the Python code and submits a job to the cluster to gather resources during execution. Although this "Driver" process can be run on the login node for smaller examples, using a node in the "single" partition is recommended.		When activated by a Spark module, a python module is added to the path which allows provisioning of an Apache Spark cluster as a job on a Slurm cluster. This class is instantiated right inside of the Python code and submits a job to the cluster to gather resources during execution. Although this "Driver" process can be run on the login node for smaller examples, using a node in the "single" partition is recommended.

Dschulz: /* Instantiate Spark Cluster */

2022-03-08T20:29:14Z

Instantiate Spark Cluster

← Older revision		Revision as of 20:29, 8 March 2022
Line 33:		Line 33:

	In your Python file or terminal load the appropriate python modules and instantiate the cluster:		In your Python file or terminal load the appropriate python modules and instantiate the cluster:
	<syntaxhighlight lang=python>		<syntaxhighlight lang="python">

	import os		import os
Line 49:		Line 49:
	conf.setMaster("spark://%s:%s"% (os.environ['SPARK_MASTER_HOST'],os.environ['SPARK_MASTER_PORT']))		conf.setMaster("spark://%s:%s"% (os.environ['SPARK_MASTER_HOST'],os.environ['SPARK_MASTER_PORT']))
	sc=pyspark.SparkContext(conf=conf)		sc=pyspark.SparkContext(conf=conf)

			#You need this line if you want to use SparkSQL
			sqlCtx=SQLContext(sc)

			#YOUR CODE GOES HERE

	</syntaxhighlight>		</syntaxhighlight>

Dschulz: /* Procedure */

2022-03-08T20:26:44Z

Procedure

← Older revision		Revision as of 20:26, 8 March 2022
Line 12:		Line 12:

	= Procedure =		= Procedure =
			'''Note:'''The jupyter.ucalgary.ca system is being decommissioned and will soon redirect to the new system.

	== Jupyter Notebook ==		== Jupyter Notebook ==

	# Point your browser to ~~jupyter~~.ucalgary.ca and login using your ~~IT username~~ and ~~password~~		# Point your browser to ood-arc.rcs.ucalgary.ca and login using the IT portal. If you have recently signed in to your email this may not even happen.
			# On the Home screen click the "Jupyter + Spark Cluster" App
			# Fill out the form presented with the number of nodes/workers per node/memory per cpu. These define the size of your Spark cluster.
			# If you like jupyterlab go ahead and check the box to use it.
	# Create a new Python 3 notebook		# Create a new Python 3 notebook
	# Paste the below code block into a cell in your new notebook		# Paste the below code block into a cell in your new notebook
Line 30:		Line 34:
	In your Python file or terminal load the appropriate python modules and instantiate the cluster:		In your Python file or terminal load the appropriate python modules and instantiate the cluster:
	<syntaxhighlight lang=python>		<syntaxhighlight lang=python>

	import os		import os
	import atexit		import atexit
	import sys		import sys
	import ~~time~~		import re

	import pyspark		import pyspark
			from pyspark.conf import SparkConf
	from pyspark.context import SparkContext		from pyspark.context import SparkContext
	from pyspark.sql import SQLContext		from pyspark.sql import SQLContext
	~~import findspark~~		conflines=[tuple(a.rstrip().split(" ")) for a in open(os.environ['SPARK_CONFIG_FILE']).readlines()]
	~~from sparkhpc import sparkjob~~		conf=SparkConf()
			conf.setAll(conflines)
	~~#Exit handler to clean up the Spark cluster if the script exits or crashes~~		conf.setMaster("spark://%s:%s"% (os.environ['SPARK_MASTER_HOST'],os.environ['SPARK_MASTER_PORT']))
	~~def exitHandler~~(~~sj,sc):~~		sc=pyspark.SparkContext(conf=conf)
	~~try:~~
	~~print~~(~~'Trapped Exit cleaning up Spark Context'~~)
	sc.~~stop~~()
	~~except:~~
	~~pass~~
	~~try:~~
	~~print~~('~~Trapped Exit cleaning up Spark Job~~')
	sj.~~stop~~()
	~~except:~~
	~~pass~~

	~~findspark~~.~~init~~()

	~~#Parameters for the Spark cluster~~
	~~nodes=1~~
	~~tasks_per_node=24~~
	~~memory_per_task=4096 #4 gig per process, adjust accordingly~~
	~~# Please estimate walltime carefully to keep unused Spark clusters from sitting~~
	~~# idle so that others may use the resources~~.
	~~walltime=~~"3:00" ~~#3 hours~~
	#os.environ['~~SBATCH_PARTITION~~']='~~cpu2019~~' ~~#Set the appropriate ARC partition~~

	~~sj = sparkjob.sparkjob(~~
	~~ncores=nodes*tasks_per_node,~~
	~~cores_per_executor=tasks_per_node,~~
	~~memory_per_core=memory_per_task,~~
	~~walltime=walltime~~
	)

	~~sj.wait_to_start()~~
	~~time.sleep(60~~)
	sc = sj.~~start_spark~~()

	~~#Register the exit handler~~
	~~atexit.register(exitHandler,sj,sc~~)

	~~#You need this line if you want to use SparkSQL~~
	~~sqlCtx=SQLContext(sc)~~
	</syntaxhighlight>		</syntaxhighlight>

	You now have a sc (Spark Context) and sqlCtx (SQL Context) objects to operate on. Please remember to ~~cal sc.stop()~~ and ~~sj.stop()~~ when you are finished.		You now have a sc (Spark Context) and sqlCtx (SQL Context) objects to operate on. Please remember to return to the OOD screen and terminate the Jupyter + Spark app when you are finished.

	There are many Spark tutorials out there. Here are some good places to look:		There are many Spark tutorials out there. Here are some good places to look:

Ian.percel: /* Overview */

2021-10-27T22:29:19Z

Overview

← Older revision		Revision as of 22:29, 27 October 2021
Line 7:		Line 7:
	How to get a node in the '''single''' partition on ARC for '''3 hours''':		How to get a node in the '''single''' partition on ARC for '''3 hours''':
	$ salloc -p single -N 1 -n 8 -c 1 --mem=0 -t 3:00:00		$ salloc -p single -N 1 -n 8 -c 1 --mem=0 -t 3:00:00
			The first time that you start up spark on ARC, you may need to install some additional packages locally, depending on which libraries you are including. At a minimum, graphframes:graphframes will be needed to use basic spark data structures. This can be obtained by running the following command from the command line in an interactive job:
			$ pyspark --packages=graphframes:graphframes:0.3.0-spark2.0-s_2.11 --repositories=<nowiki>https://repos.spark-packages.org</nowiki>
			You may need to install multiple packages to resolve all of the missing 3rd party spark modules, depending on what you are doing. You can also correct the repositories list by modifying the PYSPARK_SUBMIT_ARGS environment variable to include the relevant string. After the first time that you run this, the jar files and configuration files are downloaded to your ~/.ivy2 directory, and you shouldn't have to pass these arguments again in the future.

	= Procedure =		= Procedure =

Ian.percel: /* From the Command Line */

2021-10-18T21:23:50Z

From the Command Line

← Older revision		Revision as of 21:23, 18 October 2021
Line 20:		Line 20:

	# Login to arc.ucalgary.ca using your IT username and password		# Login to arc.ucalgary.ca using your IT username and password
	# Load the spark module with "module load spark/~~2.2.0~~" BEFORE starting your Python interpreter.		# Load the spark module with "module load spark/jupyterhub" BEFORE starting your Python interpreter.
	# For interactive work, simply start your preferred Python. Examples: Jupyter Notebook, ipython, python.		# For interactive work, simply start your preferred Python. Examples: Jupyter Notebook, ipython, python.

Ian.percel: /* Instantiate Spark Cluster */

2021-03-18T15:03:40Z

Instantiate Spark Cluster

← Older revision		Revision as of 15:03, 18 March 2021
Line 30:		Line 30:
	import atexit		import atexit
	import sys		import sys
			import time

	import pyspark		import pyspark

Ian.percel: insert sleep/* Instantiate Spark Cluster */

2021-03-18T15:03:18Z

insert sleepInstantiate Spark Cluster

← Older revision		Revision as of 15:03, 18 March 2021
Line 69:		Line 69:

	sj.wait_to_start()		sj.wait_to_start()
			time.sleep(60)
	sc = sj.start_spark()		sc = sj.start_spark()