Apache Spark on Marc - Revision history

Lleung: Added navbox

2023-09-20T22:26:05Z

Added navbox

← Older revision		Revision as of 22:26, 20 September 2023
Line 99:		Line 99:
	[[Category:Marc]]		[[Category:Marc]]
	[[Category:Software]]		[[Category:Software]]

			{{Navbox Software}}

Dschulz at 19:36, 29 June 2022

2022-06-29T19:36:34Z

← Older revision		Revision as of 19:36, 29 June 2022
Line 87:		Line 87:
	</syntaxhighlight>		</syntaxhighlight>

	You now have a sc (Spark Context) and sqlCtx (SQL Context) objects to operate on. Please remember to return to the ~~OOD~~ screen and terminate the Jupyter + Spark ~~app~~ when you are finished.		You now have a sc (Spark Context) and sqlCtx (SQL Context) objects to operate on. Please remember to return to the Putty window screen and terminate the Jupyter + Spark job when you are finished or the resources will stay allocated to you until the timelimit runs out which will prevent others from accessing the limited resources on Marc.

	There are many Spark tutorials out there. Here are some good places to look:		There are many Spark tutorials out there. Here are some good places to look:

Dschulz at 19:35, 29 June 2022

2022-06-29T19:35:32Z

← Older revision		Revision as of 19:35, 29 June 2022
Line 54:		Line 54:
	* Click Apply		* Click Apply
	* Return to the myappmf tab in your browser and start a Google Chrome there. '''Note:''' It is important that you use the Chrome from myappmf not your desktop.		* Return to the myappmf tab in your browser and start a Google Chrome there. '''Note:''' It is important that you use the Chrome from myappmf not your desktop.
	* Paste the link containing http://localhost:~~8888~~?token= found in the putty terminal window into the browser started from myappmf and you should be rewarded with a Jupyter Notebook session.		* Paste the link containing http://localhost:...?token= found in the putty terminal window into the browser started from myappmf and you should be rewarded with a Jupyter Notebook session.
	==Troubleshooting Forwarding==		==Troubleshooting Forwarding==
	* There are a number of things that can go wrong with the port forwarding above.		* There are a number of things that can go wrong with the port forwarding above.
	'''Problem:''' The job running the Jupyter notebook and Spark cluster can run out of time in which case you'll have to start a new Jupyter/Spark job.<BR>		'''Problem:''' The job running the Jupyter notebook and Spark cluster can run out of time in which case you'll have to start a new Jupyter/Spark job.<BR>
	'''Solution:''' Start a new on and follow the solution to the next problem to "clean up"<BR>		'''Solution:''' Start a new on and follow the solution to the next problem to "clean up"<BR>
	~~<BR>~~
	~~'''Problem:''' The port forwarding from a previous job is still active.<BR>~~
	~~'''Solution:''' Go through every Putty window that you have open and make sure the forwarded ports box is clear in every running Putty window.<BR>~~
	~~<BR>~~
	~~'''Problem:''' Someone else already has port 8888 in use on the machine that runs Putty. <BR>~~
	'''Solution:''' Try 8889 (or some other number above 1024 and below 65535). You'll haved to set Source port in the putty forward to this new port number and edit the URL that gives you the Jupyter notebook (change the :8888 to :8889 etc.)<BR>
	~~<BR>~~

	In your Python file or terminal load the appropriate python modules and instantiate the cluster:		In your Python file or terminal load the appropriate python modules and instantiate the cluster:

Dschulz at 19:33, 29 June 2022

2022-06-29T19:33:35Z

← Older revision		Revision as of 19:33, 29 June 2022
Line 41:		Line 41:
	</syntaxhighlight>		</syntaxhighlight>

	* This message says that the Jupyter server is running on node sg5 on port ~~54674~~. Port ~~8888~~ needs to be forwarded from the server running Putty to the cluster.		* This message says that the Jupyter server is running on node sg5 on port 13452. Port 13452 needs to be forwarded from the server running Putty to the cluster.
	* Click the Putty icon in the upper left corner of the Putty window and select "Change Settings"		* Click the Putty icon in the upper left corner of the Putty window and select "Change Settings"
	[[File:ChangeSettings.png\|alt=Image showing where the "Change Settings" option for Putty is located\|none\|thumb\|Image showing where the "Change Settings" option for Putty is located]]		[[File:ChangeSettings.png\|alt=Image showing where the "Change Settings" option for Putty is located\|none\|thumb\|Image showing where the "Change Settings" option for Putty is located]]
	* If there's already ~~8888~~ listed in the Forwarded ports box click it and click Remove -- the Forwarded ports box should be empty.		* If there's already 13452 listed in the Forwarded ports box click it and click Remove -- the Forwarded ports box should be empty.
	* In the Change Settings dialogue box pick "Tunnels" in the left hand box (you'll have to open the Various levels of the tree to get there)		* In the Change Settings dialogue box pick "Tunnels" in the left hand box (you'll have to open the Various levels of the tree to get there)
	* ~~Enter 8888~~ into the Source port box[[File:SettingsDialogueBox.png\|alt=Putty Settings dialogue box showing the required port forwarding.\|none\|thumb\|Putty Settings dialogue box showing the required port forwarding.]]		* Copy the circled source port "13452" into the Source port box in Putty
	* ~~Copy the circled value from the putty window into the Destination box and click~~ the Add button.		* Copy the circled value from the putty window into the Destination box
	* The "Forwarded ports" box should now contain a concatenation of the text in the source port and destination boxes. i.e. "~~8888~~:sg4:~~54674~~"		[[File:SettingsDialogueBox.png\|alt=Putty Settings dialogue box showing the required port forwarding.\|none\|thumb\|Putty Settings dialogue box showing the required port forwarding.]]
			* Click the Add button. (Forgetting to do this is the biggest cause of problems)
			* The "Forwarded ports" box should now contain a concatenation of the text in the source port and destination boxes. i.e. "13452:sg4:13452"

	* Click Apply		* Click Apply

Dschulz at 19:27, 29 June 2022

2022-06-29T19:27:41Z

← Older revision		Revision as of 19:27, 29 June 2022
Line 17:		Line 17:
	$ cat slurm-14790.out		$ cat slurm-14790.out
	<snip -- lots of logging messages that can be ignored unless there's problems>		<snip -- lots of logging messages that can be ignored unless there's problems>
	Discovered Jupyter Notebook server listening on port ~~54674~~!		Discovered Jupyter Notebook server listening on port 13452!
	===========================================================================		===========================================================================
	===========================================================================		===========================================================================
	===========================================================================		===========================================================================
	Your Jupyter Notebook with Spark cluster is now active. Inside Putty		Your Jupyter Notebook with Spark cluster is now active. Inside Putty
	You will now need to forward ~~port 8888 to~~:		You will now need to forward:

	~~sg5~~:~~54674~~		Source port: 13452
			Destination: sg6:13452

	Then you can point your browser started from myappmf.ucalgary.ca to		Then you can point your browser started from myappmf.ucalgary.ca to
Line 30:		Line 31:

	Currently		Currently
	http://localhost:~~8888~~/?token=~~dccb02273b9be90c833edf658fd3183aa62fa9fabadce6d8~~		http://localhost:13452/?token=6b1bc28386204207f67d6ae9d3fb97de455fa105b9b8e242

	Please see the RCS website rcs.ucalgary.ca for information on how to		Please see the RCS website rcs.ucalgary.ca for information on how to

Dschulz at 18:02, 29 June 2022

2022-06-29T18:02:45Z

← Older revision		Revision as of 18:02, 29 June 2022
Line 52:		Line 52:
	* Return to the myappmf tab in your browser and start a Google Chrome there. '''Note:''' It is important that you use the Chrome from myappmf not your desktop.		* Return to the myappmf tab in your browser and start a Google Chrome there. '''Note:''' It is important that you use the Chrome from myappmf not your desktop.
	* Paste the link containing http://localhost:8888?token= found in the putty terminal window into the browser started from myappmf and you should be rewarded with a Jupyter Notebook session.		* Paste the link containing http://localhost:8888?token= found in the putty terminal window into the browser started from myappmf and you should be rewarded with a Jupyter Notebook session.
			==Troubleshooting Forwarding==
			* There are a number of things that can go wrong with the port forwarding above.
			'''Problem:''' The job running the Jupyter notebook and Spark cluster can run out of time in which case you'll have to start a new Jupyter/Spark job.<BR>
			'''Solution:''' Start a new on and follow the solution to the next problem to "clean up"<BR>
			<BR>
			'''Problem:''' The port forwarding from a previous job is still active.<BR>
			'''Solution:''' Go through every Putty window that you have open and make sure the forwarded ports box is clear in every running Putty window.<BR>
			<BR>
			'''Problem:''' Someone else already has port 8888 in use on the machine that runs Putty. <BR>
			'''Solution:''' Try 8889 (or some other number above 1024 and below 65535). You'll haved to set Source port in the putty forward to this new port number and edit the URL that gives you the Jupyter notebook (change the :8888 to :8889 etc.)<BR>
			<BR>

	In your Python file or terminal load the appropriate python modules and instantiate the cluster:		In your Python file or terminal load the appropriate python modules and instantiate the cluster:
	<syntaxhighlight lang="python">		<syntaxhighlight lang="python">

Dschulz at 17:54, 29 June 2022

2022-06-29T17:54:46Z

← Older revision		Revision as of 17:54, 29 June 2022
Line 42:		Line 42:
	* This message says that the Jupyter server is running on node sg5 on port 54674. Port 8888 needs to be forwarded from the server running Putty to the cluster.		* This message says that the Jupyter server is running on node sg5 on port 54674. Port 8888 needs to be forwarded from the server running Putty to the cluster.
	* Click the Putty icon in the upper left corner of the Putty window and select "Change Settings"		* Click the Putty icon in the upper left corner of the Putty window and select "Change Settings"
			[[File:ChangeSettings.png\|alt=Image showing where the "Change Settings" option for Putty is located\|none\|thumb\|Image showing where the "Change Settings" option for Putty is located]]
	* If there's already 8888 listed in the Forwarded ports box click it and click Remove -- the Forwarded ports box should be empty.		* If there's already 8888 listed in the Forwarded ports box click it and click Remove -- the Forwarded ports box should be empty.
	* In the Change Settings dialogue box pick "Tunnels" in the left hand box (you'll have to open the Various levels of the tree to get there)		* In the Change Settings dialogue box pick "Tunnels" in the left hand box (you'll have to open the Various levels of the tree to get there)
	* Enter 8888 into the Source port box		* Enter 8888 into the Source port box[[File:SettingsDialogueBox.png\|alt=Putty Settings dialogue box showing the required port forwarding.\|none\|thumb\|Putty Settings dialogue box showing the required port forwarding.]]
	* Copy the circled value from the putty window into the Destination box and click the Add button.		* Copy the circled value from the putty window into the Destination box and click the Add button.
			* The "Forwarded ports" box should now contain a concatenation of the text in the source port and destination boxes. i.e. "8888:sg4:54674"

	* Click Apply		* Click Apply
	* Return to the myappmf tab in your browser and start a Google Chrome there. '''Note:''' It is important that you use the Chrome from myappmf not your desktop.		* Return to the myappmf tab in your browser and start a Google Chrome there. '''Note:''' It is important that you use the Chrome from myappmf not your desktop.
	* Paste the link containing http://localhost:8888?token= into ~~this~~ browser and you should be rewarded with a Jupyter Notebook session.		* Paste the link containing http://localhost:8888?token= found in the putty terminal window into the browser started from myappmf and you should be rewarded with a Jupyter Notebook session.

	In your Python file or terminal load the appropriate python modules and instantiate the cluster:		In your Python file or terminal load the appropriate python modules and instantiate the cluster:

Dschulz: Created page with "= Overview = This guide gives an overview of running Apache Spark clusters under the existing scheduling system of the Level 3/4 Rated Marc cluster at the University of Calga..."

2022-06-29T17:46:56Z

Created page with "= Overview = This guide gives an overview of running Apache Spark clusters under the existing scheduling system of the Level 3/4 Rated Marc cluster at the University of Calga..."

New page

= Overview =

This guide gives an overview of running Apache Spark clusters under the existing scheduling system of the Level 3/4 Rated Marc cluster at the University of Calgary.

Due to certain mandatory security restrictions required when accessing Level 3 and 4 data, we are not able to provide access to Sparc via Open On Demand as it is done on Arc. There is a slightly more complicated procedure to access Spark and the associated Jupyter notebook but it is only slightly more difficult.

== Connecting ==

* Login to Marc normally as described on [MARC_accounts]
* Start the Jupyter notebook and Spark cluster as a job with something like the following (changing the number of cpus and memory to fit your requirements:
<syntaxhighlight lang="bash>
sbatch -N1 -n 38 --cpus-per-task=1 --mem=100G --time=1:00:00 /global/software/spark/sparkOnMarc/sparkOnMarc.sh
Submitted batch job 14790
</syntaxhighlight>
* The above command will return a job number "14790" in this example. A file named slurm-<jobnumber>.out will be created in the current directory. After a few moments the file will contain the connection information required for you to access a Jupyter notebook that has access to the Spark cluster that was just started.
<syntaxhighlight lang="bash>
$ cat slurm-14790.out
<snip -- lots of logging messages that can be ignored unless there's problems>
Discovered Jupyter Notebook server listening on port 54674!
===========================================================================
===========================================================================
===========================================================================
Your Jupyter Notebook with Spark cluster is now active. Inside Putty
You will now need to forward port 8888 to:

sg5:54674

Then you can point your browser started from myappmf.ucalgary.ca to
the URL:

Currently
http://localhost:8888/?token=dccb02273b9be90c833edf658fd3183aa62fa9fabadce6d8

Please see the RCS website rcs.ucalgary.ca for information on how to
access Spark from the Jupyter notebook.

===========================================================================
===========================================================================
===========================================================================
</syntaxhighlight>

* This message says that the Jupyter server is running on node sg5 on port 54674. Port 8888 needs to be forwarded from the server running Putty to the cluster.
* Click the Putty icon in the upper left corner of the Putty window and select "Change Settings"
* If there's already 8888 listed in the Forwarded ports box click it and click Remove -- the Forwarded ports box should be empty.
* In the Change Settings dialogue box pick "Tunnels" in the left hand box (you'll have to open the Various levels of the tree to get there)
* Enter 8888 into the Source port box
* Copy the circled value from the putty window into the Destination box and click the Add button.
* Click Apply
* Return to the myappmf tab in your browser and start a Google Chrome there. '''Note:''' It is important that you use the Chrome from myappmf not your desktop.
* Paste the link containing http://localhost:8888?token= into this browser and you should be rewarded with a Jupyter Notebook session.

In your Python file or terminal load the appropriate python modules and instantiate the cluster:
<syntaxhighlight lang="python">

import os
import atexit
import sys
import re

import pyspark
from pyspark.conf import SparkConf
from pyspark.context import SparkContext
from pyspark.sql import SQLContext
conflines=[tuple(a.rstrip().split(" ")) for a in open(os.environ['SPARK_CONFIG_FILE']).readlines()]
conf=SparkConf()
conf.setAll(conflines)
conf.setMaster("spark://%s:%s"% (os.environ['SPARK_MASTER_HOST'],os.environ['SPARK_MASTER_PORT']))
sc=pyspark.SparkContext(conf=conf)

#You need this line if you want to use SparkSQL
sqlCtx=SQLContext(sc)

#YOUR CODE GOES HERE

</syntaxhighlight>

You now have a sc (Spark Context) and sqlCtx (SQL Context) objects to operate on. Please remember to return to the OOD screen and terminate the Jupyter + Spark app when you are finished.

There are many Spark tutorials out there. Here are some good places to look:

* https://spark.apache.org/docs/latest/quick-start.html
* https://spark.apache.org/docs/latest/rdd-programming-guide.html
* https://spark.apache.org/docs/latest/sql-programming-guide.html

'''HINT''': It helps to google "pyspark" as that returns Python results instead of Scala which is another common language used to interact with Spark.

[[Category:Marc]]
[[Category:Software]]