Jupyter Notebooks - Revision history

Dmitri: /* Running a jupyter notebook on ARC's compute node */

2025-02-18T20:34:38Z

Running a jupyter notebook on ARC's compute node

← Older revision		Revision as of 20:34, 18 February 2025
Line 89:		Line 89:

	Load a python software module you want to use and start a '''jupyter notebook''' server.		Load a python software module you want to use and start a '''jupyter notebook''' server.

	Before starting the notebook we have to unset the '''XDG_RUNTIME_DIR''' variable,		Before starting the notebook we have to unset the '''XDG_RUNTIME_DIR''' variable,
	otherwise the notebook ~~crashes~~.		otherwise the notebook may crash (you '''do not need''' to do this now).
	<pre>		<pre>
	$ module load python/3.12.5		$ module load python/3.12.5

Dmitri: /* Running a jupyter notebook on ARC's compute node */

2025-02-18T20:33:20Z

Running a jupyter notebook on ARC's compute node

← Older revision		Revision as of 20:33, 18 February 2025
Line 79:		Line 79:
	Start an interactive job on ARC:		Start an interactive job on ARC:
	<pre>		<pre>
	$ salloc -~~N 1~~ -~~n 1~~ -~~c 12~~ --mem=0 -t 3:00:00 -p ~~parallel~~		$ salloc -N1 -n1 -c4 --mem=16gb -t 3:00:00 -p single
	salloc: Granted job allocation ~~5486616~~		salloc: Granted job allocation 33874452
	~~salloc: Waiting for resource configuration~~		salloc: Nodes h14 are ready for job
	salloc: Nodes ~~cn0526~~ are ready for job		[username@h14 ~]$
	[username@~~cn0526~~ ~]$
	</pre>		</pre>
	The command requested a 3 hours interactive job on the Parallel partition.		The command requested a 3 hours interactive job on the Parallel partition.
	The request includes all 12 CPUs and all the ~~23GB~~ of RAM on the compute node.		The request includes all 4 CPUs and all the 16GB of RAM on the compute node.
	The compute node '''~~cn026~~''', was allocated to the job.		The compute node '''h14''', was allocated to the job.

	Load a python software module you want to use and start a '''jupyter notebook''' server.		Load a python software module you want to use and start a '''jupyter notebook''' server.
Line 93:		Line 92:
	otherwise the notebook crashes.		otherwise the notebook crashes.
	<pre>		<pre>
	$ module load python/~~anaconda3-2018~~.12		$ module load python/3.12.5
	$ unset XDG_RUNTIME_DIR		$ unset XDG_RUNTIME_DIR

	$ jupyter notebook --no-browser --ip=0.0.0.0		$ jupyter notebook --no-browser --ip=0.0.0.0

	[I 16:04:31.~~618 NotebookApp~~] ~~JupyterLab~~ extension ~~loaded from /global/software/anaconda/anaconda3-2018.12/lib/python3~~.~~6/site-packages/jupyterlab~~		[I 2025-02-18 13:30:12.439 ServerApp] jupyter_lsp \| extension was successfully linked.
	[I 16:04:31.~~618 NotebookApp~~] ~~JupyterLab application directory is /global/software/anaconda/anaconda3-2018~~.~~12/share/jupyter/lab~~		[I 2025-02-18 13:30:12.443 ServerApp] jupyter_server_terminals \| extension was successfully linked.
	[I 16:04:31.~~621 NotebookApp~~] ~~Serving notebooks from local directory: /home/drozmano~~		[I 2025-02-18 13:30:12.449 ServerApp] jupyterlab \| extension was successfully linked.
	[I 16:04:31.~~621 NotebookApp~~] ~~The Jupyter Notebook is running at:~~		[I 2025-02-18 13:30:12.454 ServerApp] notebook \| extension was successfully linked.
	[I 16:04:31.~~621 NotebookApp~~] ~~http://(cn0526 or 127.0.0~~.~~1):8888/?token=bedfe1920e97a309c583a5f2895cf1367e37a7dac416494b~~		[I 2025-02-18 13:30:12.961 ServerApp] notebook_shim \| extension was successfully linked.
	[I 16:04:31.~~621 NotebookApp~~] ~~Use Control-C to stop this server and shut down all kernels (twice to skip confirmation)~~.		[I 2025-02-18 13:30:13.019 ServerApp] notebook_shim \| extension was successfully loaded.
	[C 16:04:31.~~673 NotebookApp~~]		....
			....
			[C 2025-02-18 13:30:13.073 ServerApp]

	To access the ~~notebook~~, open this file in a browser:		To access the server, open this file in a browser:
	file:///home/drozmano/.local/share/jupyter/runtime/~~nbserver~~-~~25483~~-open.html		file:///home/drozmano/.local/share/jupyter/runtime/jpserver-78009-open.html
	Or copy and paste one of these URLs:		Or copy and paste one of these URLs:
	http://~~(cn0526 or~~ 127.0.0.1):8888/?token=~~bedfe1920e97a309c583a5f2895cf1367e37a7dac416494b~~		http://h14:8888/tree?token=9beed319458a43e947fea149caca0fcfb226f21c0f0669ad
			http://127.0.0.1:8888/tree?token=9beed319458a43e947fea149caca0fcfb226f21c0f0669ad
			....
	</pre>		</pre>
	~~At this point the notebook server is up and listening for connections.~~
	~~You need to use your browser on your local computer (desktop or laptop) to connect to it.~~
	~~You have to use the URL with the token from the last line of that print out.~~


	The ~~problem is that the~~ compute node name, '''~~cn0526~~''', is an internal name to the ARC cluster.		The compute node name, '''h14''', is an internal name to the ARC cluster.
	This name is not known outside of ARC, so really, you cannot connect to it as it is.		This name is '''not known outside of ARC''', so really, you cannot connect to it as it is.
	So we have to setup an '''SSH tunnel''' from your local computer to that node '''~~cn0526~~'''.		So we have to setup an '''SSH tunnel''' from your local computer to that node '''h14'''.

	=== Mac or Linux local computers ===		=== Mac or Linux local computers ===
Line 124:		Line 123:
	with the '''tunnel''' option:		with the '''tunnel''' option:
	<pre>		<pre>
	$ ssh username@arc.ucalgary.ca -L 8888:~~cn0526~~:8888		$ ssh username@arc.ucalgary.ca -L 8888:h14:8888
	....		....
	</pre>		</pre>
	This is a normal ssh connection with a usual text mode session, but we need this session on to keep the tunnel open.		This is a normal ssh connection with a usual text mode session, but we need this session on to keep the tunnel open.

	The ssh client creates a tunnel from '''your local computer''''s port 8888 to the '''compute node ~~cn0526~~''' via the ARC's cluster '''login node'''.		The ssh client creates a tunnel from '''your local computer''''s port 8888 to the '''compute node h14''' via the ARC's cluster '''login node'''.
	The trick is that your local computer does not know the '''~~cn0526~~''' name and has no way of connecting to it, but the ARC's login node does.		The trick is that your local computer does not know the '''h14''' name and has no way of connecting to it, but the ARC's login node does.

	So, any connection made to your own computer on the port 8888 will be '''tunneled''' through		So, any connection made to your own computer on the port 8888 will be '''tunneled''' through
	the ARC's login node to the port 8888 of ARC's compute node '''~~cn0526~~'''.		the ARC's login node to the port 8888 of ARC's compute node '''h14'''.
	Conveniently, this is the port the notebook is expecting the connections on.		Conveniently, this is the port the notebook is expecting the connections on.

Lleung: Removed how-tos and guides category as it's just a software page

2023-09-21T20:37:24Z

Removed how-tos and guides category as it's just a software page

← Older revision		Revision as of 20:37, 21 September 2023
Line 150:		Line 150:
	If your local computer is outside of the UofC campus network you have to use the Fort VPN client to connect to the UofC network first.		If your local computer is outside of the UofC campus network you have to use the Fort VPN client to connect to the UofC network first.

	~~[[Category:Guides]]~~
	~~[[Category:How-Tos]]~~
	[[Category:Software]]		[[Category:Software]]
	{{Navbox Software}}		{{Navbox Software}}

Lleung: Added categories and navbox

2023-09-20T22:33:43Z

Added categories and navbox

← Older revision		Revision as of 22:33, 20 September 2023
Line 150:		Line 150:
	If your local computer is outside of the UofC campus network you have to use the Fort VPN client to connect to the UofC network first.		If your local computer is outside of the UofC campus network you have to use the Fort VPN client to connect to the UofC network first.

	~~=== Windows ===~~		[[Category:Guides]]
			[[Category:How-Tos]]
	~~To be determined....~~		[[Category:Software]]
			{{Navbox Software}}

Dmitri: /* Compute Canada Jupyter Hub for UofC (Syzygy) */

2023-04-03T15:57:52Z

Compute Canada Jupyter Hub for UofC (Syzygy)

← Older revision		Revision as of 15:57, 3 April 2023
Line 22:		Line 22:
	Jupyter Notebook can be launched through Open OnDemand on ARC accessible at https://ood-arc.rcs.ucalgary.ca . You may launch Jupyter Notebook or JupyterLab directly on the cluster with support for a wide range of programming languages using one of the many pre-made Jupyter environments. This is the preferred way to launch Jupyter Notebooks due to its ease of use and flexibility.		Jupyter Notebook can be launched through Open OnDemand on ARC accessible at https://ood-arc.rcs.ucalgary.ca . You may launch Jupyter Notebook or JupyterLab directly on the cluster with support for a wide range of programming languages using one of the many pre-made Jupyter environments. This is the preferred way to launch Jupyter Notebooks due to its ease of use and flexibility.

	== ~~Compute Canada~~ Jupyter Hub for UofC (Syzygy) ==		== The Alliance Jupyter Hub for UofC (Syzygy) ==

			The '''Pacific Institute for the Mathematical Sciences''' in collaboration with '''the Digital Research Alliance of Canada'''
	~~Hosted by Compute Canada for UofC.~~		and '''Cybera''' offer cloud-based hubs to universities and schools.

	The '''Pacific Institute for the Mathematical Sciences''' in collaboration with '''~~Compute~~ Canada''' and '''Cybera''' offer cloud-based hubs to universities and schools.
	Each institution can have its own hub where users authenticate with their credentials from that institution.		Each institution can have its own hub where users authenticate with their credentials from that institution.
	The hubs are hosted on the '''~~Compute Canada~~ Cloud''' and are essentially for '''training purposes'''.		The hubs are hosted on the '''Alliance Cloud''' and are essentially for '''training purposes'''.

	Login with your '''UofC credentials'''.		Login with your '''UofC credentials'''.

Dmitri: /* Jupyter on Compute Canada clusters */

2023-04-03T15:56:31Z

Jupyter on Compute Canada clusters

← Older revision		Revision as of 15:56, 3 April 2023
Line 48:		Line 48:
	: https://intro.syzygy.ca/troubleshooting/		: https://intro.syzygy.ca/troubleshooting/

	== Jupyter on ~~Compute~~ Canada clusters ==		== Jupyter on the Digital Research Alliance of Canada clusters ==


	~~Compute Canada~~ services require ~~a Compute Canada~~ account.		The Alliance services require an Alliance account.
	An account can be obtained at https://ccdb.computecanada.ca		An account can be obtained at https://ccdb.computecanada.ca

	* General information on running a Jupyter server on a compute node:		* General information on running a Jupyter server on a compute node:
	: https://docs.~~computecanada~~.ca/wiki/~~Jupyter~~		: https://docs.alliancecan.ca/wiki/JupyterHub

	=== Beluga cluster Jupyter Hub ===		=== Beluga cluster Jupyter Hub ===

Dmitri: /* ARC */

2022-10-19T16:24:17Z

ARC

← Older revision		Revision as of 16:24, 19 October 2022
Line 21:		Line 21:

	Jupyter Notebook can be launched through Open OnDemand on ARC accessible at https://ood-arc.rcs.ucalgary.ca . You may launch Jupyter Notebook or JupyterLab directly on the cluster with support for a wide range of programming languages using one of the many pre-made Jupyter environments. This is the preferred way to launch Jupyter Notebooks due to its ease of use and flexibility.		Jupyter Notebook can be launched through Open OnDemand on ARC accessible at https://ood-arc.rcs.ucalgary.ca . You may launch Jupyter Notebook or JupyterLab directly on the cluster with support for a wide range of programming languages using one of the many pre-made Jupyter environments. This is the preferred way to launch Jupyter Notebooks due to its ease of use and flexibility.

	~~=== Jupyter Hub on ARC ===~~
	~~A Jupyter Hub is running on the ARC cluster and is available at http://jupyter.ucalgary.ca from a University campus network. Please be aware of the following limitations:~~

	* You must have an ARC account to connect to the hub.
	* Currently supports '''python3''' only.
	* '''Connections''' are only available for computers that are part of the UofC campus network or through the University General VPN (FortiVPN).


	== Compute Canada Jupyter Hub for UofC (Syzygy) ==		== Compute Canada Jupyter Hub for UofC (Syzygy) ==

Dmitri: /* Open OnDemand */

2021-10-21T22:58:01Z

Open OnDemand

← Older revision		Revision as of 22:58, 21 October 2021
Line 20:		Line 20:
	''Main Article: [[Open OnDemand]]''		''Main Article: [[Open OnDemand]]''

	Jupyter Notebook can be launched through Open OnDemand on ARC accessible at https://ood-arc.ucalgary.ca. You may launch Jupyter Notebook or JupyterLab directly on the cluster with support for a wide range of programming languages using one of the many pre-made Jupyter environments. This is the preferred way to launch Jupyter Notebooks due to its ease of use and flexibility.		Jupyter Notebook can be launched through Open OnDemand on ARC accessible at https://ood-arc.rcs.ucalgary.ca . You may launch Jupyter Notebook or JupyterLab directly on the cluster with support for a wide range of programming languages using one of the many pre-made Jupyter environments. This is the preferred way to launch Jupyter Notebooks due to its ease of use and flexibility.

	=== Jupyter Hub on ARC ===		=== Jupyter Hub on ARC ===

Lleung at 17:16, 17 August 2021

2021-08-17T17:16:31Z

← Older revision		Revision as of 17:16, 17 August 2021
Line 1:		Line 1:
	~~= General =~~		Jupyter Notebook is an interactive web-based environment used for creating a wide range of workflows in data science, machine learning, and general computation. There are a wide range of programming languages that are supported through Jupyter Notebooks including Python, Julia, and R.

			There are related products which help launch and manage Jupyter Notebooks including:

			* Jupyter Hub which allows users to launch a notebook instance from a web portal and
			* JupyterLab which is Jupyter's next-generation notebook interface and comes with a more flexible user interface.

			This page will go over the available options available to you to get started with Jupyter Notebook.

			Resources:
	* Project site: http://jupyter.org		* Project site: http://jupyter.org
	* Docs: https://jupyter.org/documentation		* Docs: https://jupyter.org/documentation

	* Manual: https://jupyter.readthedocs.io/en/latest/		* Manual: https://jupyter.readthedocs.io/en/latest/

Lleung at 17:15, 17 August 2021

2021-08-17T17:15:54Z

← Older revision		Revision as of 17:15, 17 August 2021
Line 6:		Line 6:
	* Manual: https://jupyter.readthedocs.io/en/latest/		* Manual: https://jupyter.readthedocs.io/en/latest/

	~~= Jupyter Hub on ARC =~~

	~~There is a~~ Jupyter ~~Hub attached to the~~ ARC ~~cluster~~. ~~One~~ must have an ARC account ~~first, to connect to~~ the ~~hub~~.		== ARC ==
			RCS offers the following methods to use Jupyter on ARC. You must have an ARC account for the following options.

	* http:~~//jupyter.ucalgary.ca.~~		=== Open OnDemand ===
			''Main Article: [[Open OnDemand]]''

	~~Currently supports '''python3''' only~~.		Jupyter Notebook can be launched through Open OnDemand on ARC accessible at https://ood-arc.ucalgary.ca. You may launch Jupyter Notebook or JupyterLab directly on the cluster with support for a wide range of programming languages using one of the many pre-made Jupyter environments. This is the preferred way to launch Jupyter Notebooks due to its ease of use and flexibility.

	~~'''Connections''' are only~~ available ~~for computers that are part of the UofC~~ campus network.		=== Jupyter Hub on ARC ===
	~~To connect from outside~~ of the ~~UofC campus network, please use the Fort VPN client to connect to the campus network, first.~~		A Jupyter Hub is running on the ARC cluster and is available at http://jupyter.ucalgary.ca from a University campus network. Please be aware of the following limitations:

	= Compute Canada Jupyter Hub for UofC (Syzygy) =		* You must have an ARC account to connect to the hub.
			* Currently supports '''python3''' only.
			* '''Connections''' are only available for computers that are part of the UofC campus network or through the University General VPN (FortiVPN).


			== Compute Canada Jupyter Hub for UofC (Syzygy) ==


Line 43:		Line 49:
	: https://intro.syzygy.ca/troubleshooting/		: https://intro.syzygy.ca/troubleshooting/

	= Jupyter on Compute Canada clusters =		== Jupyter on Compute Canada clusters ==


Line 52:		Line 58:
	: https://docs.computecanada.ca/wiki/Jupyter		: https://docs.computecanada.ca/wiki/Jupyter

	== Beluga cluster Jupyter Hub ==		=== Beluga cluster Jupyter Hub ===

	At https://jupyterhub.beluga.calculquebec.ca/hub/login		At https://jupyterhub.beluga.calculquebec.ca/hub/login
Line 63:		Line 69:
	* To exit, select the '''File -> Logout''' in the menu bar.		* To exit, select the '''File -> Logout''' in the menu bar.

	== Jupyter Hub at the University of Toronto ==		=== Jupyter Hub at the University of Toronto ===

	* Niagara cluster.		* Niagara cluster.
Line 70:		Line 76:
	* Point your browser to https://jupyter.scinet.utoronto.ca/		* Point your browser to https://jupyter.scinet.utoronto.ca/

	= Running a jupyter notebook on ARC's compute node =		== Running a jupyter notebook on ARC's compute node ==
	By using jupyter from a python installation on ARC, one can run jupyter notebooks on ARC's compute nodes and request resources according to one's need.		By using jupyter from a python installation on ARC, one can run jupyter notebooks on ARC's compute nodes and request resources according to one's need.
	This is most flexible way to run a notebook, but it does require more steps to setup.		This is most flexible way to run a notebook, but it does require more steps to setup.
Line 117:		Line 123:
	So we have to setup an '''SSH tunnel''' from your local computer to that node '''cn0526'''.		So we have to setup an '''SSH tunnel''' from your local computer to that node '''cn0526'''.

	== Mac or Linux local computers ==		=== Mac or Linux local computers ===
	On you Mac or you Linux computer, open a terminal session and connect to ARC using the normal '''SSH client'''		On you Mac or you Linux computer, open a terminal session and connect to ARC using the normal '''SSH client'''
	with the '''tunnel''' option:		with the '''tunnel''' option:
Line 147:		Line 153:
	If your local computer is outside of the UofC campus network you have to use the Fort VPN client to connect to the UofC network first.		If your local computer is outside of the UofC campus network you have to use the Fort VPN client to connect to the UofC network first.

	== Windows ==		=== Windows ===

	To be determined....		To be determined....