Jupyter Notebooks

General

• Project site: http://jupyter.org
• Docs: https://jupyter.org/documentation

• Manual: https://jupyter.readthedocs.io/en/latest/

Jupyter Hub on ARC

There is a Jupyter Hub attached to the ARC cluster. One must have an ARC account first, to connect to the hub.

• http://jupyter.ucalgary.ca.

Currently supports python3 only.

Connections are only available for computers that are part of the UofC campus network. To connect from outside of the UofC campus network, please use the Fort VPN client to connect to the campus network, first.

Compute Canada Jupyter Hub for UofC

Hosted by Compute Canada for UofC.

The Pacific Institute for the Mathematical Sciences in collaboration with Compute Canada and Cybera offer cloud-based hubs to universities and schools. Each institution can have its own hub where users authenticate with their credentials from that institution. The hubs are hosted on the Compute Canada Cloud and are essentially for training purposes.

Login with your UofC credentials.

• https://ucalgary.syzygy.ca

Supports python2, python3, julia, and R.

• An article about the service:

https://www.computecanada.ca/featured/compute-canada-and-pims-launch-jupyter-service-for-researchers/

Jupyter on Compute Canada clusters

Compute Canada services require a Compute Canada account. An account can be obtained at https://ccdb.computecanada.ca

• General information on running a Jupyter server on a compute node:

https://docs.computecanada.ca/wiki/Jupyter

Beluga cluster Jupyter Hub

At https://jupyterhub.beluga.calculquebec.ca/hub/login

• Login with you Compute Canada credentials.
• In the Server Options dialog, select the account, time, number of CPUs, and amount of memory.
• Click Start button.
• Once the server starts, click on the Python 3.7 notebook button to get a Jupyter Python notebook session.

• To exit, select the File -> Logout in the menu bar.

Jupyter Hub at the University of Toronto

• Niagara cluster.

https://docs.scinet.utoronto.ca/index.php/Jupyter_Hub

• Point your browser to https://jupyter.scinet.utoronto.ca/

Running a jupyter notebook on ARC's compute node

By using jupyter from a python installation on ARC, one can run jupyter notebooks on ARC's compute nodes and request resources according to one's need. This is most flexible way to run a notebook, but it does require more steps to setup.

Start an interactive job on ARC:

$ salloc -N 1 -n 1 -c 12 --mem=0 -t 3:00:00 -p parallel
    salloc: Granted job allocation 5486616
    salloc: Waiting for resource configuration
    salloc: Nodes cn0526 are ready for job
[username@cn0526 ~]$ 

The command requested a 3 hours interactive job on the Parallel partition. The request includes all 12 CPUs and all the 23GB of RAM on the compute node. The compute node cn026, was allocated to the job.

Load a python software module you want to use and start a jupyter notebook server. Before starting the notebook we have to unset the XDG_RUNTIME_DIR variable, otherwise the notebook crashes.

$ module load python/anaconda3-2018.12
$ unset XDG_RUNTIME_DIR

$ jupyter notebook --no-browser --ip=0.0.0.0

[I 16:04:31.618 NotebookApp] JupyterLab extension loaded from /global/software/anaconda/anaconda3-2018.12/lib/python3.6/site-packages/jupyterlab
[I 16:04:31.618 NotebookApp] JupyterLab application directory is /global/software/anaconda/anaconda3-2018.12/share/jupyter/lab
[I 16:04:31.621 NotebookApp] Serving notebooks from local directory: /home/drozmano
[I 16:04:31.621 NotebookApp] The Jupyter Notebook is running at:
[I 16:04:31.621 NotebookApp] http://(cn0526 or 127.0.0.1):8888/?token=bedfe1920e97a309c583a5f2895cf1367e37a7dac416494b
[I 16:04:31.621 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 16:04:31.673 NotebookApp] 
    
    To access the notebook, open this file in a browser:
        file:///home/drozmano/.local/share/jupyter/runtime/nbserver-25483-open.html
    Or copy and paste one of these URLs:
        http://(cn0526 or 127.0.0.1):8888/?token=bedfe1920e97a309c583a5f2895cf1367e37a7dac416494b

At this point the notebook server is up and listening for connections. You need to use your browser on your local computer (desktop or laptop) to connect to it. You have to use the URL with the token from the last line of that print out.

The problem is that the compute node name, cn0526, is an internal name to the ARC cluster. This name is not known outside of ARC, so really, you cannot connect to it as it is. So we have to setup an SSH tunnel from your local computer to that node cn0526.

Mac or Linux local computers

On you Mac or you Linux computer, open a terminal session and connect to ARC using the normal SSH client with the tunnel option:

$ ssh username@arc.ucalgary.ca -L 8888:cn0526:8888
....

This is a normal ssh connection with a usual text mode session, but we need this session on to keep the tunnel open.

The ssh client creates a tunnel from your local computer's port 8888 to the compute node cn0526 via the ARC's cluster login node. The trick is that your local computer does not know the cn0526 name and has no way of connecting to it, but the ARC's login node does.

So, any connection made to your own computer on the port 8888 will be tunneled through the ARC's login node to the port 8888 of ARC's compute node cn0526. Conveniently, this is the port the notebook is expecting the connections on.

Now start you browser on you local computer and connect to the URL from the notebook's print out:

http://localhost:8888/?token=bedfe1920e97a309c583a5f2895cf1367e37a7dac416494b

Note, that the name of the web server is localhost, you have to edit the URL manually.

Done.

The notebook server will run for the duration of the interactive job you requested on ARC. In this example, for 3 hours. Then the notebook server will be killed by the system, so make sure that you save your progress before this happens.

If your local computer is outside of the UofC campus network you have to use the Fort VPN client to connect to the UofC network first.

Windows

To be determined....