MARC Cluster Guide: Difference between revisions

From RCSWiki
Jump to navigation Jump to search
No edit summary
No edit summary
 
(25 intermediate revisions by 6 users not shown)
Line 1: Line 1:
This guide gives an overview of the MARC (Medical Advanced Research Computing) cluster at the University of Calgary.
{{MARC Cluster Status}}
{{Message Box
|title=Need Help or have other MARC Related Questions?
|message=For all general RCS related issues, questions, or comments, please contact us at support@hpc.ucalgary.ca.
|icon=Support Icon.png}}


It is intended to be read by new account holders getting started on MARC, covering such topics as the hardware and performance characteristics, available software, usage policies and how to log in.
This guide gives an overview of the MARC (Medical Advanced Research Computing) cluster at the University of Calgary and is intended to be read by new account holders getting started on MARC. This guide covers information on MARC's restrictions, hardware, performance characteristics, and storage information.  


If you are looking for how to login to MARC or how to get an account, please see [[MARC_accounts]]
== Introduction ==
 
MARC is a cluster of Linux-based computers that were purchased in 2019. It has been specifically designed to meet the security requirements for handling Level 3 and Level 4 classified data, as defined by the University of Calgary Information Security Classification Standard, which you can find here: https://www.ucalgary.ca/policies/files/policies/im010-03-security-standard_0.pdf
For MARC-related questions not answered here, please write to support@hpc.ucalgary.ca .
 
'''Cybersecurity awareness at the U of C'''


Please note that there are typically about 950 phishing attempts targeting University of Calgary accounts each month. This is just a reminder to be careful about computer security issues, both at home and at the University. See [https://it.ucalgary.ca/it-security] for more information, such as tips for secure computing and how to report suspected security problems.
To ensure the security of Level 3/4 data, MARC has implemented several restrictions:


== Introduction ==
* '''Account Requirement''': All users must have IT accounts.
MARC is a cluster comprised of Linux-based computers purchased in 2019
* '''Project ID Requirement''': To use MARC, a project ID is required. This project ID is the same number used on SCDS.
* '''SSH Access via Citrix''': Access to MARC via SSH must be done through the IT Citrix system. Admin VPN access is neither sufficient nor necessary.
* '''Internet Access''': Neither compute nodes nor login nodes have internet access.
* '''Data Ingestion via SCDS''': All data must be transferred to MARC by first copying it to SCDS (Secure Compute Data Store) and then fetching it from SCDS to MARC.
* '''Data Retrieval via SCDS''': Resulting data, such as analysis outputs, must be copied to SCDS and then fetched from SCDS to its intended destination using established methods.
* '''Data Auditing''': All file accesses are logged for auditing purposes.


The MARC cluster has been designed with controls appropriate for Level 3 and Level 4 classified data. The University of Calgary Information Security Classification Standard is published here: https://www.ucalgary.ca/policies/files/policies/im010-03-security-standard_0.pdf
These measures have been put in place to ensure the security and controlled handling of sensitive data on the MARC cluster and to prevent intentional or accidental data exfiltration.


Due to security requirements for Level 3/4 data, some necessary restrictions have been placed on MARC to prevent accidental (or otherwise) data exfiltration.
== Obtaining an account ==
* Compute nodes and login nodes have no access to the internet
Please refer to the [[how to get a MARC account]] page for more information on how to obtain and log in to MARC.
* All data must be ingested to MARC by first copying it to SCDS (Secure Compute Data Store) and then fetching it from SCDS to MARC.
* Resulting data (outputs of analyses) must be copyied to SCDS and then fetching it from SCDS to wherever it needs to go using established means.
* All file accesses are recorded for auditing purposes.
* ssh connections to MARC must be through the IT Citrix system (Admin VPN is not sufficient nor necessary)
* All accounts must be IT accounts
* A project ID is required to use MARC. This project ID is the same number that is used on SCDS


== Hardware ==
== Hardware ==
MARC has compute nodes of two different varieties:  
MARC has compute nodes of two different varieties:  
* 8 GPU (Graphics Processing Unit)-enabled nodes containing:
* 4 GPU (Graphics Processing Unit)-enabled nodes containing:
** 40-cores: each node having 2 sockets. Each socket has an Intel Xeon Gold 6148 20-core processor, running at 2.4 GHz.  
** 40-cores: each node having 2 sockets. Each socket has an Intel Xeon Gold 6148 20-core processor, running at 2.4 GHz.  
** The 40 cores on the individual compute nodes share about 750 GB of RAM (memory) but, jobs should request no more than 753000 MB.
** The 40 cores on the individual compute nodes share about 750 GB of RAM (memory) but, jobs should request no more than 753000 MB.
Line 34: Line 34:
** 80-cores: node with 4 sockets.  Each socket has an Intel Xeon Gold 6148 20-core processor, running at 2.4 GHz.   
** 80-cores: node with 4 sockets.  Each socket has an Intel Xeon Gold 6148 20-core processor, running at 2.4 GHz.   
** The 80 cores on the node share about 3 TB of RAM (memory), but, jobs should request no more than 3000000 MB.
** The 80 cores on the node share about 3 TB of RAM (memory), but, jobs should request no more than 3000000 MB.
The hardware is broken out into three distinct partitions with the following restrictions.
{| class="wikitable"
!Partition
!Purpose and description
!CPUs per Node
!GPUs per Node
!Memory per Node
|-
|cpu2019
|For non-GPU jobs; homogeneous nodes
|Up to 38
|No GPUs
|Up to 500GB
|-
|gpu2019
|For jobs requiring nVidia v100 gpu jobs
|Up to 40
|1 or 2 GPUs
|Up to 750GB
|-
|bigmem
|For very large memory jobs
|Up to 80
|No GPUs
|Up to 3TB
|}
== Storage ==
About a petabyte of raw disk storage is available to the MARC cluster, but for error checking and performance reasons, the amount of usable storage for researchers' projects is considerably less than that.  From a user's perspective, the total amount of storage is less important than the individual storage limits.
MARC storage is not accessible outside of MARC.
=== Home file system: /home ===
Each account on MARC has a home directory under <code>/home</code> with a 25 GB per-user quota. This limit is fixed and cannot be increased. The intended use for the home directory is for software, scripts, and configuration files and must only contain Level 1/2 data only. Do not store patient identifiable files (Level 3/4) in your home directory. Level 3/4 data is only appropriate under <code>/project</code>.
=== Project file system: /project ===
All projects will have a directory under <code>/project</code> named after your project ID and will be the same as your SCDS share name. All files related to a project should only be stored within its assigned <code>/project</code> directory. Quotas in <code>/project</code> are somewhat flexible and are assigned based on the project requirements. Please contact support@hpc.ucalgary.ca if you require additional space.
The SCDS share and the project directory on MARC are two separate storage systems. Data transferred to the <code>/project</code> directory is a second copy of the data on the SCDS share. On a SCDS share deleting files will not create free space. The NetApp device that hosts the SCDS drive maintains a copy of the deleted files in ‘snapshot space’. Hence the quota is consumed by the files in the SCDS share plus the files in ‘snapshot space’.
=== Temporary file system: /tmp ===
Each compute node has a temporary storage location available under <code>/tmp</code> that is only accessible from within that node and for the running job. Data stored in <code>/tmp</code> will be deleted immediately after the job terminates. It is suitable for all levels of data.
Because the login node is a shared system, you must take care that data stored within <code>/tmp</code> on the login node is restricted to only your account using the appropriate file modes (<code>chmod</code> and <code>chown</code>/<code>chgrp</code> commands) and deleted after use.


=== cpu2019 ===
== Software installations ==
Allows non-GPU jobs to use:
* Up to 38 cpus per node
* No gpus.
* Up to 500GB memory
* Are the same


=== gpu2019 ===
=== Python ===
Allows jobs requiring nVidia v100 gpu jobs to use:
* 1 or 2 gpus per node
* Up to 40 cpus per node.
* Up to 750GB memory


=== bigmem ===
There are some complications in using Python on MARC relative to using ARC. 
For very large memory jobs:
* Up to 80 cpus
* Up to 3TB memory
* No gpus


== Storage ==
Normally, we would recommend installing Conda in user's home directory.
About a petabyte of raw disk storage is available to the MARC cluster, but for error checking and performance reasons, the amount of usable storage for researchers' projects is considerably less than that. From a user's perspective, the total amount of storage is less important than the individual storage limits.  As described below, there are two storage areas: home and project.
On MARC, security requirements for working with Level 4 data require that we block outgoing and incoming internet connections.  
As a result, new packages cannot be downloaded with conda.  


=== Home file system: /home ===
Depending on what you need, the two recommendations we can make are
There is a per-user quota of 50 GB under /home. This limit is fixed and cannot be increased.  Each user has a directory under /home, which is the default working directory when logging in to MARC. It is expected that most researchers will work from /project and only use home for software and such things. /home is expected to be used only for L1/L2 data and not for your patient identifiable files.  The identifiable files go in the appropriate directory under /project.
* Download the standard anaconda distribution from the anaconda website to a personal computer: https://repo.anaconda.com/archive/Anaconda3-2020.07-Linux-x86_64.sh
** Transfer the script to MARC via SCDS
** Copy it to your /home directory
** Install it in your home directory with $bash Anaconda3-2020.07-Linux-x86_64.sh
** you will be asked to agree to a license agreement and to confirm that you wish to create a folder anaconda3 once the installation completes, you will have a new directory under your home directory ~/anaconda3. In order to make it possible to use the local conda instance you will need to change the system path to include your local python directories $ export PATH=~/anaconda3/bin:$PATH
* Download a docker container with the software that you need including python (e.g. tensorflow-gpu)
** Transfer the docker container to MARC via SCDS
** Copy it to your /home directory
** Run it with singularity


=== Project file system for larger projects: /project ===
* Non-open source software which requires a connection to a license server may require admin assistance to set up. contact [mailto:support@hpc.ucalgary.ca support@hpc.ucalgary.ca] for support.
Directories will be created in /project named after your project ID. This name will be the same as your SCDS share name. The expectation is that all files to do with that project will be stored in /project/projectid.  Quotas in /project are somewhat flexible.  Please write to support@hpc.ucalgary.ca with an estimate of how much space you will require.


== Further Reading ==
== Further Reading ==
See [[Running_jobs]] for information on starting a job
See [[Running jobs]] for information on how to submit a job in the HPC cluster.
 
[[Category:MARC]]
[[Category:Guides]]

Latest revision as of 17:10, 13 August 2024

MARC status: Cluster operational

No upgrades planned. Please contact us if you experience system issues.

See the MARC Cluster Status page for system notices.

Support Icon.png

Need Help or have other MARC Related Questions?

For all general RCS related issues, questions, or comments, please contact us at support@hpc.ucalgary.ca.

This guide gives an overview of the MARC (Medical Advanced Research Computing) cluster at the University of Calgary and is intended to be read by new account holders getting started on MARC. This guide covers information on MARC's restrictions, hardware, performance characteristics, and storage information.

Introduction

MARC is a cluster of Linux-based computers that were purchased in 2019. It has been specifically designed to meet the security requirements for handling Level 3 and Level 4 classified data, as defined by the University of Calgary Information Security Classification Standard, which you can find here: https://www.ucalgary.ca/policies/files/policies/im010-03-security-standard_0.pdf

To ensure the security of Level 3/4 data, MARC has implemented several restrictions:

  • Account Requirement: All users must have IT accounts.
  • Project ID Requirement: To use MARC, a project ID is required. This project ID is the same number used on SCDS.
  • SSH Access via Citrix: Access to MARC via SSH must be done through the IT Citrix system. Admin VPN access is neither sufficient nor necessary.
  • Internet Access: Neither compute nodes nor login nodes have internet access.
  • Data Ingestion via SCDS: All data must be transferred to MARC by first copying it to SCDS (Secure Compute Data Store) and then fetching it from SCDS to MARC.
  • Data Retrieval via SCDS: Resulting data, such as analysis outputs, must be copied to SCDS and then fetched from SCDS to its intended destination using established methods.
  • Data Auditing: All file accesses are logged for auditing purposes.

These measures have been put in place to ensure the security and controlled handling of sensitive data on the MARC cluster and to prevent intentional or accidental data exfiltration.

Obtaining an account

Please refer to the how to get a MARC account page for more information on how to obtain and log in to MARC.

Hardware

MARC has compute nodes of two different varieties:

  • 4 GPU (Graphics Processing Unit)-enabled nodes containing:
    • 40-cores: each node having 2 sockets. Each socket has an Intel Xeon Gold 6148 20-core processor, running at 2.4 GHz.
    • The 40 cores on the individual compute nodes share about 750 GB of RAM (memory) but, jobs should request no more than 753000 MB.
    • Two Tesla V100-PCIE-16GB GPUs.
  • 1 Bigmem Node
    • 80-cores: node with 4 sockets. Each socket has an Intel Xeon Gold 6148 20-core processor, running at 2.4 GHz.
    • The 80 cores on the node share about 3 TB of RAM (memory), but, jobs should request no more than 3000000 MB.

The hardware is broken out into three distinct partitions with the following restrictions.

Partition Purpose and description CPUs per Node GPUs per Node Memory per Node
cpu2019 For non-GPU jobs; homogeneous nodes Up to 38 No GPUs Up to 500GB
gpu2019 For jobs requiring nVidia v100 gpu jobs Up to 40 1 or 2 GPUs Up to 750GB
bigmem For very large memory jobs Up to 80 No GPUs Up to 3TB

Storage

About a petabyte of raw disk storage is available to the MARC cluster, but for error checking and performance reasons, the amount of usable storage for researchers' projects is considerably less than that. From a user's perspective, the total amount of storage is less important than the individual storage limits.

MARC storage is not accessible outside of MARC.

Home file system: /home

Each account on MARC has a home directory under /home with a 25 GB per-user quota. This limit is fixed and cannot be increased. The intended use for the home directory is for software, scripts, and configuration files and must only contain Level 1/2 data only. Do not store patient identifiable files (Level 3/4) in your home directory. Level 3/4 data is only appropriate under /project.

Project file system: /project

All projects will have a directory under /project named after your project ID and will be the same as your SCDS share name. All files related to a project should only be stored within its assigned /project directory. Quotas in /project are somewhat flexible and are assigned based on the project requirements. Please contact support@hpc.ucalgary.ca if you require additional space.

The SCDS share and the project directory on MARC are two separate storage systems. Data transferred to the /project directory is a second copy of the data on the SCDS share. On a SCDS share deleting files will not create free space. The NetApp device that hosts the SCDS drive maintains a copy of the deleted files in ‘snapshot space’. Hence the quota is consumed by the files in the SCDS share plus the files in ‘snapshot space’.

Temporary file system: /tmp

Each compute node has a temporary storage location available under /tmp that is only accessible from within that node and for the running job. Data stored in /tmp will be deleted immediately after the job terminates. It is suitable for all levels of data.

Because the login node is a shared system, you must take care that data stored within /tmp on the login node is restricted to only your account using the appropriate file modes (chmod and chown/chgrp commands) and deleted after use.

Software installations

Python

There are some complications in using Python on MARC relative to using ARC.

Normally, we would recommend installing Conda in user's home directory. On MARC, security requirements for working with Level 4 data require that we block outgoing and incoming internet connections. As a result, new packages cannot be downloaded with conda.

Depending on what you need, the two recommendations we can make are

  • Download the standard anaconda distribution from the anaconda website to a personal computer: https://repo.anaconda.com/archive/Anaconda3-2020.07-Linux-x86_64.sh
    • Transfer the script to MARC via SCDS
    • Copy it to your /home directory
    • Install it in your home directory with $bash Anaconda3-2020.07-Linux-x86_64.sh
    • you will be asked to agree to a license agreement and to confirm that you wish to create a folder anaconda3 once the installation completes, you will have a new directory under your home directory ~/anaconda3. In order to make it possible to use the local conda instance you will need to change the system path to include your local python directories $ export PATH=~/anaconda3/bin:$PATH
  • Download a docker container with the software that you need including python (e.g. tensorflow-gpu)
    • Transfer the docker container to MARC via SCDS
    • Copy it to your /home directory
    • Run it with singularity
  • Non-open source software which requires a connection to a license server may require admin assistance to set up. contact support@hpc.ucalgary.ca for support.

Further Reading

See Running jobs for information on how to submit a job in the HPC cluster.

Retrieved from "https://rcs.ucalgary.ca/index.php?title=MARC_Cluster_Guide&oldid=3494"