MARC Cluster Guide - Revision history

Lleung at 17:10, 13 August 2024

2024-08-13T17:10:49Z

← Older revision		Revision as of 17:10, 13 August 2024
Line 5:		Line 5:
	\|icon=Support Icon.png}}		\|icon=Support Icon.png}}

	This guide gives an overview of the MARC (Medical Advanced Research Computing) cluster at the University of Calgary.		This guide gives an overview of the MARC (Medical Advanced Research Computing) cluster at the University of Calgary and is intended to be read by new account holders getting started on MARC. This guide covers information on MARC's restrictions, hardware, performance characteristics, and storage information.

	It is intended to be read by new account holders getting started on MARC, ~~covering such topics as the~~ hardware ~~and~~ performance characteristics, ~~available software, usage policies~~ and ~~how to log in~~.

	~~If you are looking for how to login to MARC or how to get an account, please see [[MARC_accounts]]~~

	== Introduction ==		== Introduction ==
Line 25:		Line 21:

	These measures have been put in place to ensure the security and controlled handling of sensitive data on the MARC cluster and to prevent intentional or accidental data exfiltration.		These measures have been put in place to ensure the security and controlled handling of sensitive data on the MARC cluster and to prevent intentional or accidental data exfiltration.

			== Obtaining an account ==
			Please refer to the [[how to get a MARC account]] page for more information on how to obtain and log in to MARC.

	== Hardware ==		== Hardware ==
Line 73:		Line 72:
	All projects will have a directory under <code>/project</code> named after your project ID and will be the same as your SCDS share name. All files related to a project should only be stored within its assigned <code>/project</code> directory. Quotas in <code>/project</code> are somewhat flexible and are assigned based on the project requirements. Please contact support@hpc.ucalgary.ca if you require additional space.		All projects will have a directory under <code>/project</code> named after your project ID and will be the same as your SCDS share name. All files related to a project should only be stored within its assigned <code>/project</code> directory. Quotas in <code>/project</code> are somewhat flexible and are assigned based on the project requirements. Please contact support@hpc.ucalgary.ca if you require additional space.

	The SCDS share and the project directory on MARC are two separate ~~storages~~. Data transferred to the <code>/project</code> directory is a second copy of the data on the SCDS share. On a SCDS share deleting files will not create free space. The NetApp device that hosts the SCDS drive maintains a copy of the deleted files in ‘snapshot space’. Hence the quota is consumed by the files in the SCDS share plus the files in ‘snapshot space’.		The SCDS share and the project directory on MARC are two separate storage systems. Data transferred to the <code>/project</code> directory is a second copy of the data on the SCDS share. On a SCDS share deleting files will not create free space. The NetApp device that hosts the SCDS drive maintains a copy of the deleted files in ‘snapshot space’. Hence the quota is consumed by the files in the SCDS share plus the files in ‘snapshot space’.

	=== Temporary file system: /tmp ===		=== Temporary file system: /tmp ===
Line 86:		Line 85:
	There are some complications in using Python on MARC relative to using ARC.		There are some complications in using Python on MARC relative to using ARC.

	Normally, we would recommend installing ~~conda~~ in user's home directory.		Normally, we would recommend installing Conda in user's home directory.
	On MARC, security requirements for working with L4 data require that we block outgoing and incoming internet connections.		On MARC, security requirements for working with Level 4 data require that we block outgoing and incoming internet connections.
	As a result, new packages cannot be downloaded with conda.		As a result, new packages cannot be downloaded with conda.

Line 105:		Line 104:

	== Further Reading ==		== Further Reading ==
	See [[~~Running_jobs~~]] for information on ~~starting~~ a job		See [[Running jobs]] for information on how to submit a job in the HPC cluster.

	[[Category:MARC]]		[[Category:MARC]]
	[[Category:Guides]]		[[Category:Guides]]

Lleung: clarified /tmp usage

2023-09-05T21:18:59Z

clarified /tmp usage

← Older revision		Revision as of 21:18, 5 September 2023
Line 62:		Line 62:
	\|}		\|}

	=== Storage ===		== Storage ==
	About a petabyte of raw disk storage is available to the MARC cluster, but for error checking and performance reasons, the amount of usable storage for researchers' projects is considerably less than that. From a user's perspective, the total amount of storage is less important than the individual storage limits.		About a petabyte of raw disk storage is available to the MARC cluster, but for error checking and performance reasons, the amount of usable storage for researchers' projects is considerably less than that. From a user's perspective, the total amount of storage is less important than the individual storage limits.

	MARC storage is not accessible outside of MARC.		MARC storage is not accessible outside of MARC.

	==== Home file system: /home ====		=== Home file system: /home ===
	~~There is~~ a ~~per-user quota of 25 GB~~ under <code>/home</code>. This limit is fixed and cannot be increased. ~~Each user has a directory under <code>/~~home~~</code>, which is the default working~~ directory ~~when logging in to MARC. It~~ is ~~expected that most researchers will work from <code>/project</code> and only use home~~ for software and ~~such things. <code>/home</code> is expected to be used~~ only ~~for '''L1~~/L2 data~~''' and~~ not ~~for your~~ patient identifiable files. ~~The identifiable files go in the~~ appropriate ~~directory~~ under /project.		Each account on MARC has a home directory under <code>/home</code> with a 25 GB per-user quota. This limit is fixed and cannot be increased. The intended use for the home directory is for software, scripts, and configuration files and must only contain Level 1/2 data only. Do not store patient identifiable files (Level 3/4) in your home directory. Level 3/4 data is only appropriate under <code>/project</code>.

	~~===~~=== Project file system: /project ~~===~~===		=== Project file system: /project ===
	~~Larger~~ projects ~~may~~ have a <code>/project</code> ~~directory and is~~ named after your project ID and will be the same as your SCDS share name. All files related to a project should only be stored within its assigned <code>/project</code> directory. Quotas in <code>/project</code> are somewhat flexible and are assigned based on the project requirements. Please ~~write to~~ support@hpc.ucalgary.ca if you require additional space.		All projects will have a directory under <code>/project</code> named after your project ID and will be the same as your SCDS share name. All files related to a project should only be stored within its assigned <code>/project</code> directory. Quotas in <code>/project</code> are somewhat flexible and are assigned based on the project requirements. Please contact support@hpc.ucalgary.ca if you require additional space.

	The SCDS share and the project directory on MARC are two separate storages. Data transferred to the <code>/project</code> directory is a second copy of the data on the SCDS share. On a SCDS share deleting files will not create free space. The NetApp device that hosts the SCDS drive maintains a copy of the deleted files in ‘snapshot space’. Hence the quota is consumed by the files in the SCDS share plus the files in ‘snapshot space’.		The SCDS share and the project directory on MARC are two separate storages. Data transferred to the <code>/project</code> directory is a second copy of the data on the SCDS share. On a SCDS share deleting files will not create free space. The NetApp device that hosts the SCDS drive maintains a copy of the deleted files in ‘snapshot space’. Hence the quota is consumed by the files in the SCDS share plus the files in ‘snapshot space’.

	~~===~~=== Temporary file system: /tmp ~~===~~===		=== Temporary file system: /tmp ===
	Each node has a temporary storage location available under <code>/tmp</code> that is only accessible from within that node and for ~~that specific~~ job. Data stored in <code>/tmp</code> will be deleted immediately after the job terminates. It is suitable for all levels of data.		Each compute node has a temporary storage location available under <code>/tmp</code> that is only accessible from within that node and for the running job. Data stored in <code>/tmp</code> will be deleted immediately after the job terminates. It is suitable for all levels of data.

			Because the login node is a shared system, you must take care that data stored within <code>/tmp</code> on the login node is restricted to only your account using the appropriate file modes (<code>chmod</code> and <code>chown</code>/<code>chgrp</code> commands) and deleted after use.

	== Software installations ==		== Software installations ==

Lleung: updated storage information and formatting

2023-09-05T21:10:47Z

updated storage information and formatting

← Older revision		Revision as of 21:10, 5 September 2023
Line 16:		Line 16:
	To ensure the security of Level 3/4 data, MARC has implemented several restrictions:		To ensure the security of Level 3/4 data, MARC has implemented several restrictions:

			* '''Account Requirement''': All users must have IT accounts.
			* '''Project ID Requirement''': To use MARC, a project ID is required. This project ID is the same number used on SCDS.
			* '''SSH Access via Citrix''': Access to MARC via SSH must be done through the IT Citrix system. Admin VPN access is neither sufficient nor necessary.
	* '''Internet Access''': Neither compute nodes nor login nodes have internet access.		* '''Internet Access''': Neither compute nodes nor login nodes have internet access.
	* '''Data Ingestion''': All data must be transferred to MARC by first copying it to SCDS (Secure Compute Data Store) and then fetching it from SCDS to MARC.		* '''Data Ingestion via SCDS''': All data must be transferred to MARC by first copying it to SCDS (Secure Compute Data Store) and then fetching it from SCDS to MARC.
	* '''Data Retrieval''': Resulting data, such as analysis outputs, must be copied to SCDS and then fetched from SCDS to its intended destination using established methods.		* '''Data Retrieval via SCDS''': Resulting data, such as analysis outputs, must be copied to SCDS and then fetched from SCDS to its intended destination using established methods.
	* '''Auditing''': All file accesses are logged for auditing purposes.		* '''Data Auditing''': All file accesses are logged for auditing purposes.
	* '''SSH Connections''': Access to MARC via SSH must be done through the IT Citrix system. Admin VPN access is neither sufficient nor necessary.
	* '''Account Requirement''': All users must have IT accounts.
	* '''Project ID''': To use MARC, a project ID is required. This project ID is the same number used on SCDS.

	These measures have been put in place to ensure the security and controlled handling of sensitive data on the MARC cluster and to prevent intentional or accidental data exfiltration.		These measures have been put in place to ensure the security and controlled handling of sensitive data on the MARC cluster and to prevent intentional or accidental data exfiltration.
Line 63:		Line 63:

	=== Storage ===		=== Storage ===
	About a petabyte of raw disk storage is available to the MARC cluster, but for error checking and performance reasons, the amount of usable storage for researchers' projects is considerably less than that. From a user's perspective, the total amount of storage is less important than the individual storage limits~~. As described below, there are two storage areas: home and project.~~		About a petabyte of raw disk storage is available to the MARC cluster, but for error checking and performance reasons, the amount of usable storage for researchers' projects is considerably less than that. From a user's perspective, the total amount of storage is less important than the individual storage limits.

	~~=== Home file system: /home ===~~
	There is a per-user quota of 25 GB under /home. This limit is fixed and cannot be increased. Each user has a directory under /home, which is the default working directory when logging in to MARC. It is expected that most researchers will work from /project and only use home for software and such things. /home is expected to be used only for '''L1/L2 data''' and not for your patient identifiable files. The identifiable files go in the appropriate directory under /project.

	~~=== Project file system for larger projects: /project ===~~
	Directories will be created in /project named after your project ID. This name will be the same as your SCDS share name. The expectation is that all files to do with that project will be stored in /project/projectid. Quotas in /project are somewhat flexible. Please write to support@hpc.ucalgary.ca with an estimate of how much space you will require.

	~~However, the SCDS share and the project directory on~~ MARC ~~are two separate storages~~.		MARC storage is not accessible outside of MARC.

	~~Data transferred to the~~ /~~project directory~~ is a ~~second copy~~ of the ~~data on the SCDS share~~. ~~MARC storage~~ is not ~~accessible outside of MARC~~.		==== Home file system: /home ====
			There is a per-user quota of 25 GB under <code>/home</code>. This limit is fixed and cannot be increased. Each user has a directory under <code>/home</code>, which is the default working directory when logging in to MARC. It is expected that most researchers will work from <code>/project</code> and only use home for software and such things. <code>/home</code> is expected to be used only for '''L1/L2 data''' and not for your patient identifiable files. The identifiable files go in the appropriate directory under /project.

	On a SCDS share ~~deleting~~ files ~~will not create free space. The NetApp device that hosts the SCDS drive maintains~~ a ~~copy of the deleted files in ‘snapshot space’~~. ~~Hence the quota is consumed by the files~~ in the ~~SCDS share plus the files in ‘snapshot space’~~.		====== Project file system: /project ======
			Larger projects may have a <code>/project</code> directory and is named after your project ID and will be the same as your SCDS share name. All files related to a project should only be stored within its assigned <code>/project</code> directory. Quotas in <code>/project</code> are somewhat flexible and are assigned based on the project requirements. Please write to support@hpc.ucalgary.ca if you require additional space.

	~~=== Temporary file system:~~ /~~tmp ===~~		The SCDS share and the project directory on MARC are two separate storages. Data transferred to the <code>/project</code> directory is a second copy of the data on the SCDS share. On a SCDS share deleting files will not create free space. The NetApp device that hosts the SCDS drive maintains a copy of the deleted files in ‘snapshot space’. Hence the quota is consumed by the files in the SCDS share plus the files in ‘snapshot space’.

	Each node has a location ~~called~~ <code>/tmp</code> that is only accessible from within that node. <code>/tmp</code> ~~is a temporary storage. The user has access to <code>/tmp</code> during the run time of a job and can not access <code>/tmp</code> of that node~~ after the job is ~~completed. The user need to ensure that '''no L4~~ data~~''' is left in the /tmp directory. The jobs must clean up /tmp before exiting and any files or directory created in <code>/tmp</code> must not be world readable or writable~~.		====== Temporary file system: /tmp ======
			Each node has a temporary storage location available under <code>/tmp</code> that is only accessible from within that node and for that specific job. Data stored in <code>/tmp</code> will be deleted immediately after the job terminates. It is suitable for all levels of data.

	== Software installations ==		== Software installations ==

Lleung: rewording introduction

2023-09-05T21:01:21Z

rewording introduction

← Older revision		Revision as of 21:01, 5 September 2023
Line 12:		Line 12:

	== Introduction ==		== Introduction ==
	MARC is a cluster ~~comprised~~ of Linux-based computers purchased in 2019		MARC is a cluster of Linux-based computers that were purchased in 2019. It has been specifically designed to meet the security requirements for handling Level 3 and Level 4 classified data, as defined by the University of Calgary Information Security Classification Standard, which you can find here: https://www.ucalgary.ca/policies/files/policies/im010-03-security-standard_0.pdf

	~~The MARC cluster has been designed with controls appropriate for~~ Level 3 ~~and Level~~ 4 ~~classified~~ data~~. The University of Calgary Information Security Classification Standard is published here~~: ~~https://www.ucalgary.ca/policies/files/policies/im010-03-security-standard_0.pdf~~		To ensure the security of Level 3/4 data, MARC has implemented several restrictions:

	~~Due to security requirements for Level 3/4 data, some necessary restrictions have been placed on MARC to prevent accidental (or otherwise) data exfiltration.~~		* '''Internet Access''': Neither compute nodes nor login nodes have internet access.
	* ~~Compute~~ nodes ~~and~~ login nodes have no access ~~to the internet~~		* '''Data Ingestion''': All data must be transferred to MARC by first copying it to SCDS (Secure Compute Data Store) and then fetching it from SCDS to MARC.
	* All data must be ~~ingested~~ to MARC by first copying it to SCDS (Secure Compute Data Store) and then fetching it from SCDS to MARC.		* '''Data Retrieval''': Resulting data, such as analysis outputs, must be copied to SCDS and then fetched from SCDS to its intended destination using established methods.
	* Resulting data (outputs ~~of analyses)~~ must be copied to SCDS and then ~~fetching it~~ from SCDS to ~~wherever it needs to go~~ using established ~~means~~.		* '''Auditing''': All file accesses are logged for auditing purposes.
	* All file accesses are ~~recorded~~ for auditing purposes.		* '''SSH Connections''': Access to MARC via SSH must be done through the IT Citrix system. Admin VPN access is neither sufficient nor necessary.
	* ~~ssh connections~~ to MARC must be through the IT Citrix system (Admin VPN is ~~not~~ sufficient nor necessary)		* '''Account Requirement''': All users must have IT accounts.
	* All ~~accounts~~ must be IT accounts		* '''Project ID''': To use MARC, a project ID is required. This project ID is the same number used on SCDS.
	* A project ID is required ~~to use MARC~~. This project ID is the same number ~~that is~~ used on SCDS
			These measures have been put in place to ensure the security and controlled handling of sensitive data on the MARC cluster and to prevent intentional or accidental data exfiltration.

	== Hardware ==		== Hardware ==
Line 78:		Line 79:
	=== Temporary file system: /tmp ===		=== Temporary file system: /tmp ===

	Each node has a location called /tmp that is only accessible from within that node. /tmp is a temporary storage. The user has access to /tmp during the run time of a job and can not access /tmp of that node after the job is completed. The user need to ensure that '''no L4 data''' is left in the /tmp directory. The jobs must clean up /tmp before exiting and any files or directory created in /tmp must not be world readable or writable.		Each node has a location called <code>/tmp</code> that is only accessible from within that node. <code>/tmp</code> is a temporary storage. The user has access to <code>/tmp</code> during the run time of a job and can not access <code>/tmp</code> of that node after the job is completed. The user need to ensure that '''no L4 data''' is left in the /tmp directory. The jobs must clean up /tmp before exiting and any files or directory created in <code>/tmp</code> must not be world readable or writable.

	== Software installations ==		== Software installations ==
Line 84:		Line 85:
	=== Python ===		=== Python ===

	There are some complications in using Python on MARC relative to using ARC.		There are some complications in using Python on MARC relative to using ARC.

	Normally, we would recommend installing conda in user's home directory.		Normally, we would recommend installing conda in user's home directory.
	On MARC, security requirements for working with L4 data require that we block outgoing and incoming internet connections.		On MARC, security requirements for working with L4 data require that we block outgoing and incoming internet connections.

Lleung: Spell check and formatting

2023-09-05T20:55:51Z

Spell check and formatting

@@ Line 19: / Line 19: @@
 * Compute nodes and login nodes have no access to the internet
 * All data must be ingested to MARC by first copying it to SCDS (Secure Compute Data Store) and then fetching it from SCDS to MARC.
-* Resulting data (outputs of analyses) must be copyied to SCDS and then fetching it from SCDS to wherever it needs to go using established means.
+* Resulting data (outputs of analyses) must be copied to SCDS and then fetching it from SCDS to wherever it needs to go using established means.
 * All file accesses are recorded for auditing purposes.
 * ssh connections to MARC must be through the IT Citrix system (Admin VPN is not sufficient nor necessary)
@@ Line 34: / Line 34: @@
 ** 80-cores: node with 4 sockets.  Each socket has an Intel Xeon Gold 6148 20-core processor, running at 2.4 GHz.
 ** The 80 cores on the node share about 3 TB of RAM (memory), but, jobs should request no more than 3000000 MB.
-=== cpu2019 ===
+=== Storage ===
 About a petabyte of raw disk storage is available to the MARC cluster, but for error checking and performance reasons, the amount of usable storage for researchers' projects is considerably less than that.  From a user's perspective, the total amount of storage is less important than the individual storage limits.  As described below, there are two storage areas: home and project.
@@ Line 94: / Line 101: @@
 ** Run it with singularity
 * Non-open source software which requires a connection to a license server may require admin assistance to set up. contact [mailto:support@hpc.ucalgary.ca support@hpc.ucalgary.ca] for support.
 == Further Reading ==

Tannistha.nandi: /* Project file system for larger projects: /project */

2023-04-14T18:37:24Z

Project file system for larger projects: /project

← Older revision		Revision as of 18:37, 14 April 2023
Line 64:		Line 64:

	However, the SCDS share and the project directory on MARC are two separate storages.		However, the SCDS share and the project directory on MARC are two separate storages.

			Data transferred to the /project directory is a second copy of the data on the SCDS share. MARC storage is not accessible outside of MARC.

	On a SCDS share deleting files will not create free space. The NetApp device that hosts the SCDS drive maintains a copy of the deleted files in ‘snapshot space’. Hence the quota is consumed by the files in the SCDS share plus the files in ‘snapshot space’.		On a SCDS share deleting files will not create free space. The NetApp device that hosts the SCDS drive maintains a copy of the deleted files in ‘snapshot space’. Hence the quota is consumed by the files in the SCDS share plus the files in ‘snapshot space’.

Tannistha.nandi: /* Temporary file system: /tmp */

2023-04-14T18:08:45Z

Temporary file system: /tmp

← Older revision		Revision as of 18:08, 14 April 2023
Line 69:		Line 69:
	=== Temporary file system: /tmp ===		=== Temporary file system: /tmp ===

	Each node has a location called /tmp that is only accessible from within that node. /tmp is a temporary storage. The user has access to /tmp during the run time of a job and can not access /tmp of that node after the job is completed. The user need to ensure that no '''L4 data''' is left in the /tmp directory. The jobs must clean up /tmp before exiting and any files or directory created in /tmp must not be world readable or writable.		Each node has a location called /tmp that is only accessible from within that node. /tmp is a temporary storage. The user has access to /tmp during the run time of a job and can not access /tmp of that node after the job is completed. The user need to ensure that '''no L4 data''' is left in the /tmp directory. The jobs must clean up /tmp before exiting and any files or directory created in /tmp must not be world readable or writable.

	== Software installations ==		== Software installations ==

Tannistha.nandi: /* Temporary file system: /tmp */

2023-04-14T18:07:51Z

Temporary file system: /tmp

← Older revision		Revision as of 18:07, 14 April 2023
Line 69:		Line 69:
	=== Temporary file system: /tmp ===		=== Temporary file system: /tmp ===

	Each node has a location called /tmp that is only accessible from within that node. /tmp is a temporary storage. The user has access to /tmp during the run time of a job and can not access /tmp of that node after the job is completed. The user need to ensure that no '''L4 data''' is left in the /tmp directory.~~Maybe~~ The jobs must clean up /tmp before exiting and any files or directory created in /tmp must not be world readable or writable.		Each node has a location called /tmp that is only accessible from within that node. /tmp is a temporary storage. The user has access to /tmp during the run time of a job and can not access /tmp of that node after the job is completed. The user need to ensure that no '''L4 data''' is left in the /tmp directory. The jobs must clean up /tmp before exiting and any files or directory created in /tmp must not be world readable or writable.

	== Software installations ==		== Software installations ==

Tannistha.nandi: /* Temporary file system: /tmp */

2023-04-14T18:07:17Z

Temporary file system: /tmp

← Older revision		Revision as of 18:07, 14 April 2023
Line 69:		Line 69:
	=== Temporary file system: /tmp ===		=== Temporary file system: /tmp ===

	Each node has a location called /tmp that is only accessible from within that node. /tmp is a temporary storage. The user has access to /tmp during the run time of a job and can not access /tmp of that node after the job is completed. The user need to ensure that no L4 data is left in the /tmp directory.		Each node has a location called /tmp that is only accessible from within that node. /tmp is a temporary storage. The user has access to /tmp during the run time of a job and can not access /tmp of that node after the job is completed. The user need to ensure that no '''L4 data''' is left in the /tmp directory.Maybe The jobs must clean up /tmp before exiting and any files or directory created in /tmp must not be world readable or writable.

	== Software installations ==		== Software installations ==

Tannistha.nandi: /* Temporary file system: /tmp */

2023-04-14T17:59:18Z

Temporary file system: /tmp

← Older revision		Revision as of 17:59, 14 April 2023
Line 69:		Line 69:
	=== Temporary file system: /tmp ===		=== Temporary file system: /tmp ===

	Each node has a location called /tmp that is only accessible from within that node. /tmp is a temporary storage. ~~You can~~ access /tmp during the run time of a job and can not access /tmp of that node after the job is completed. The user need to ensure that no L4 data is left in the /tmp directory.		Each node has a location called /tmp that is only accessible from within that node. /tmp is a temporary storage. The user has access to /tmp during the run time of a job and can not access /tmp of that node after the job is completed. The user need to ensure that no L4 data is left in the /tmp directory.

	== Software installations ==		== Software installations ==