ARC Storage Terms of Use - Revision history

Lleung: Added category and navbox

2023-09-20T22:30:14Z

Added category and navbox

← Older revision		Revision as of 22:30, 20 September 2023
Line 110:		Line 110:

	__NOTOC__		__NOTOC__

			[[Category:ARC]]
			{{Navbox ARC}}

Tthomas at 14:42, 9 July 2021

2021-07-09T14:42:59Z

← Older revision		Revision as of 14:42, 9 July 2021
Line 97:		Line 97:
	* Multiple copies of same data between individual researchers		* Multiple copies of same data between individual researchers
	* Using ARC storage for archiving		* Using ARC storage for archiving
			* Not ensuring your data and derived results are backed up outside of ARC storage
	\|		\|
	* Run a test job, and scale the data requirements appropriately		* Run a test job, and scale the data requirements appropriately

Tthomas: added link for backing up ARC data to OneDrive

2021-07-09T14:31:52Z

added link for backing up ARC data to OneDrive

← Older revision		Revision as of 14:31, 9 July 2021
Line 41:		Line 41:
	\|}		\|}

	Currently all users are responsible for managing their own backups.		Currently all users are responsible for managing their own backups. You can back up data to your personal UofC OneDrive for business cloud storage. see: [[How to transfer data#rclone:%20rsync%20for%20cloud%20storage\|https://rcs.ucalgary.ca/How_to_transfer_data#rclone:_rsync_for_cloud_storage]] This allocation starts at 5TB. Contact the support center for questions regarding OneDrive for Business.

	== Performance ==		== Performance ==

Lleung at 17:32, 2 December 2020

2020-12-02T17:32:30Z

← Older revision		Revision as of 17:32, 2 December 2020
Line 63:		Line 63:
	===/scratch===		===/scratch===
	/scratch is intended for temporary data for the duration of the job.		/scratch is intended for temporary data for the duration of the job.

	Directories in /scratch are created when a user job starts. The naming of the directory is /scratch/JOBID, where JOBID is the job id that is assigned by Slurm. It is expected that jobs clean up the contents of their scratch directory when the job completes, as part of the Slurm batch job. Any files older than 10 days will be automatically deleted.		Directories in /scratch are created when a user job starts. The naming of the directory is /scratch/JOBID, where JOBID is the job id that is assigned by Slurm. It is expected that jobs clean up the contents of their scratch directory when the job completes, as part of the Slurm batch job. Any files older than 10 days will be automatically deleted.

	Each user can store up to a maximum of 15 TB in /scratch. However, due to the shared nature of /scratch, we cannot make any guarantees that the full 15 TB will be available at any given time.		Each user can store up to a maximum of 15 TB in /scratch. However, due to the shared nature of /scratch, we cannot make any guarantees that the full 15 TB will be available at any given time.

	If /scratch becomes more than 75% full, we reserve the right to delete the files as needed.		If /scratch becomes more than 75% full, we reserve the right to delete the files as needed.
	===/work===		===/work===
	/work is intended for projects whose data requirements exceeds the storage allocation for /home. /work is requested by the Principal Investigator (PI) on behalf of the entire group. The request should be made as needed. Please contact us at support@hpc.ucagary.ca for further assistance.		/work is intended for projects whose data requirements exceeds the storage allocation for /home. /work is requested by the Principal Investigator (PI) on behalf of the entire group. The request should be made as needed.

			Please contact us at support@hpc.ucagary.ca for further assistance.

	7-day snapshots are available.		7-day snapshots are available.
	===/bulk===		===/bulk===
	/bulk is intended for large allocations that have lesser I/O needs intended for streaming reads and writes. /bulk is requested by the Principal Investigator (PI) on behalf of the entire group. The request should be made as needed. Please contact us at support@hpc.ucagary.ca for further assistance.		/bulk is intended for large allocations that have lesser I/O needs intended for streaming reads and writes. /bulk is requested by the Principal Investigator (PI) on behalf of the entire group. The request should be made as needed. Please contact us at support@hpc.ucagary.ca for further assistance.

	7-day snapshots are available.		7-day snapshots are available.

	Portion of /bulk file system will be available to scientific instruments on campus, on a request basis.		Portion of /bulk file system will be available to scientific instruments on campus, on a request basis.

	Please contact support@hpc.ucalgary.ca for further assistance.		Please contact support@hpc.ucalgary.ca for further assistance.

Lleung at 17:26, 2 December 2020

2020-12-02T17:26:59Z

← Older revision		Revision as of 17:26, 2 December 2020
Line 48:		Line 48:
	! Work		! Work
	\|-		\|-
	\|Single node streaming R/W 540/730 MB/s		\|
	Aggregate (multi-node) R/W 4140/4400 MB/s		* Single node streaming R/W 540/730 MB/s
	Untar Linux kernel (~39000 files): 3:09 (m:ss)		* Aggregate (multi-node) R/W 4140/4400 MB/s
	\|Single node streaming R/W 620/750 MB/s		* Untar Linux kernel (~39000 files): 3:09 (m:ss)
	Aggregate (multi-node) R/W 6131/5616 MB/s		\|
	Untar Linux kernel (~39000 files): 2:30 (m:ss)		* Single node streaming R/W 620/750 MB/s
			* Aggregate (multi-node) R/W 6131/5616 MB/s
			* Untar Linux kernel (~39000 files): 2:30 (m:ss)
	\|}		\|}

Lleung at 17:26, 2 December 2020

2020-12-02T17:26:36Z

← Older revision		Revision as of 17:26, 2 December 2020
Line 47:		Line 47:
	! Bulk		! Bulk
	! Work		! Work
			\|-
	\|Single node streaming R/W 540/730 MB/s		\|Single node streaming R/W 540/730 MB/s
	Aggregate (multi-node) R/W 4140/4400 MB/s		Aggregate (multi-node) R/W 4140/4400 MB/s

Lleung: Created page with "There are several options for disk storage on the ARC Cluster. Please review this section carefully to decide where to place your data. Contact systems staff at support@hpc.uc..."

2020-12-02T17:26:17Z

Created page with "There are several options for disk storage on the ARC Cluster. Please review this section carefully to decide where to place your data. Contact systems staff at support@hpc.uc..."

New page

There are several options for disk storage on the ARC Cluster. Please review this section carefully to decide where to place your data. Contact systems staff at support@hpc.ucalgary.ca if you have any questions.
As this is a limited resource, please use the space responsibly. Disk space on the ARC Cluster should not be used as archival storage, nor should it be used as a backup (2nd copy location) for other systems (e.g. desktops, laptops, etc.).

== Summary of File Storage Options ==
{| class=wikitable
! File System !! Type !! Snapshots !! Backups !! Quota
|-
! /home
| NetApp
FAS8200, NFS
| 7 days
| No
| 500 GB/user
|-
! /scratch
| NetApp
FAS8200, NFS
| None
| No
| 15 TB/user
|-
! /work
| NetApp
FAS8200, NFS
| 7 days
| No
| By request/group
|-
! /bulk
| NetApp
FAS2720, NFS
| 7 days
| No
| By request/group
|-
! /tmp
| Local
| None
| No
| 100 GB
|}

Currently all users are responsible for managing their own backups.

== Performance ==
{| class=wikitable
! Bulk
! Work
|Single node streaming R/W 540/730 MB/s
Aggregate (multi-node) R/W 4140/4400 MB/s
Untar Linux kernel (~39000 files): 3:09 (m:ss)
|Single node streaming R/W 620/750 MB/s
Aggregate (multi-node) R/W 6131/5616 MB/s
Untar Linux kernel (~39000 files): 2:30 (m:ss)
|}

===/home===
Each user has a home directory called /home/username. The /home file system has a quota of 500 GB per user which cannot be increased, and seven days of snapshots.
===/scratch===
/scratch is intended for temporary data for the duration of the job.
Directories in /scratch are created when a user job starts. The naming of the directory is /scratch/JOBID, where JOBID is the job id that is assigned by Slurm. It is expected that jobs clean up the contents of their scratch directory when the job completes, as part of the Slurm batch job. Any files older than 10 days will be automatically deleted.
Each user can store up to a maximum of 15 TB in /scratch. However, due to the shared nature of /scratch, we cannot make any guarantees that the full 15 TB will be available at any given time.
If /scratch becomes more than 75% full, we reserve the right to delete the files as needed.
===/work===
/work is intended for projects whose data requirements exceeds the storage allocation for /home. /work is requested by the Principal Investigator (PI) on behalf of the entire group. The request should be made as needed. Please contact us at support@hpc.ucagary.ca for further assistance.
7-day snapshots are available.
===/bulk===
/bulk is intended for large allocations that have lesser I/O needs intended for streaming reads and writes. /bulk is requested by the Principal Investigator (PI) on behalf of the entire group. The request should be made as needed. Please contact us at support@hpc.ucagary.ca for further assistance.
7-day snapshots are available.
Portion of /bulk file system will be available to scientific instruments on campus, on a request basis.
Please contact support@hpc.ucalgary.ca for further assistance.

==Best Practices==

{| class=wikitable
! Bad
! Good
|-
|
* Submitting many jobs without a good understanding of how much data it would generate
* Directories with millions of files
* Excessively long file names Short descriptive names for files
* The /bulk or /work file systems should not be used as a temporary directory for applications
* Multiple copies of same data between individual researchers
* Using ARC storage for archiving
|
* Run a test job, and scale the data requirements appropriately
* Use appropriate file formats or tools that reduce the number of files
* Use /scratch or /tmp file systems instead
* Request for a shared space
* Contact support@hpc.ucalgary.ca to discuss requirements
|}

Please note that we reserve the right to perform emergency and critical maintenance at any moment with minimal warning.

__NOTOC__