Transferring Data from CHGI - Revision history

Lleung: /* Starting a File Transfer */

2020-10-28T21:31:39Z

Starting a File Transfer

← Older revision		Revision as of 21:31, 28 October 2020
Line 95:		Line 95:
	Before initiating a file transfer, ensure that you have the proper read permissions to your CHGI files:		Before initiating a file transfer, ensure that you have the proper read permissions to your CHGI files:
	# Log in to the ARC DTN via SSH at arc-dtn.ucalgary.ca.		# Log in to the ARC DTN via SSH at arc-dtn.ucalgary.ca.
	# Ensure that you have read permissions to the files you are interested. Verify this by navigating to the filesystem based on the mapping shown above and try listing or reading your files.		# Ensure that you have read permissions to the files you are interested in. Verify this by navigating to the filesystem based on the mapping shown above and try listing or reading your files.
	#* If you do not have read permission or have any difficulties, please contact us at support@hpc.ucalgary.ca.		#* If you do not have read permission or have any difficulties, please contact us at support@hpc.ucalgary.ca.
	#* If you have read permissions, continue with instructions below.		#* If you have read permissions, continue with instructions below.

Lleung: /* Transferring files with rsync within screen */

2020-10-28T21:28:18Z

Transferring files with rsync within screen

← Older revision		Revision as of 21:28, 28 October 2020
Line 99:		Line 99:
	#* If you have read permissions, continue with instructions below.		#* If you have read permissions, continue with instructions below.

			Log in to ARC DTN via SSH. If you are unable to connect, ensure that you are [https://ucalgary.service-now.com/kb_view.do?sysparm_article=KB0032795 connected to the University of Calgary General VPN]. Once connected, start a new screen session:
			<syntaxhighlight lang="text">
			# Connect to arc-dtn.ucalgary.ca.
			$ ssh user@arc-dtn.ucalgary.ca

	~~Log in to ARC DTN and start a new screen session:~~
	~~<syntaxhighlight lang="text">~~
	# Start a new screen session named 'transfer'		# Start a new screen session named 'transfer'
	$ screen -S transfer		$ screen -S transfer

Lleung at 21:24, 28 October 2020

2020-10-28T21:24:35Z

← Older revision		Revision as of 21:24, 28 October 2020
Line 4:		Line 4:
	Users needing to migrate their data from CHGI to ARC are able to do so through the read-only NFS mounts that have been set up on the ARC DTN node. The NFS mounts will automatically take advantage of the dedicated 10Gbit fibre connection to help maximize your transfer speed. Access to the NFS filesystems are restricted to authorized CHGI group members. Please contact support@hpc.ucalgary.ca if you have difficulty accessing your files.		Users needing to migrate their data from CHGI to ARC are able to do so through the read-only NFS mounts that have been set up on the ARC DTN node. The NFS mounts will automatically take advantage of the dedicated 10Gbit fibre connection to help maximize your transfer speed. Access to the NFS filesystems are restricted to authorized CHGI group members. Please contact support@hpc.ucalgary.ca if you have difficulty accessing your files.

	Most filesystems at CHGI have been made available under the <code>/external</code> mount point on ARC DTN. Please refer to the following ~~table~~ to help determine your filesystem path on ARC-DTN.		Most filesystems at CHGI have been made available under the <code>/external</code> mount point on ARC DTN. Please refer to the following tables to help determine your filesystem path on ARC-DTN.

	<table>		<table>

Lleung at 21:23, 28 October 2020

2020-10-28T21:23:48Z

← Older revision		Revision as of 21:23, 28 October 2020
Line 1:		Line 1:
	[[File:ARCDTN-CHGI-Mapping.jpeg\|350px\|thumb\|right\|ARC-DTN and CHGI NFS mounts]]		[[File:ARCDTN-CHGI-Mapping.jpeg\|350px\|thumb\|right\|ARC-DTN and CHGI NFS mounts]]
	Due to the large datasets that are currently stored at the Center for Health Genomics and Informatics (CHGI), we have set up a dedicated 10Gbit fibre connection between the ARC DTN and CHGI to help users quickly transfer files as part of the [[CHGI Transition]].		Due to the large datasets that are currently stored at the Center for Health Genomics and Informatics (CHGI), we have set up a dedicated 10Gbit fibre connection between the ARC DTN (Data Transfer Node) and CHGI to help users quickly transfer files as part of the [[CHGI Transition]].

	Users needing to migrate their data from CHGI to ARC are able to do so through the read-only NFS mounts that have been set up on the ARC DTN node. The NFS mounts will automatically take advantage of the dedicated 10Gbit fibre connection to help maximize your transfer speed. Access to the NFS filesystems are restricted to authorized CHGI group members. Please contact support@hpc.ucalgary.ca if you have difficulty accessing your files.		Users needing to migrate their data from CHGI to ARC are able to do so through the read-only NFS mounts that have been set up on the ARC DTN node. The NFS mounts will automatically take advantage of the dedicated 10Gbit fibre connection to help maximize your transfer speed. Access to the NFS filesystems are restricted to authorized CHGI group members. Please contact support@hpc.ucalgary.ca if you have difficulty accessing your files.

Lleung at 21:19, 28 October 2020

2020-10-28T21:19:07Z

← Older revision		Revision as of 21:19, 28 October 2020
Line 2:		Line 2:
	Due to the large datasets that are currently stored at the Center for Health Genomics and Informatics (CHGI), we have set up a dedicated 10Gbit fibre connection between the ARC DTN and CHGI to help users quickly transfer files as part of the [[CHGI Transition]].		Due to the large datasets that are currently stored at the Center for Health Genomics and Informatics (CHGI), we have set up a dedicated 10Gbit fibre connection between the ARC DTN and CHGI to help users quickly transfer files as part of the [[CHGI Transition]].

	Users needing to migrate their data from ~~the~~ CHGI to ARC are able to do so through the read-only NFS mounts that have been set up on the ARC DTN node. The NFS mounts will automatically take advantage of the dedicated 10Gbit fibre connection to help maximize your transfer speed. Access to the NFS filesystems are restricted to authorized CHGI group members. Please contact support@hpc.ucalgary.ca if you have difficulty accessing your files.		Users needing to migrate their data from CHGI to ARC are able to do so through the read-only NFS mounts that have been set up on the ARC DTN node. The NFS mounts will automatically take advantage of the dedicated 10Gbit fibre connection to help maximize your transfer speed. Access to the NFS filesystems are restricted to authorized CHGI group members. Please contact support@hpc.ucalgary.ca if you have difficulty accessing your files.

	Most filesystems at CHGI have been made available under the <code>/external</code> mount point on ARC DTN. Please refer to the following table to help determine your filesystem path on ARC-DTN.		Most filesystems at CHGI have been made available under the <code>/external</code> mount point on ARC DTN. Please refer to the following table to help determine your filesystem path on ARC-DTN.

Lleung at 21:18, 28 October 2020

2020-10-28T21:18:40Z

← Older revision		Revision as of 21:18, 28 October 2020
Line 51:		Line 51:
	{\| class="wikitable"		{\| class="wikitable"
	! colspan="2" \| Tiered		! colspan="2" \| Tiered
			\|-
	! ARC DTN		! ARC DTN
	! CHGI		! CHGI

Lleung at 21:18, 28 October 2020

2020-10-28T21:18:30Z

← Older revision		Revision as of 21:18, 28 October 2020
Line 9:		Line 9:
	<tr><td>		<tr><td>
	{\| class="wikitable"		{\| class="wikitable"
			! colspan="2" \| GPFS
			\|-
	! ARC DTN		! ARC DTN
	! CHGI		! CHGI
Line 48:		Line 50:
	</td><td style="vertical-align: top;">		</td><td style="vertical-align: top;">
	{\| class="wikitable"		{\| class="wikitable"
			! colspan="2" \| Tiered
	! ARC DTN		! ARC DTN
	! CHGI		! CHGI

Lleung at 21:16, 28 October 2020

2020-10-28T21:16:03Z

← Older revision		Revision as of 21:16, 28 October 2020
Line 6:		Line 6:
	Most filesystems at CHGI have been made available under the <code>/external</code> mount point on ARC DTN. Please refer to the following table to help determine your filesystem path on ARC-DTN.		Most filesystems at CHGI have been made available under the <code>/external</code> mount point on ARC DTN. Please refer to the following table to help determine your filesystem path on ARC-DTN.

			<table>
	~~{{Clear}}~~		<tr><td>
	<~~div class="row"~~>
	<~~div class="col-md-6"~~>
	{\| class="wikitable"		{\| class="wikitable"
	! ARC DTN		! ARC DTN
Line 48:		Line 46:
	\| /external/gpfs/vetmed_stage \|\| /gpfs/vetmed_stage		\| /external/gpfs/vetmed_stage \|\| /gpfs/vetmed_stage
	\|}		\|}
	</~~div~~>		</td><td style="vertical-align: top;">
	<~~div class~~="~~col~~-~~md-6~~">
	{\| class="wikitable"		{\| class="wikitable"
	! ARC DTN		! ARC DTN
Line 80:		Line 77:
	\| /external/tiered/snyder_data \|\| /tiered/snyder_data		\| /external/tiered/snyder_data \|\| /tiered/snyder_data
	\|}		\|}
	</~~div~~>		</td></tr></table>
	~~{{Clear}}~~

	~~To get started with your data migration:~~
	~~# Log in to the ARC DTN via SSH at arc-dtn.ucalgary.ca.~~
	~~# Ensure that you have read permissions to the files you are interested. Verify this by navigating to the filesystem based on the mapping shown above and try listing or reading your files.~~
	~~#* If you do not have read permission or have any difficulties, please contact us at support@hpc.ucalgary.ca.~~
	~~#* If you have read permissions, you may follow the instructions at [[How to transfer data]] or read the sections for help on initiating a file transfer.~~

	When transferring your files from CHGI to ARC, please remember that:		When transferring your files from CHGI to ARC, please remember that:
	* Your ARC home directory has a 500GB quota. ~~If you are transferring files from your~~ CHGI home directory to your ~~ARC home~~ directory~~, ensure you have sufficient disk space~~.		* Your ARC home directory has a 500GB quota. Your CHGI home directory may have significantly more data than this quota. Please transfer large datasets to your group's work directory instead.
	* File permissions via NFS are made possible using the CHGI group permissions. Your files at CHGI must be group readable in order for them to be accessible via the NFS mount.		* File permissions via NFS are made possible using the CHGI group permissions. Your files at CHGI must be group readable in order for them to be accessible via the NFS mount.

Line 98:		Line 88:
	=== Transferring files with rsync within screen ===		=== Transferring files with rsync within screen ===
	rsync is a file transfer utility which you can use to help transfer files from one location to another. Since processes spawned by your session typically terminates when you log out, we will need to use a terminal multiplexer program (such as screen or tmux) to help ensure your processes remain running after you log out. In this section, we will show you how to run rsync within a screen session to ensure your file transfers remain uninterrupted when you log out.		rsync is a file transfer utility which you can use to help transfer files from one location to another. Since processes spawned by your session typically terminates when you log out, we will need to use a terminal multiplexer program (such as screen or tmux) to help ensure your processes remain running after you log out. In this section, we will show you how to run rsync within a screen session to ensure your file transfers remain uninterrupted when you log out.

			Before initiating a file transfer, ensure that you have the proper read permissions to your CHGI files:
			# Log in to the ARC DTN via SSH at arc-dtn.ucalgary.ca.
			# Ensure that you have read permissions to the files you are interested. Verify this by navigating to the filesystem based on the mapping shown above and try listing or reading your files.
			#* If you do not have read permission or have any difficulties, please contact us at support@hpc.ucalgary.ca.
			#* If you have read permissions, continue with instructions below.


	Log in to ARC DTN and start a new screen session:		Log in to ARC DTN and start a new screen session:

Lleung at 21:09, 28 October 2020

2020-10-28T21:09:20Z

← Older revision		Revision as of 21:09, 28 October 2020
Line 7:		Line 7:


			{{Clear}}
	<div class="row">		<div class="row">
	<div class="col-md-6">		<div class="col-md-6">
Line 80:		Line 81:
	\|}		\|}
	</div>		</div>
			{{Clear}}

	To get started with your data migration:		To get started with your data migration:

Lleung at 21:09, 28 October 2020

2020-10-28T21:09:00Z

← Older revision		Revision as of 21:09, 28 October 2020
Line 5:		Line 5:

	Most filesystems at CHGI have been made available under the <code>/external</code> mount point on ARC DTN. Please refer to the following table to help determine your filesystem path on ARC-DTN.		Most filesystems at CHGI have been made available under the <code>/external</code> mount point on ARC DTN. Please refer to the following table to help determine your filesystem path on ARC-DTN.


			<div class="row">
			<div class="col-md-6">
	{\| class="wikitable"		{\| class="wikitable"
	! ARC DTN		! ARC DTN
Line 41:		Line 45:
	\| /external/gpfs/vetmed_data \|\| /gpfs/vetmed_data		\| /external/gpfs/vetmed_data \|\| /gpfs/vetmed_data
	\|-		\|-
	\| /external/gpfs/vetmed_stage \|\| /gpfs/vetmed_stage		\| /external/gpfs/vetmed_stage \|\| /gpfs/vetmed_stage
			\|}
			</div>
			<div class="col-md-6">
			{\| class="wikitable"
			! ARC DTN
			! CHGI
	\|-		\|-
	\| /external/tiered/achri_data \|\| /tiered/achri_data		\| /external/tiered/achri_data \|\| /tiered/achri_data
Line 67:		Line 77:
	\| /external/tiered/smorrissy \|\| /tiered/smorrissy		\| /external/tiered/smorrissy \|\| /tiered/smorrissy
	\|-		\|-
	\| /external/tiered/snyder_data \|\| /tiered/snyder_data		\| /external/tiered/snyder_data \|\| /tiered/snyder_data
	\|}		\|}
			</div>

	To get started with your data migration:		To get started with your data migration: