Difference between revisions of "How to transfer data"

From RCSWiki
Jump to navigation Jump to search
Line 1: Line 1:
 
 
{{Message Box
 
{{Message Box
 
|title=Data Transfer Nodes
 
|title=Data Transfer Nodes
Line 5: Line 4:
 
For performance and resource reasons, file transfers should be performed on the data transfer node arc-dtn.ucalgary.ca rather than on the the ARC login node.  Since the ARC DTN has the same shares as ARC, any files you transfer to the DTN will also be available on ARC.
 
For performance and resource reasons, file transfers should be performed on the data transfer node arc-dtn.ucalgary.ca rather than on the the ARC login node.  Since the ARC DTN has the same shares as ARC, any files you transfer to the DTN will also be available on ARC.
 
}}
 
}}
 
== Transferring Large Datasets ==
 
=== Using screen and rsync ===
 
If you want to transfer a large amount of data from a remote Unix system to ARC you can use '''rsync'' to handle the transfer.
 
However, you will have to keep your SSH session from your workstation alive during the entire transfer.
 
Very often this is not convenient or not feasible.
 
 
To overcome this one can run the '''rsync''' transfer inside a '''screen''' virtual session on ARC.
 
'''screen''' creates an SSH session local to ARC and allows for reconnection from SSH sessions from your workstation.
 
 
To initialize
 
<pre>
 
# Login to ARC
 
$ ssh username@arc.ucalgary.ca
 
 
# Start a screen session
 
$ screen
 
 
# While in the new screen session, start the transfer with rsync.
 
$ rsync -axv ext_user@external.system:path/to/remote/data  .
 
 
# Disconnect from the screen session with the hotkey 'Ctrl-a d'
 
# You may now disconnect from ARC or close the lid of you laptop or turn off the computer.
 
</pre>
 
 
To check if the transfer has been finished.
 
<pre>
 
# Login to ARC
 
$ ssh username@arc.ucalgary.ca
 
 
# Reconnect to the screen session
 
$ screen -r
 
 
# If the transfer has been finished close the screen session.
 
$ exit
 
 
# If the transfer is still running, disconnect from the screen session with the hotkey 'Ctrl-a d'
 
</pre>
 
 
=== Very large files ===
 
If the files are large and the transfer speed is low the transfer may fail before the file has been transferred.
 
'''rsync''' may not be of help here, as it will not restart the file transfer (have not tested recently).
 
 
The solution may be to split the large file into smaller chunks, transfer them using rsync and then join them on the remote system (ARC for example):
 
 
<pre>
 
# Large file is 506MB in this example.
 
$ ls -l t.bin
 
-rw-r--r-- 1 drozmano drozmano 530308481 Jun  8 11:06 t.bin
 
 
# split the file:
 
$ split -b 100M t.bin t.bin_chunk.
 
 
# Check the chunks.
 
$ ls -l t.bin_chunk.*
 
-rw-r--r-- 1 drozmano drozmano 104857600 Jun  8 11:09 t.bin_chunk.aa
 
-rw-r--r-- 1 drozmano drozmano 104857600 Jun  8 11:09 t.bin_chunk.ab
 
-rw-r--r-- 1 drozmano drozmano 104857600 Jun  8 11:09 t.bin_chunk.ac
 
-rw-r--r-- 1 drozmano drozmano 104857600 Jun  8 11:09 t.bin_chunk.ad
 
-rw-r--r-- 1 drozmano drozmano 104857600 Jun  8 11:09 t.bin_chunk.ae
 
-rw-r--r-- 1 drozmano drozmano  6020481 Jun  8 11:09 t.bin_chunk.af
 
 
# Transfer the files:
 
$ rsync -axv t.bin_chunks.* username@arc.ucalgary.ca:
 
</pre>
 
 
Then login to ARC and join the files:
 
<pre>
 
$ cat t.bin_chunk.* > t.bin
 
 
$ ls -l
 
-rw-r--r-- 1 drozmano drozmano 530308481 Jun  8 11:06 t.bin
 
</pre>
 
Success.
 
  
  
== File Transfer Tools ==
+
= Command Line File Transfer Tools =
=== Command-line based ===
 
 
You may use the following command-line file transfer utilities on Linux, MacOS, and Windows. File transfers using these methods require your computer to be on the University of Calgary campus network or via the University of Calgary IT General VPN.
 
You may use the following command-line file transfer utilities on Linux, MacOS, and Windows. File transfers using these methods require your computer to be on the University of Calgary campus network or via the University of Calgary IT General VPN.
  
 
If you are working on a Windows computer, you will need to install these utilities separately as they are not installed by default. Newer versions of Windows 10 (1903 and up) have '''SSH''' built-in as part of the '''openssh''' package. However, you may be better off using one of the [[#GUI File Transfer]] tools listed in the following section.
 
If you are working on a Windows computer, you will need to install these utilities separately as they are not installed by default. Newer versions of Windows 10 (1903 and up) have '''SSH''' built-in as part of the '''openssh''' package. However, you may be better off using one of the [[#GUI File Transfer]] tools listed in the following section.
  
==== <code>scp</code>: Secure Copy ====
+
== <code>scp</code>: Secure Copy ==
 
<code>scp</code> is a secure and encrypted method of transferring files between machines via SSH. It is available on Linux and Mac computers by default and can be installed on Windows by installing the OpenSSH package.
 
<code>scp</code> is a secure and encrypted method of transferring files between machines via SSH. It is available on Linux and Mac computers by default and can be installed on Windows by installing the OpenSSH package.
  
Line 100: Line 24:
 
You may see all the available options with <code>scp</code> by viewing the [http://man7.org/linux/man-pages/man1/scp.1.html man page].
 
You may see all the available options with <code>scp</code> by viewing the [http://man7.org/linux/man-pages/man1/scp.1.html man page].
  
===== Example Usage =====
+
=== Example Usage ===
 
Common operations are given below. On your desktop, to:
 
Common operations are given below. On your desktop, to:
 
* Transfer a single file (eg. <code>data.dat</code>) to ARC: <syntaxhighlight lang="bash">
 
* Transfer a single file (eg. <code>data.dat</code>) to ARC: <syntaxhighlight lang="bash">
Line 112: Line 36:
 
</syntaxhighlight>
 
</syntaxhighlight>
  
==== <code>rsync</code> ====
+
== <code>rsync</code> ==
 
<code>rsync</code> is a utility for transferring and synchronizing files efficiently. The efficiency for its file synchronization is achieved by its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination.   
 
<code>rsync</code> is a utility for transferring and synchronizing files efficiently. The efficiency for its file synchronization is achieved by its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination.   
  
Line 126: Line 50:
 
You may see all the available options with <code>rsync</code> by viewing the [http://man7.org/linux/man-pages/man1/rsync.1.html man page].
 
You may see all the available options with <code>rsync</code> by viewing the [http://man7.org/linux/man-pages/man1/rsync.1.html man page].
  
===== Example Usage =====
+
=== Example Usage ===
 
Common operations are given below. On your desktop, to:
 
Common operations are given below. On your desktop, to:
  
Line 149: Line 73:
 
</syntaxhighlight>
 
</syntaxhighlight>
  
==== <code>sftp</code>: secure file transfer protocol ====
+
== <code>sftp</code>: secure file transfer protocol ==
  
 
* Manual page on-line: http://man7.org/linux/man-pages/man1/sftp.1.html
 
* Manual page on-line: http://man7.org/linux/man-pages/man1/sftp.1.html
Line 163: Line 87:
 
Commands are case insensitive.
 
Commands are case insensitive.
  
==== <code>rclone</code>: rsync for cloud storage ====
+
== <code>rclone</code>: rsync for cloud storage ==
 
'''Rclone''' is a command line program to sync files and directories to and from a number of on-line storage services.
 
'''Rclone''' is a command line program to sync files and directories to and from a number of on-line storage services.
  
 
* https://rclone.org/
 
* https://rclone.org/
  
=== Graphical based ===
+
= Graphical File Transfer Tools =
==== FileZilla ====
+
== FileZilla ==
 
FileZilla is a free cross-platform file transfer program that can transfer files via sFTP.  
 
FileZilla is a free cross-platform file transfer program that can transfer files via sFTP.  
  
 
https://filezilla-project.org/download.php?type=client (Note: Installer may bundle ads and unwanted software. Be careful when clicking through.)
 
https://filezilla-project.org/download.php?type=client (Note: Installer may bundle ads and unwanted software. Be careful when clicking through.)
  
==== MobaXterm (Windows) ====
+
== MobaXterm (Windows) ==
 
'''MobaXterm''' is the recommended tool for remote access and data transfer in '''Windows''' OSes.
 
'''MobaXterm''' is the recommended tool for remote access and data transfer in '''Windows''' OSes.
  
Line 184: Line 108:
 
* Website: https://mobaxterm.mobatek.net/
 
* Website: https://mobaxterm.mobatek.net/
  
==== WinSCP (Windows) ====
+
== WinSCP (Windows) ==
 
WinSCP is a free Windows file transfer tool.
 
WinSCP is a free Windows file transfer tool.
  
 
https://winscp.net/eng/index.php
 
https://winscp.net/eng/index.php
  
==== Globus File Transfer ====
+
= Cloud based Fire Transfer Servcies =
 +
== Globus File Transfer ==
 
Globus File Transfer is a cloud based service for file transfer and file sharing. It uses GridFTP for high speed and reliable data transfers<br>
 
Globus File Transfer is a cloud based service for file transfer and file sharing. It uses GridFTP for high speed and reliable data transfers<br>
  
===== How to get started =====
+
=== How to get started ===
 
1. Navigate to the web page https://www.globusid.org<br>
 
1. Navigate to the web page https://www.globusid.org<br>
 
2. Create a Globus ID and password using your UCalgary email address.  
 
2. Create a Globus ID and password using your UCalgary email address.  
  
===== Use Globus Web Application to transfer files =====
+
=== Use Globus Web Application to transfer files ===
 
1. To initiate data transfers using the Globus Web Application, navigate to https://www.globusid.org/login and log into your Globus account<br>
 
1. To initiate data transfers using the Globus Web Application, navigate to https://www.globusid.org/login and log into your Globus account<br>
 
2. On the left panel, click on File Manager to define/select the endpoints. For example, to transfer data from ARC cluster (endpoint 1) to Compute Canada cedar cluster (endpoint 2)  
 
2. On the left panel, click on File Manager to define/select the endpoints. For example, to transfer data from ARC cluster (endpoint 1) to Compute Canada cedar cluster (endpoint 2)  
Line 202: Line 127:
 
* Next, for Compute Canada cedar data transfer node choose 'endpoint 2' as 'computecanada#cedar-dtn' from the drop down menu. Again authenticate your access using Compute Canada credentials. Navigate to the location where you want to transfer the file<br>
 
* Next, for Compute Canada cedar data transfer node choose 'endpoint 2' as 'computecanada#cedar-dtn' from the drop down menu. Again authenticate your access using Compute Canada credentials. Navigate to the location where you want to transfer the file<br>
 
* Select the file to be transferred from 'endpoint 1' and initiate the transfer process.
 
* Select the file to be transferred from 'endpoint 1' and initiate the transfer process.
 +
 +
= Special cases =
 +
 +
== Transferring Large Datasets ==
 +
=== Using screen and rsync ===
 +
If you want to transfer a large amount of data from a remote Unix system to ARC you can use '''rsync'' to handle the transfer.
 +
However, you will have to keep your SSH session from your workstation alive during the entire transfer.
 +
Very often this is not convenient or not feasible.
 +
 +
To overcome this one can run the '''rsync''' transfer inside a '''screen''' virtual session on ARC.
 +
'''screen''' creates an SSH session local to ARC and allows for reconnection from SSH sessions from your workstation.
 +
 +
To initialize
 +
<pre>
 +
# Login to ARC
 +
$ ssh username@arc.ucalgary.ca
 +
 +
# Start a screen session
 +
$ screen
 +
 +
# While in the new screen session, start the transfer with rsync.
 +
$ rsync -axv ext_user@external.system:path/to/remote/data  .
 +
 +
# Disconnect from the screen session with the hotkey 'Ctrl-a d'
 +
# You may now disconnect from ARC or close the lid of you laptop or turn off the computer.
 +
</pre>
 +
 +
To check if the transfer has been finished.
 +
<pre>
 +
# Login to ARC
 +
$ ssh username@arc.ucalgary.ca
 +
 +
# Reconnect to the screen session
 +
$ screen -r
 +
 +
# If the transfer has been finished close the screen session.
 +
$ exit
 +
 +
# If the transfer is still running, disconnect from the screen session with the hotkey 'Ctrl-a d'
 +
</pre>
 +
 +
=== Very large files ===
 +
If the files are large and the transfer speed is low the transfer may fail before the file has been transferred.
 +
'''rsync''' may not be of help here, as it will not restart the file transfer (have not tested recently).
 +
 +
The solution may be to split the large file into smaller chunks, transfer them using rsync and then join them on the remote system (ARC for example):
 +
 +
<pre>
 +
# Large file is 506MB in this example.
 +
$ ls -l t.bin
 +
-rw-r--r-- 1 drozmano drozmano 530308481 Jun  8 11:06 t.bin
 +
 +
# split the file:
 +
$ split -b 100M t.bin t.bin_chunk.
 +
 +
# Check the chunks.
 +
$ ls -l t.bin_chunk.*
 +
-rw-r--r-- 1 drozmano drozmano 104857600 Jun  8 11:09 t.bin_chunk.aa
 +
-rw-r--r-- 1 drozmano drozmano 104857600 Jun  8 11:09 t.bin_chunk.ab
 +
-rw-r--r-- 1 drozmano drozmano 104857600 Jun  8 11:09 t.bin_chunk.ac
 +
-rw-r--r-- 1 drozmano drozmano 104857600 Jun  8 11:09 t.bin_chunk.ad
 +
-rw-r--r-- 1 drozmano drozmano 104857600 Jun  8 11:09 t.bin_chunk.ae
 +
-rw-r--r-- 1 drozmano drozmano  6020481 Jun  8 11:09 t.bin_chunk.af
 +
 +
# Transfer the files:
 +
$ rsync -axv t.bin_chunks.* username@arc.ucalgary.ca:
 +
</pre>
 +
 +
Then login to ARC and join the files:
 +
<pre>
 +
$ cat t.bin_chunk.* > t.bin
 +
 +
$ ls -l
 +
-rw-r--r-- 1 drozmano drozmano 530308481 Jun  8 11:06 t.bin
 +
</pre>
 +
Success.
  
 
[[Category:Guides]]
 
[[Category:Guides]]

Revision as of 16:25, 25 January 2021

Information Icon.png

Data Transfer Nodes

For performance and resource reasons, file transfers should be performed on the data transfer node arc-dtn.ucalgary.ca rather than on the the ARC login node. Since the ARC DTN has the same shares as ARC, any files you transfer to the DTN will also be available on ARC.


Command Line File Transfer Tools

You may use the following command-line file transfer utilities on Linux, MacOS, and Windows. File transfers using these methods require your computer to be on the University of Calgary campus network or via the University of Calgary IT General VPN.

If you are working on a Windows computer, you will need to install these utilities separately as they are not installed by default. Newer versions of Windows 10 (1903 and up) have SSH built-in as part of the openssh package. However, you may be better off using one of the #GUI File Transfer tools listed in the following section.

scp: Secure Copy

scp is a secure and encrypted method of transferring files between machines via SSH. It is available on Linux and Mac computers by default and can be installed on Windows by installing the OpenSSH package.

The general format for the command is:

$ scp [options] source destination
  • The source and destination fields can be a local file / directory or a remote one.
  • The local location is a normal Unix path, absolute or relative and
  • The remote location has a format username@remote.system.name:file/path.
  • The remote relative file path is relative to the home directory of the username on the remote system.

You may see all the available options with scp by viewing the man page.

Example Usage

Common operations are given below. On your desktop, to:

  • Transfer a single file (eg. data.dat) to ARC:
    desktop$ scp data.dat username@arc-dtn.ucalgary.ca:/desired/destination
    
  • Transfer all files ending with .dat to ARC:
    desktop$ scp *.dat username@arc-dtn.ucalgary.ca:/desired/destination
    
  • To transfer an entire directory to ARC:
    desktop$ scp -r my_data_directory/ username@arc-dtn.ucalgary.ca:/desired/destination
    

rsync

rsync is a utility for transferring and synchronizing files efficiently. The efficiency for its file synchronization is achieved by its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination.

rsync can be used to copy files and directories locally on a system or between multiple computers via SSH. Unlike scp. Because it is designed to synchronize two locations, partial transfers can be restarted by re-running rsync without losing progress. Resuming a partial transfer is not possible with scp.

The general format for the command is similar to scp:

$ rsync [options] source destination
  • The source and destination fields can be a local file / directory or a remote one.
  • The local location is a normal Unix path, absolute or relative and
  • The remote location has a format username@remote.system.name:file/path.
  • The remote relative file path is relative to the home directory of the username on the remote system.

You may see all the available options with rsync by viewing the man page.

Example Usage

Common operations are given below. On your desktop, to:

  • Upload a single file (eg. data.dat) from your workstation to your ARC:
    desktop$ rsync -v data.dat username@arc-dtn.ucalgary.ca:/desired/destination
    
  • Upload all files matching a wildcard (eg. ending in *.dat):
     $ rsync -v *.dat username@arc-dtn.ucalgary.ca:/desired/destination
    
  • Upload an entire directory (eg. my_data to ~/projects/project2):
     $ rsync -axv my_data username@arc-dtn.ucalgary.ca:~projects/project2/
    
  • Upload more than one directory:
    desktop$ rsync -axv my_data1 my_data2 my_data3 username@arc-dtn.ucalgary.ca:/desired/destination
    
  • Download one file (eg. output.dat) from ARC to the current directory on your workstation:
    ## Note the '.' at the end of the command which references the current working directory on your computer
    desktop$ rsync -v username@arc-dtn.ucalgary.ca:projects/project1/output.dat .
    
  • Download one directory (eg. outputs) from ARC to the current directory on your workstation:
    desktop$ rsync -axv username@arc-dtn.ucalgary.ca:projects/project1/outputs .
    

sftp: secure file transfer protocol


sftp is a file transfer program, similar to ftp, which performs all operations over an encrypted ssh transport. It may also use many features of ssh, such as public key authentication and compression.


sftp has an interactive mode, in which sftp understands a set of commands similar to those of ftp. Commands are case insensitive.

rclone: rsync for cloud storage

Rclone is a command line program to sync files and directories to and from a number of on-line storage services.

Graphical File Transfer Tools

FileZilla

FileZilla is a free cross-platform file transfer program that can transfer files via sFTP.

https://filezilla-project.org/download.php?type=client (Note: Installer may bundle ads and unwanted software. Be careful when clicking through.)

MobaXterm (Windows)

MobaXterm is the recommended tool for remote access and data transfer in Windows OSes.

MobaXterm is a one-stop solution for most remote access work on a compute cluster or a Unix / Linux server.

It provides many Unix like utilities for Windows including an SSH client and X11 graphics server. It provides a graphical interface for data transfer operations.

WinSCP (Windows)

WinSCP is a free Windows file transfer tool.

https://winscp.net/eng/index.php

Cloud based Fire Transfer Servcies

Globus File Transfer

Globus File Transfer is a cloud based service for file transfer and file sharing. It uses GridFTP for high speed and reliable data transfers

How to get started

1. Navigate to the web page https://www.globusid.org
2. Create a Globus ID and password using your UCalgary email address.

Use Globus Web Application to transfer files

1. To initiate data transfers using the Globus Web Application, navigate to https://www.globusid.org/login and log into your Globus account
2. On the left panel, click on File Manager to define/select the endpoints. For example, to transfer data from ARC cluster (endpoint 1) to Compute Canada cedar cluster (endpoint 2)

  • Under collection, for ARC data transfer node choose 'endpoint 1' as 'ucalgary#arc-dtn.ucalgary.ca' from the drop down menu. Authenticate your access using UCalgary IT credentials. This will bring you to the home directory on ARC.
  • Next, for Compute Canada cedar data transfer node choose 'endpoint 2' as 'computecanada#cedar-dtn' from the drop down menu. Again authenticate your access using Compute Canada credentials. Navigate to the location where you want to transfer the file
  • Select the file to be transferred from 'endpoint 1' and initiate the transfer process.

Special cases

Transferring Large Datasets

Using screen and rsync

If you want to transfer a large amount of data from a remote Unix system to ARC you can use 'rsync to handle the transfer. However, you will have to keep your SSH session from your workstation alive during the entire transfer. Very often this is not convenient or not feasible.

To overcome this one can run the rsync transfer inside a screen virtual session on ARC. screen creates an SSH session local to ARC and allows for reconnection from SSH sessions from your workstation.

To initialize

# Login to ARC
$ ssh username@arc.ucalgary.ca

# Start a screen session
$ screen

# While in the new screen session, start the transfer with rsync.
$ rsync -axv ext_user@external.system:path/to/remote/data  .

# Disconnect from the screen session with the hotkey 'Ctrl-a d'
# You may now disconnect from ARC or close the lid of you laptop or turn off the computer.

To check if the transfer has been finished.

# Login to ARC
$ ssh username@arc.ucalgary.ca

# Reconnect to the screen session
$ screen -r

# If the transfer has been finished close the screen session.
$ exit

# If the transfer is still running, disconnect from the screen session with the hotkey 'Ctrl-a d'

Very large files

If the files are large and the transfer speed is low the transfer may fail before the file has been transferred. rsync may not be of help here, as it will not restart the file transfer (have not tested recently).

The solution may be to split the large file into smaller chunks, transfer them using rsync and then join them on the remote system (ARC for example):

# Large file is 506MB in this example.
$ ls -l t.bin
-rw-r--r-- 1 drozmano drozmano 530308481 Jun  8 11:06 t.bin

# split the file:
$ split -b 100M t.bin t.bin_chunk.

# Check the chunks.
$ ls -l t.bin_chunk.*
-rw-r--r-- 1 drozmano drozmano 104857600 Jun  8 11:09 t.bin_chunk.aa
-rw-r--r-- 1 drozmano drozmano 104857600 Jun  8 11:09 t.bin_chunk.ab
-rw-r--r-- 1 drozmano drozmano 104857600 Jun  8 11:09 t.bin_chunk.ac
-rw-r--r-- 1 drozmano drozmano 104857600 Jun  8 11:09 t.bin_chunk.ad
-rw-r--r-- 1 drozmano drozmano 104857600 Jun  8 11:09 t.bin_chunk.ae
-rw-r--r-- 1 drozmano drozmano   6020481 Jun  8 11:09 t.bin_chunk.af

# Transfer the files:
$ rsync -axv t.bin_chunks.* username@arc.ucalgary.ca:

Then login to ARC and join the files:

$ cat t.bin_chunk.* > t.bin

$ ls -l 
-rw-r--r-- 1 drozmano drozmano 530308481 Jun  8 11:06 t.bin

Success.