How to transfer data

Linux and MacOS

While you can find transfer programs for MacOS and Linux that have graphical point-and-click interface, both the operating system come with pre-installed (most of the time) command line transfer tools: scp, rsync, sftp. These are powerful and convenient tools that can handle any practical data transfer to and from our compute clusters.

scp -- secure copy

• Manual page on-line: http://man7.org/linux/man-pages/man1/scp.1.html

scp copies files between hosts on a network. It uses ssh for data transfer, and uses the same authentication and provides the same security as ssh.

In practical terms scp is a minimal and also sufficient transfer tool to copy files between network connected Unix based computers in a secure manner.

rsync -- Remote SYNCronizer

• Manual page on-line: http://man7.org/linux/man-pages/man1/rsync.1.html

Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use.

In practice, rsync is scp on steroids. It is designed to synchronize two locations, that is to make them the same. So, if a transfer stops for some reason, if one restarts the transfer, rsync will check the destination and only transfers what is needed. This way, you can conveniently restart the transfer at any moment without loosing the progress. With scp this is not an option.

sftp -- secure file transfer protocol

• Manual page on-line: http://man7.org/linux/man-pages/man1/sftp.1.html

sftp is a file transfer program, similar to ftp, which performs all operations over an encrypted ssh transport. It may also use many features of ssh, such as public key authentication and compression.