How to access ResearchFS storage from ARC

From RCSWiki
Jump to navigation Jump to search

Background

ResearchFS is a University of Calgary-hosted storage solution funded and operated by RCS. It is available by request to faculty and staff for storing active research data.


ResearchFS storage is presented to researchers as mountable Windows SMB/CIFS network shares. Direct mounting of ResearchFS shares on ARC is not supported. Data can instead be transferred between the ARC cluster and ResearchFS using rclone.


Rclone is a command-line program for managing files on remote storage systems. It can access ResearchFS directly using the SMB protocol, without mounting the network share on ARC.

Rclone configuration

rclone can access SMB/CIFS storage using its native SMB/CIFS connector without mounting the share on the cluster.

WARNING: When configuring a new remote using the interactive

$ rclone config

command, do not enter your UofC password into the rclone configuration. Instead, leave the password blank and enable Kerberos authentication.

If a password is provided, rclone stores an obscured version of it in its configuration file. Obscuring a password is not encryption and should not be considered secure storage for your UofC password.


Kerberos is a secure network authentication protocol that allows users to authenticate to services in a consistent and secure way. It is a more appropriate way to authenticate to ResearchFS and avoids storing your UofC password in the rclone configuration file.


Run 

$ rclone config

and choose the following options:

  • New remote
  • Name --> ResearchFS
  • Storage --> smb
  • Host --> researchfs.uc.ucalgary.ca
  • Port --> default (445)
  • User --> <your_uofc_IT_username>
  • Samba password --> leave blank
  • Domain --> UC
  • SPN --> leave blank
  • use_kerberos --> true
  • Edit advanced config --> No
  • Keep this remote? --> Yes
  • Quit

Alternatively, if you already have an rclone configuration file, you can edit it directly.

Add the following section to ~/.config/rclone/rclone.conf: 

[ResearchFS]
type = smb
host = researchfs.uc.ucalgary.ca
user = <uofc_username>
domain = UC
use_kerberos = true

Obtaining a Kerberos ticket

Before accessing ResearchFS, you need a valid Kerberos ticket.

Obtain one with: 

$ kinit
Password for username@UC.UCALGARY.CA:

Enter your UofC password when prompted. The password is used to obtain a Kerberos ticket and is not stored in the rclone configuration.

You can check your Kerberos tickets with: 

$ klist

Ticket cache: FILE:/tmp/krb5cc_11111111
Default principal: username@UC.UCALGARY.CA

Valid starting     Expires            Service principal
08/14/26 15:06:05  08/15/26 01:06:05  krbtgt/UC.UCALGARY.CA@UC.UCALGARY.CA
        renew until 08/21/26 15:05:58

The Kerberos credential cache (for example, /tmp/krb5cc_11111111) contains the ticket that rclone uses when authenticating to ResearchFS.

Kerberos tickets have a limited lifetime. The expiration time is shown by klist. If your ticket has expired, run kinit again to obtain a new one.

Accessing ResearchFS shares

To list the shares available on ResearchFS: 

$ rclone lsf ResearchFS:

To list files and directories inside your share: 

$ rclone lsf ResearchFS:my_share/

Transferring directories between ResearchFS and ARC

To copy the contents of a local directory named arc_data to a directory named arc_data on ResearchFS: 

$ rclone copy -P arc_data ResearchFS:my_share/arc_data

The -P option displays the progress of the transfer, including the amount of data transferred, transfer speed, and estimated completion time.

To copy the contents of the ResearchFS directory back to a local arc_data directory on ARC: 

$ rclone copy -P ResearchFS:my_share/arc_data arc_data

Note that rclone copy copies the contents of the source directory into the specified destination. It does not automatically append the source directory name to the destination path.

For example: 

$ rclone copy -P arc_data ResearchFS:my_share/arc_data

copies the contents of the local arc_data directory into my_share/arc_data on ResearchFS.

If a transfer is interrupted, the same rclone copy command can be run again. Files that have already been successfully transferred and are unchanged do not need to be copied again.

Verifying transferred data

After transferring important data, you can use rclone check to compare the source and destination: 

$ rclone check arc_data ResearchFS:my_share/arc_data

To check a transfer in the opposite direction: 

$ rclone check ResearchFS:my_share/arc_data arc_data

The check operation compares the files at the source and destination without modifying either location. It reports missing or different files, making it useful for verifying that a transfer completed successfully.

Links

How-Tos

Retrieved from "https://rcs.ucalgary.ca/index.php?title=How_to_access_ResearchFS_storage_from_ARC&oldid=4138"