How to access ResearchFS storage from ARC - Revision history

Dmitri: Dmitri moved page How to access ResearchFS stroage from ARC to How to access ResearchFS storage from ARC without leaving a redirect

2026-08-18T21:39:27Z

Dmitri moved page How to access ResearchFS stroage from ARC to How to access ResearchFS storage from ARC without leaving a redirect

← Older revision		Revision as of 21:39, 18 August 2026
(No difference)

Dmitri: /* Rclone configuration */

2026-08-18T21:37:15Z

Rclone configuration

← Older revision		Revision as of 21:37, 18 August 2026
Line 32:		Line 32:

	'''Kerberos''' is a secure network authentication protocol that allows users to authenticate to services in a consistent and secure way. It is a more appropriate way to authenticate to '''ResearchFS''' and avoids storing your UofC password in the '''rclone''' configuration file.		'''Kerberos''' is a secure network authentication protocol that allows users to authenticate to services in a consistent and secure way. It is a more appropriate way to authenticate to '''ResearchFS''' and avoids storing your UofC password in the '''rclone''' configuration file.


	Run		Run
	$ rclone config		$ rclone config

	and choose the following options ~~during <code>rclone config</code>~~:		and choose the following options:

	* New remote		* New remote
	* Name --> ResearchFS		* Name --> ResearchFS

Dmitri: /* Transferring directories between ResearchFS and ARC */

2026-08-18T19:36:02Z

Transferring directories between ResearchFS and ARC

← Older revision		Revision as of 19:36, 18 August 2026
Line 109:		Line 109:

	To copy the '''contents''' of a local directory named <code>arc_data</code> to a directory named <code>arc_data</code> on ResearchFS:		To copy the '''contents''' of a local directory named <code>arc_data</code> to a directory named <code>arc_data</code> on ResearchFS:
			$ rclone copy -P arc_data ResearchFS:my_share/arc_data

	~~$ rclone copy -P arc_data ResearchFS:my_share/arc_data~~		The <code>-P</code> option displays the progress of the transfer,
			including the amount of data transferred, transfer speed, and estimated completion time.
	The <code>-P</code> option displays the progress of the transfer, including the amount of data transferred, transfer speed, and estimated completion time.

	To copy the contents of the ResearchFS directory back to a local <code>arc_data</code> directory on ARC:		To copy the contents of the ResearchFS directory back to a local <code>arc_data</code> directory on ARC:
			$ rclone copy -P ResearchFS:my_share/arc_data arc_data
	$ rclone copy -P ResearchFS:my_share/arc_data arc_data

	Note that <code>rclone copy</code> copies the '''contents''' of the source directory into the specified destination.		Note that <code>rclone copy</code> copies the '''contents''' of the source directory into the specified destination.
Line 122:		Line 121:

	For example:		For example:
			$ rclone copy -P arc_data ResearchFS:my_share/arc_data
	$ rclone copy -P arc_data ResearchFS:my_share/arc_data

	copies the contents of the local <code>arc_data</code> directory into <code>my_share/arc_data</code> on ResearchFS.		copies the contents of the local <code>arc_data</code> directory into <code>my_share/arc_data</code> on ResearchFS.
Line 133:		Line 131:

	After transferring important data, you can use <code>rclone check</code> to compare the source and destination:		After transferring important data, you can use <code>rclone check</code> to compare the source and destination:
			$ rclone check arc_data ResearchFS:my_share/arc_data
	$ rclone check arc_data ResearchFS:my_share/arc_data

	To check a transfer in the opposite direction:		To check a transfer in the opposite direction:
			$ rclone check ResearchFS:my_share/arc_data arc_data
	$ rclone check ResearchFS:my_share/arc_data arc_data

	The <code>check</code> operation compares the files at the source and destination without modifying either location.		The <code>check</code> operation compares the files at the source and destination without modifying either location.

Dmitri: /* Rclone configuration */

2026-08-18T19:33:29Z

Rclone configuration

← Older revision		Revision as of 19:33, 18 August 2026
Line 30:		Line 30:
	Obscuring a password is '''not encryption''' and should not be considered secure storage for your UofC password.		Obscuring a password is '''not encryption''' and should not be considered secure storage for your UofC password.

	~~Kerberos provides a more appropriate way to authenticate to ResearchFS without storing your UofC password in the rclone configuration file.~~

	~~Choose~~ the following options during <code>rclone config</code>:		'''Kerberos''' is a secure network authentication protocol that allows users to authenticate to services in a consistent and secure way. It is a more appropriate way to authenticate to '''ResearchFS''' and avoids storing your UofC password in the '''rclone''' configuration file.

			Run
			$ rclone config

			and choose the following options during <code>rclone config</code>:

	* New remote		* New remote

Dmitri: /* Background */

2026-08-18T19:25:32Z

Background

← Older revision		Revision as of 19:25, 18 August 2026
Line 1:		Line 1:
	= Background =		= Background =

			[[Storage_Options#ResearchFS \| ResearchFS]]
			is a University of Calgary-hosted storage solution funded and operated by '''RCS'''.
			It is available by request to faculty and staff for storing '''active research data'''.


			'''ResearchFS''' storage is presented to researchers as mountable Windows '''SMB/CIFS''' network shares.
			Direct mounting of '''ResearchFS shares''' on ARC is not supported.
			Data can instead be transferred between the '''ARC cluster''' and '''ResearchFS''' using '''rclone'''.


	[https://rclone.org/ Rclone] is a command-line program for managing files on remote storage systems.		[https://rclone.org/ Rclone] is a command-line program for managing files on remote storage systems.
	On the ~~ARC cluster~~, ~~it can be used to download data from and upload data to~~ the ~~'''UofC ResearchFS''' storage.~~		It can access ResearchFS directly using the SMB protocol, without mounting the network share on ARC.

	~~ResearchFS storage is presented to researchers as Windows '''SMB/CIFS'''~~ network ~~shares.~~
	~~On Linux, SMB shares can be accessed using software such as Samba or, as described here, directly using '''rclone'''~~.

	= Rclone configuration =		= Rclone configuration =

Dmitri at 19:12, 18 August 2026

2026-08-18T19:12:21Z

← Older revision		Revision as of 19:12, 18 August 2026
Line 1:		Line 1:
			= Background =

	[https://rclone.org/ Rclone] is a command-line program for managing files on remote storage systems.		[https://rclone.org/ Rclone] is a command-line program for managing files on remote storage systems.

Dmitri: /* Accessing ResearchFS shares */

2026-08-18T19:11:04Z

Accessing ResearchFS shares

← Older revision		Revision as of 19:11, 18 August 2026
Line 89:		Line 89:

	To list the shares available on ResearchFS:		To list the shares available on ResearchFS:
			$ rclone lsf ResearchFS:
	$ rclone lsf ResearchFS:

	To list files and directories inside your share:		To list files and directories inside your share:
			$ rclone lsf ResearchFS:my_share/
	$ rclone lsf ResearchFS:my_share/

	= Transferring directories between ResearchFS and ARC =		= Transferring directories between ResearchFS and ARC =

Dmitri: /* Links */

2026-08-18T19:10:36Z

Links

← Older revision		Revision as of 19:10, 18 August 2026
Line 135:		Line 135:
	= Links =		= Links =

	[[How-~~Toos~~]]		[[How-Tos]]

Dmitri: Created page with " [https://rclone.org/ Rclone] is a command-line program for managing files on remote storage systems. On the ARC cluster, it can be used to download data from and upload data to the '''UofC ResearchFS''' storage. ResearchFS storage is presented to researchers as Windows '''SMB/CIFS''' network shares. On Linux, SMB shares can be accessed using software such as Samba or, as described here, directly using '''rclone'''. = Rclone configuration = '''rclone''' can access '''..."

2026-08-18T19:10:23Z

Created page with " [https://rclone.org/ Rclone] is a command-line program for managing files on remote storage systems. On the ARC cluster, it can be used to download data from and upload data to the '''UofC ResearchFS''' storage. ResearchFS storage is presented to researchers as Windows '''SMB/CIFS''' network shares. On Linux, SMB shares can be accessed using software such as Samba or, as described here, directly using '''rclone'''. = Rclone configuration = '''rclone''' can access '''..."

New page

[https://rclone.org/ Rclone] is a command-line program for managing files on remote storage systems.
On the ARC cluster, it can be used to download data from and upload data to the '''UofC ResearchFS''' storage.

ResearchFS storage is presented to researchers as Windows '''SMB/CIFS''' network shares.
On Linux, SMB shares can be accessed using software such as Samba or, as described here, directly using '''rclone'''.

= Rclone configuration =

'''rclone''' can access '''SMB/CIFS''' storage using its native
[https://rclone.org/smb/ SMB/CIFS connector]
without mounting the share on the cluster.

'''WARNING''': When configuring a new remote using the interactive

$ rclone config

command, do '''not''' enter your UofC password into the rclone configuration.
Instead, leave the password blank and enable '''Kerberos''' authentication.

If a password is provided, rclone stores an obscured version of it in its configuration file.
Obscuring a password is '''not encryption''' and should not be considered secure storage for your UofC password.

Kerberos provides a more appropriate way to authenticate to ResearchFS without storing your UofC password in the rclone configuration file.

Choose the following options during <code>rclone config</code>:

* New remote
* Name --> ResearchFS
* Storage --> smb
* Host --> researchfs.uc.ucalgary.ca
* Port --> default (445)
* User --> <your_uofc_IT_username>
* Samba password --> leave blank
* Domain --> UC
* SPN --> leave blank
* use_kerberos --> true
* Edit advanced config --> No
* Keep this remote? --> Yes
* Quit

'''Alternatively''', if you already have an rclone configuration file, you can edit it directly.

Add the following section to <code>~/.config/rclone/rclone.conf</code>:

<syntaxhighlight lang="ini">
[ResearchFS]
type = smb
host = researchfs.uc.ucalgary.ca
user = <uofc_username>
domain = UC
use_kerberos = true
</syntaxhighlight>

= Obtaining a Kerberos ticket =

Before accessing ResearchFS, you need a valid Kerberos ticket.

Obtain one with:

<pre>
$ kinit
Password for username@UC.UCALGARY.CA:
</pre>

Enter your UofC password when prompted.
The password is used to obtain a Kerberos ticket and is not stored in the rclone configuration.

You can check your Kerberos tickets with:

<pre>
$ klist

Ticket cache: FILE:/tmp/krb5cc_11111111
Default principal: username@UC.UCALGARY.CA

Valid starting Expires Service principal
08/14/26 15:06:05 08/15/26 01:06:05 krbtgt/UC.UCALGARY.CA@UC.UCALGARY.CA
renew until 08/21/26 15:05:58
</pre>

The Kerberos credential cache (for example, <code>/tmp/krb5cc_11111111</code>) contains the ticket that rclone uses when authenticating to ResearchFS.

Kerberos tickets have a limited lifetime.
The expiration time is shown by <code>klist</code>.
If your ticket has expired, run <code>kinit</code> again to obtain a new one.

= Accessing ResearchFS shares =

To list the shares available on ResearchFS:

$ rclone lsf ResearchFS:

To list files and directories inside your share:

$ rclone lsf ResearchFS:my_share/

= Transferring directories between ResearchFS and ARC =

To copy the '''contents''' of a local directory named <code>arc_data</code> to a directory named <code>arc_data</code> on ResearchFS:

$ rclone copy -P arc_data ResearchFS:my_share/arc_data

The <code>-P</code> option displays the progress of the transfer, including the amount of data transferred, transfer speed, and estimated completion time.

To copy the contents of the ResearchFS directory back to a local <code>arc_data</code> directory on ARC:

$ rclone copy -P ResearchFS:my_share/arc_data arc_data

Note that <code>rclone copy</code> copies the '''contents''' of the source directory into the specified destination.
It does not automatically append the source directory name to the destination path.

For example:

$ rclone copy -P arc_data ResearchFS:my_share/arc_data

copies the contents of the local <code>arc_data</code> directory into <code>my_share/arc_data</code> on ResearchFS.

If a transfer is interrupted, the same <code>rclone copy</code> command can be run again.
Files that have already been successfully transferred and are unchanged do not need to be copied again.

=== Verifying transferred data ===

After transferring important data, you can use <code>rclone check</code> to compare the source and destination:

$ rclone check arc_data ResearchFS:my_share/arc_data

To check a transfer in the opposite direction:

$ rclone check ResearchFS:my_share/arc_data arc_data

The <code>check</code> operation compares the files at the source and destination without modifying either location.
It reports missing or different files, making it useful for verifying that a transfer completed successfully.

= Links =

[[How-Toos]]