How to use ARC scratch space - Revision history

Lleung: Replaced navbox with guides navbox

2023-09-21T20:35:34Z

Replaced navbox with guides navbox

2023-09-20T22:32:16Z

Added ARC category and navbox

2023-09-20T21:56:32Z

Added categories

2023-04-19T19:48:39Z

Background

← Older revision		Revision as of 19:48, 19 April 2023
Line 15:		Line 15:
	at the moment of writing.		at the moment of writing.
	The '''current storage usage''' on ARC can be checked at any moment with the <code>arc.quota</code> command.		The '''current storage usage''' on ARC can be checked at any moment with the <code>arc.quota</code> command.
			The <code>/scratch</code> is '''not faster''' than other storage on ARC, about the same performance as the <code>/home</code> file system,
			it only provides access to large temporary run time storage.

2023-04-19T19:13:42Z

Large data set

← Older revision		Revision as of 19:13, 19 April 2023
Line 43:		Line 43:
	* The scratch space is cleaned up by deleting the data, and the output from the scratch directory.		* The scratch space is cleaned up by deleting the data, and the output from the scratch directory.
	* Job ends and the scratch directory is automatically deleted.		* Job ends and the scratch directory is automatically deleted.


	This could be done with the following job script, <code>job.slurm</code>:		This could be done with the following job script, <code>job.slurm</code>:

2023-04-19T19:12:53Z

Large data set

← Older revision		Revision as of 19:12, 19 April 2023
Line 33:		Line 33:
	Let us assume that one have a large data set that is compressed into an archive <code>data.zip</code> in one's home directory.		Let us assume that one have a large data set that is compressed into an archive <code>data.zip</code> in one's home directory.
	The data set has to be processed by a Python 3 code <code>analysis.py</code> that is located in the <code>bin</code> directory in the home.		The data set has to be processed by a Python 3 code <code>analysis.py</code> that is located in the <code>bin</code> directory in the home.
	The data set is too large to ~~un-compress~~ into the home directory and the scratch space has to be used to process the data.		The data set is too large to be uncompressed into the home directory and the scratch space has to be used to process the data.
	The result of the analysis is saved by the Python code to the <code>results.log</code> in the working directory.		The result of the analysis is saved by the Python code to the <code>results.log</code> in the working directory.

2023-04-19T17:58:29Z

Background

← Older revision		Revision as of 17:58, 19 April 2023
Line 21:		Line 21:


	Before ~~the~~ job starts, its '''job ID''' is not yet known, and the '''scratch directory''' does not yet exist.		Before a job starts, its '''job ID''' is not yet known, and the '''scratch directory''' does not yet exist.
	Thus, is it impossible to stage any data into that directory before the job starts.		Thus, is it impossible to stage any data into that directory before the job starts.
	Any data staging, if required has to be during job's run time in the job script.		Any data staging, if required has to be during job's run time in the job script.

2023-04-19T17:51:16Z

Large data set

@@ Line 44: / Line 44: @@
 * Job ends and the scratch directory is automatically deleted.
+This could be done with the following job script, <code>job.slurm</code>:
 <syntaxhighlight lang=bash>
 #! /bin/bash
 # ======================================================================
 </syntaxhighlight>
 = Links =
 [[How-Tos]]

2023-04-19T17:41:43Z

Large data set

2023-04-19T17:41:30Z

Large data set

← Older revision		Revision as of 17:41, 19 April 2023
Line 43:		Line 43:
	* The scratch space is cleaned up by deleting the data, and the output from the scratch directory.		* The scratch space is cleaned up by deleting the data, and the output from the scratch directory.
	* Job ends and the scratch directory is automatically deleted.		* Job ends and the scratch directory is automatically deleted.


			<syntaxhighligh lang=bash>
			#! /bin/bash
			# ======================================================================

			</syntaxhighlight>

	= Links =		= Links =
	[[How-Tos]]		[[How-Tos]]