How to check GPU utilization - Revision history

Pemartin: Added common tips for working with GPUs

2026-03-13T19:15:11Z

Added common tips for working with GPUs

← Older revision		Revision as of 19:15, 13 March 2026
Line 55:		Line 55:

	* Find the job you want to check and there may be useful graphs for GPU usage.		* Find the job you want to check and there may be useful graphs for GPU usage.

			= Common tips for working with GPUs on ARC =

			* Confirm your software is GPU-enabled and actually using CUDA, Many tools automatically fall back to CPU without warning.
			* Avoid long salloc sessions holding idle GPUs. For interactive exploration, request small walltime intervals on older generation GPUs (V100).
			* Verify one GPU is well-utilized before scaling to multiple.
			* Increase batch size (within memory limits) to amortize overhead.
			* Use multi-threaded data loaders.
			* Avoid small, frequent I/O; prefer fewer, larger reads/writes.
			* Write active job output to high-performance scratch. Avoid home paths during training.

	[[Category:Guides]]		[[Category:Guides]]
	[[Category:How-Tos]]		[[Category:How-Tos]]
	{{Navbox Guides}}		{{Navbox Guides}}

2026-03-13T19:02:00Z

Added introduction section with the importance of GPU monitoring

← Older revision		Revision as of 19:02, 13 March 2026
Line 1:		Line 1:
	= ~~General~~ =		= Introduction =

			Compute accelerators (GPU cards) are a powerful tool for scientific research in areas such as machine learning training, scientific simulations, and large-scale data processing. These resources are expensive and limited in our compute clusters. If these shared resources are not properly utilized, significant compute capacity and time can be wasted.

			Typical GPU workflows involves three main stages in each iteration: transferring data from the CPU to the GPU, executing GPU kernels, and transferring results back from the GPU to the CPU. GPU utilization drops when the GPU spends time waiting for data or when the kernels do not fully exploit the available parallelism. By monitoring utilization metrics, users can identify when the GPU is idle or underused and take steps to address performance issues.

			Monitoring GPU utilization promotes responsible use of shared computing infrastructure. Requesting more than 1 GPU if not needed increases queue times and reduces availability for other users. By understanding and monitoring the usage patterns, users can make optimal resource requests and improve overall cluster efficiency.

	= For Running Jobs =		= For Running Jobs =

2023-09-21T20:28:52Z

Replaced links section with navbox

← Older revision		Revision as of 20:28, 21 September 2023
Line 49:		Line 49:

	* Find the job you want to check and there may be useful graphs for GPU usage.		* Find the job you want to check and there may be useful graphs for GPU usage.

	~~= Links =~~

	~~[[How-Tos]]~~

	[[Category:Guides]]		[[Category:Guides]]
	[[Category:How-Tos]]		[[Category:How-Tos]]
			{{Navbox Guides}}

2023-09-20T21:57:05Z

Added categories

2023-02-08T17:13:27Z

Using Sampled Metrics

← Older revision		Revision as of 17:13, 8 February 2023
Line 42:		Line 42:

	On the [[Open OnDemand \| OOD portal]], here https://ood-arc.rcs.ucalgary.ca		On the [[Open OnDemand \| OOD portal]], here https://ood-arc.rcs.ucalgary.ca
	login using your UofC credentials.		login using your UofC credentials (it may require your full UofC email address instead of just user name).

	Once you log in,		Once you log in,

2022-10-14T21:15:00Z

Using Sampled Metrics

← Older revision		Revision as of 21:15, 14 October 2022
Line 48:		Line 48:
	: this will open an interface to your past jobs that are available in the database.		: this will open an interface to your past jobs that are available in the database.

	* Find the job you ~~like~~ and there may be useful graphs for GPU usage.		* Find the job you want to check and there may be useful graphs for GPU usage.

	= Links =		= Links =

	[[How-Tos]]		[[How-Tos]]

2022-10-14T21:14:46Z

Using Sampled Metrics

← Older revision		Revision as of 21:14, 14 October 2022
Line 46:		Line 46:
	Once you log in,		Once you log in,
	* Select '''Help --> View my job metrics''',		* Select '''Help --> View my job metrics''',
	: this will open an interface to your past jobs ~~we keep metrics about~~.		: this will open an interface to your past jobs that are available in the database.

	* Find the job you like and there may be useful graphs for GPU usage.		* Find the job you like and there may be useful graphs for GPU usage.

2022-10-14T20:20:51Z

Using Sampled Metrics

← Older revision		Revision as of 20:20, 14 October 2022
Line 41:		Line 41:
	== Using Sampled Metrics ==		== Using Sampled Metrics ==

	On the [[Open OnDemand \| OOD portal]]		On the [[Open OnDemand \| OOD portal]], here https://ood-arc.rcs.ucalgary.ca
			login using your UofC credentials.

			Once you log in,
			* Select '''Help --> View my job metrics''',
			: this will open an interface to your past jobs we keep metrics about.

	~~once you log in, you can select "Help --> View my job metrics", this will open an interface to your past jobs we keep metrics about.~~ Find the job you like and there may be useful graphs for GPU usage.		* Find the job you like and there may be useful graphs for GPU usage.

	= Links =		= Links =

	[[How-Tos]]		[[How-Tos]]

2022-10-14T20:18:51Z

General

← Older revision		Revision as of 20:18, 14 October 2022
Line 1:		Line 1:
	= General =		= General =

			= For Running Jobs =

			== using SLURM ==

	If you have a job that is running on a GPU node and that is expected to use a GPU on that node,		If you have a job that is running on a GPU node and that is expected to use a GPU on that node,
Line 32:		Line 36:
	In this case there was 1 GPU allocated and its usage was '''30%'''.		In this case there was 1 GPU allocated and its usage was '''30%'''.
	The code <code>openDBA</code> also uses '''338 MB''' of the GPU memory.		The code <code>openDBA</code> also uses '''338 MB''' of the GPU memory.

			= For Past Jobs =

			== Using Sampled Metrics ==

			On the [[Open OnDemand \| OOD portal]]


			once you log in, you can select "Help --> View my job metrics", this will open an interface to your past jobs we keep metrics about. Find the job you like and there may be useful graphs for GPU usage.

	= Links =		= Links =

	[[How-Tos]]		[[How-Tos]]

2022-08-22T15:34:02Z

← Older revision	Revision as of 15:34, 22 August 2022
(No difference)