How to find current limits on ARC

ARC has many partitions each with different resource limits. It is important to understand these limits when planning a new computation and before submitting a job. For resources that are more scarce such as GPU and bigmem nodes, understanding the resource limits and making your jobs as efficient as possible will help increase the throughput of jobs on the cluster.

Resource allocation limits

The current resource limits can be shown by the arc.limits command:

$ arc.limits

   PartitionName        Flags       MaxTRES     MaxWall        MaxTRESPU MaxSubmitPU MinTRES
0         normal                             7-00:00:00                         4000        
1        cpu2019                    cpu=240  7-00:00:00          cpu=240        4000        
2       gpu-v100  DenyOnLimit  cpu=80,gpu=4  1-00:00:00    cpu=160,gpu=8        4000   gpu=1
3         single                    cpu=200  7-00:00:00  cpu=200,node=30        4000        
4           razi                             7-00:00:00                         4000        
5        apophis                             7-00:00:00                         4000        
6        razi-bf                    cpu=546    05:00:00          cpu=546        4000        
7     apophis-bf                    cpu=280    05:00:00          cpu=280        4000        
8        lattice                    cpu=408  7-00:00:00          cpu=408        4000        
9       parallel                    cpu=624  7-00:00:00          cpu=624        4000        
10        bigmem                     cpu=80  1-00:00:00     cpu=80,gpu=1        4000        
11       cpu2013                             7-00:00:00                         4000        
12        pawson                             7-00:00:00                         4000        
13     pawson-bf                    cpu=480    05:00:00          cpu=480        4000        
14         theia                             7-00:00:00                         4000        
15      theia-bf                    cpu=280    05:00:00                         4000        
16          demo                             7-00:00:00                         4000        
17       synergy                             7-00:00:00                         4000        
18    synergy-bf                    cpu=448    05:00:00          cpu=448        4000        
19    backfill05                   cpu=1000    05:00:00         cpu=1000        4000        
20       cpu2021                    cpu=576  7-00:00:00          cpu=576        4000        
21    backfill24                    cpu=208  1-00:00:00          cpu=208        4000        
22      sherlock                             7-00:00:00                         4000        
23      wdf-zach                             7-00:00:00                         4000        
24     wdf-think                             7-00:00:00                         4000        
25          mtst                             7-00:00:00                                     
26       cpu2022                    cpu=520  7-00:00:00          cpu=520        4000        
27      gpu-a100  DenyOnLimit  cpu=80,gpu=4  1-00:00:00    cpu=160,gpu=8        4000   gpu=1

TRES=Trackable RESources
  PU=Per User

The table shows the list of partitions and set limits for each of the partitions.

• Flags column shows settings that determine if the job may be accepted or denied if the resource request is over the limit.

• MaxTRES -- the maximum traceable resources, maximal amount of resources allowed per job on this partition.

• MaxWall -- the maximal wall time for a job, the longest time a job is allowed to run on this partition.

• MaxTRESPU -- maximum traceable resources per user, the total maximal amount of resources allowed per single user on the partition. If the limit is reached the resource requests above the limit will have to wait in the queue until some resources are freed by currently running jobs.

• MaxSubmitPU -- maximal number of jobs submitted to this partition. SLURM will reject any jobs above this limit. Please note, that there is also a global limit of 4000 jobs per user for the entire cluster.

• MinTRES -- minimal amount of resources in job's resource request that is required for a job to be accepted. Relevant on GPU partition, as jobs are expected to request at least one GPU to qualify to run on a GPU partition.

Backfill partitions

Some of the partitions on ARC are not for general use. They may be dedicated to a specific task or may belong to a specific group. However, some of the nodes from such partitions may also be assigned to additional auxiliary partitions, which names end with the -bfXX suffix, such as cpu2019-bf05, cpu2021-bf24, etc., as they are shown in the output of the arc.hardware command.

Currently, the limits for such partitions are set via two auxiliary partitions, backfill05 and backfill24 instead.

When examining the output of the arc.limits command, and you want to check the limits for any xxxxxxx-bf05 partition, for example, you have to check the limits of the backfill05 instead, to find the current limits for the partition.

Correspondingly, for any xxxxxxx-bf24 partition, please use the backfill24 limits.

Examples

gpu-v100 partition limits

For example, the gpu-v100 partition is limited to 4000 jobs per user and the maximum run time is limited to 24 hours. The job must request at least 1 GPU, but not more than 4 GPUs. The total number of GPUs working for a single user's jobs is limited to 8. If the resource request of a submitted job is over the limit, the job will be rejected.