View source for How to check states of ARC partitions and nodes

= Background =

The '''states of the nodes''' in ARC (and clusters in general) changes depending on how '''busy''' the cluster is.
Also, nodes can be taken out for '''maintenance''' and the can '''fail''' and break.
Sometimes it is important to be able to have a glance on which nodes do what in the cluster.
For example, you may want to do that before requesting an interactive job from ARC,
to make sure that there are free available nodes in the partition you are planning to use.

= Node states =

To see the current states of the nodes in ARC's partition you can use the <code>arc.nodes</code> command:
<pre>
$ arc.nodes

          Partitions: 16 (bigmem, cpu2017-bf05, cpu2019, cpu2019-bf05, cpu2021, cpu2021-bf24, cpu2022, 
                          cpu2022-bf24, cpu2023, disa, gpu-a100, gpu-v100, ood-vis, parallel, single, wdf-altis)

      ==========================================================================
                     | Total  Allocated  Down  Drained  Idle  Maint  Mixed
      --------------------------------------------------------------------------
              bigmem |     5          1     0        0     3      0      1 
        cpu2017-bf05 |    41          7     1        0    11      0     22 
             cpu2019 |    40         13     0        0     0      0     27 
        cpu2019-bf05 |    87         12     3        1    70      0      1 
             cpu2021 |    17          3     0        0     0      0     14 
        cpu2021-bf24 |    28          0     0        0    27      0      1 
             cpu2022 |    52          5     0        5     0      0     42 
        cpu2022-bf24 |    16          0     0        2    13      0      1 
             cpu2023 |    20          1     1        0     0      0     18 
                disa |     1          0     0        0     1      0      0 
            gpu-a100 |     5          2     0        0     1      0      2 
            gpu-v100 |     7          0     2        2     1      0      2 
             ood-vis |     1          0     0        0     1      0      0 
            parallel |   576        129    92      108     0      4    243 
              single |    14          0     1        1    10      0      2 
           wdf-altis |    12          0     0        0    12      0      0 
      --------------------------------------------------------------------------
       logical total |   922        173   100      119   150      4    376 
                     |
      physical total |   922        173   100      119   150      4    376 

</pre>
* The '''first column''' on the left shows the '''partition names'''. Each line presents information about this specific partition.

* The '''second''' column, '''Total''', indicates '''the total number of nodes''' in this partition.

* The following '''columns''' show '''node states'''. The table is built dynamically, so it only shows the states which currently present in ARC.
: These are standard SLURM states and their description can be found in SLURM documentation:
: https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES


* Available resources can be in the nodes in the '''Idle''' and, possibly, the '''Mixed''' states.

* The nodes marked as '''Allocated''' are completely busy with work.

* The nodes which are '''Down''', '''Drained''', '''Draining''', '''Maint''' cannot be used at this moment.

* The '''Down''' state means that the nodes are broken and off-line.


At the bottom of the table there are '''two bottom lines,''' the '''logical total''', and the '''physical total'''. 
Sometimes, the same node can be assigned to more than one partition in the cluster. 
This way, '''logically''', it will be counted in each partition as many times as the number of partitions it is in.
The '''logical''' count in this case will over count the actual number of nodes in the cluster.
The '''physical''' count corrects for this and reports the actual number of nodes in the cluster.
Thus, if both the numbers are the same, this also means that each reported node is assigned to only one partition.

= Links =
[[How-Tos]]