How to find nodenames allocated by SLURM for a job
Background
When a distributed job that is planned to run on multiple nodes requests several compute nodes from SLURM the program that is going to perform the computations
needs to know the names of the compute nodes which are allocated to the job by SLURM to use those nodes.
If the program is based on the distributed MPI library then the distribution of the computational processed to the nodes is done by the MPI program launcher,
mpirun or mpiexec. Thus, these launchers need to know the list of the allocated nodes.
If the computational code (program) was build on ARC and compiled against the OpenMPI library provided on ARC ( openmpi/4.1.1-gnu at the moment of writing),
then the launcher is aware about ARC's SLURM scheduler and can obtain the list of nodes directly from SLURM automatically.
This, however, it not always the case, some software comes in a form of pre-compiled binary files and may be build using a different implementation of the MPI library, for example using Intel MPI, or IBM MPI. Alternatively, the software may not be based on the MPI library and use some other kind of distribution mechanism. In such a case the software needs to the the list of allocated nodes explicitly.
SLURM node list
The list of allocated nodes is provided to the job environment in the SLURM_NODELIST variable.
Inside an interactive job you can check it manually:
$ salloc -N4 -n4 -c1 -t 1:00:00 --mem=1gb salloc: Granted job allocation 21486039 salloc: Waiting for resource configuration salloc: Nodes fc[107-110] are ready for job $ echo $SLURM_NODELIST fc[107-110]
This form of the node list may be difficult to handle, when passing the computational code. Alternatively, you can get the nodes in the job script with the command:
$ scontrol show hostnames fc107 fc108 fc109 fc110
This command generates an iterate-able list and can be used with a for loop:
$ for n in `scontrol show hostnames`; do echo "Allocated node: $n"; done Allocated node: fc107 Allocated node: fc108 Allocated node: fc109 Allocated node: fc110
If the output of the command is saved into a variable, the node names will be concatenated with the " " (space) character:
$ hh=`scontrol show hostnames` [drozmano@fc107 arc-data]$ echo $hh fc107 fc108 fc109 fc110
These are some of the ways to get the list of the compute nodes that SLURM has allocated for a job.