How to use array jobs in SLURM
Warning
Job arrays is an advanced technique. Some caution should be used when submitting array of jobs.
Please never submit a full set of jobs when starting a new set of calculation.
- Please, first time submit only one job of the set as a test case and make sure that it runs and completes successfully.
- Then submit an array of 2-3 jobs, to make sure that you fully understand how array will be working.
- Only then, commit to submitting the array job for the full data set.
If you submit an array of thousands of jobs and all of them will fail, this will cause a significant strain on the cluster.
The scheduler will schedule all these jobs and will be preparing nodes for them and then cleaning up the nodes after the failed jobs.
This will be a lot of work done for nothing.
Please be considerate.
Array jobs
Job arrays in SLURM allow you to run many jobs using one and same job script.
The job script you submit with the sbatch command does not accept any parameters.
Like this:
$ sbatch array_job.slurm
If you have many data files you want to process in multiple jobs, this approach may be a convenient alternative to generating many separate job scripts for each data file and submitting them one by one.
An array job script example, array_job.slurm:
#!/bin/bash
# ============================================
#SBATCH --job-name=my_array_job
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=4gb
#SBATCH --time=0-01:00:00
#SBATCH --array=1-10
# ============================================
./myapplication $SLURM_ARRAY_TASK_ID
This job script will create on auxiliary job in the queue which will generate 10 element jobs of the array from 1 to 10.
The job scripts for each element job of the array will be exactly the same.
The difference comes from the value of the environmental variable $SLURM_ARRAY_TASK_ID
which is set to the number of this element job in the array (index).
That is the first job of the array will execute command line:
./myapplication 1
and the 8th element job will run
./myapplication 8
and so on. The auxiliary job will stay in the queue until all the elemental jobs are done. The resources requested in the job script are allocated to each elemental job separately, that is, each job gets its own 4 GB of RAM and 1 CPU, as requested in the example.
With this approach, it is left to the myapplication to interpret what the input parameter,
in this case 1 or 8, mean, in terms of data processing.
Mapping task index to a data file
Simple case of numbered data files
If the data files are data-1.dat, ....., data-11.dat, you can use this mapping:
#!/bin/bash
....
# ============================================
DATA="data-${SLURM_ARRAY_TASK_ID}.dat"
./myapplication $DATA
or, if they are data-001.dat, ....., data-111.dat, you can use the printf command for desired formatting:
#!/bin/bash
....
# ============================================
# This command produces number of the same width, padded to 3 characters. For example, 001.
printf -v NUM %.3d $SLURM_ARRAY_TASK_ID
DATA="data-${NUM}.dat"
./myapplication $DATA
Mapping job index to a line in a file
The command to extract a specific line from a text file, in this case the line number 10:
sed -n 10p filename.txt
With this, we can put data file names into into a text file index.idx, one data file name per line.
Then in the job script the data file names can be extracted using the command above
from the line specified by the job task index:
#!/bin/bash
....
# ============================================
MAP_FILE="index.idx"
DATA=`sed -n ${SLURM_ARRAY_TASK_ID}p $MAP_FILE`
./myapplication $DATA
Mapping job index to multiple data files in a file
If you have to process several computations from an element job of the job array you are going to run, you job script have to run several over several data files form your data mapping file.
For example, you want to process 1000 jobs using an array of 20 jobs, from 0 to 19, processing 50 data files in
each element job.
The data mapping file, index.idx, will look like that:
data-0001.dat data-0002.dat data-0003.dat .... data-0998.dat data-0999.dat data-1000.dat
Then, the array job script, array.slurm, can be:
#! /bin/bash
# ====================================================
....
#SBATCH --array=0-19
# ====================================================
SIZE=50
INDEX_FILE="index.idx"
IDXBEG=$(( SLURM_ARRAY_TASK_ID * SIZE ))
IDXEND=$(( IDXBEG + SIZE - 1 ))
for IDX in `seq $IDXBEG $IDXEND`; do
DATA=`sed -n ${IDX}p $INDEX_FILE`
# add your work here....
./myapplication $DATA
done
# ====================================================
More Arrays
Array options
You can indicate a stride for the array elements to be used in element jobs:
#SBATCH --array=1-100:4
This option in your job script sets the stride to 4 and will run 25 jobs with SLURM_ARRAY_TASK_ID
having the values of {1, 5, 9, .... 97}.
Another option is to indicate the number of element jobs that can run at the same time. By default all the element jobs can be executed at the same time, provided that resources are available. With this option you can limit the number of jobs to, for example, one:
#SBATCH --array=1-100%1
you can combine it with the stride as well, to limit your 25 jobs to run 5 at a time:
#SBATCH --array=1-100:4%5
Limiting a job array to run 1 job at a time may be a practical way to resubmit the same job a fixed number of times. If you want to run some simulation for 1 000 000 steps, it can be done as an array job of, say 100 element, each element running 10 000 steps, one after another.
Array Environmental variables
For a first element jobs in an --array=1-100
the following variables will be set to
SLURM_JOB_ID=87654321 SLURM_ARRAY_JOB_ID=87654321 SLURM_ARRAY_TASK_ID=1 SLURM_ARRAY_TASK_COUNT=100 SLURM_ARRAY_TASK_MAX=100 SLURM_ARRAY_TASK_MIN=1