How to use array jobs in SLURM

Warning

Job arrays is an advanced technique. Some caution should be used when submitting array of jobs.

Please never submit a full set of jobs when starting a new set of calculation.

• Please, first time submit only one job of the set as a test case and make sure that it runs and completes successfully.
• Then submit an array of 2-3 jobs, to make sure that you fully understand how array will be working.
• Only then, commit to submitting the array job for the full data set.

If you submit an array of thousands of jobs and all of them will fail, this will cause a significant strain on the cluster. The scheduler will schedule all these jobs and will be preparing nodes for them and then cleaning up the nodes after the failed jobs. This will be a lot of work done for nothing. Please be considerate.

Array jobs

Job arrays in SLURM allow you to run many jobs using one and same job script. The job script you submit with the sbatch command does not accept any parameters. Like this:

$ sbatch array_job.slurm

If you have many data files you want to process in multiple jobs, this approach may be a convenient alternative to generating many separate job scripts for each data file and submitting them one by one.

An array job script example, array_job.slurm:

#!/bin/bash
# ============================================
#SBATCH --job-name=my_array_job
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=4gb
#SBATCH --time=0-01:00:00

#SBATCH --array=1-10
# ============================================

./myapplication $SLURM_ARRAY_TASK_ID

This job script will create on auxiliary job in the queue which will generate 10 element jobs of the array from 1 to 10. The job scripts for each element job of the array will be exactly the same. The difference comes from the value of the environmental variable $SLURM_ARRAY_TASK_ID which is set to the number of this element job in the array (index).

That is the first job of the array will execute command line:

 ./myapplication 1

and the 8th element job will run

 ./myapplication 8

and so on. The auxiliary job will stay in the queue until all the elemental jobs are done. The resources requested in the job script are allocated to each elemental job separately, that is, each job gets its own 4 GB of RAM and 1 CPU, as requested in the example.

With this approach, it is left to the myapplication to interpret what the input parameter, in this case 1 or 8, mean, in terms of data processing.

Mapping task index to a data file

Simple case of numbered data files

If the data files are data-1.dat, ....., data-11.dat, you can use this mapping:

#!/bin/bash
....
# ============================================
DATA="data-${SLURM_ARRAY_TASK_ID}.dat"

./myapplication $DATA

or, if they are data-001.dat, ....., data-111.dat, you can use the printf command for desired formatting:

#!/bin/bash
....
# ============================================
# This command produces number of the same width, padded to 3 characters.  For example, 001.
printf -v NUM %.3d $SLURM_ARRAY_TASK_ID
DATA="data-${NUM}.dat"

./myapplication $DATA

Mapping job index to a line in a file

The command to extract a specific line from a text file, in this case the line number 10:

sed -n 10p filename.txt

With this, we can put data file names into into a text file index.idx, one data file name per line. Then in the job script the data file names can be extracted using the command above from the line specified by the job task index:

#!/bin/bash
....
# ============================================
MAP_FILE="index.idx"
DATA=`sed -n ${SLURM_ARRAY_TASK_ID}p $MAP_FILE`

./myapplication $DATA

Mapping job index to multiple data files in a file

If you have to process several computations from an element job of the job array you are going to run, you job script have to run several over several data files form your data mapping file.

For example, you want to process 1000 jobs using an array of 20 jobs, from 0 to 19, processing 50 data files in each element job. The data mapping file, index.idx, will look like that:

data-0001.dat
data-0002.dat
data-0003.dat
....
data-0998.dat
data-0999.dat
data-1000.dat

Then, the array job script, array.slurm, can be:

#! /bin/bash
# ====================================================
....
#SBATCH --array=0-19

# ====================================================
SIZE=50
INDEX_FILE="index.idx"

IDXBEG=$(( SLURM_ARRAY_TASK_ID * SIZE ))
IDXEND=$(( IDXBEG + SIZE - 1 ))

for IDX in `seq $IDXBEG $IDXEND`; do
        DATA=`sed -n ${IDX}p $INDEX_FILE`

        # add your work here....
        ./myapplication $DATA
done
# ====================================================

More Arrays

Array options

You can indicate a stride for the array elements to be used in element jobs:

#SBATCH --array=1-100:4

This option in your job script sets the stride to 4 and will run 25 jobs with SLURM_ARRAY_TASK_ID having the values of {1, 5, 9, .... 97}.

Another option is to indicate the number of element jobs that can run at the same time. By default all the element jobs can be executed at the same time, provided that resources are available. With this option you can limit the number of jobs to, for example, one:

#SBATCH --array=1-100%1

you can combine it with the stride as well, to limit your 25 jobs to run 5 at a time:

#SBATCH --array=1-100:4%5

Limiting a job array to run 1 job at a time may be a practical way to resubmit the same job a fixed number of times. If you want to run some simulation for 1 000 000 steps, it can be done as an array job of, say 100 element, each element running 10 000 steps, one after another.

Array Environmental variables

For a first element jobs in an --array=1-100 the following variables will be set to

SLURM_JOB_ID=87654321
SLURM_ARRAY_JOB_ID=87654321
SLURM_ARRAY_TASK_ID=1
SLURM_ARRAY_TASK_COUNT=100
SLURM_ARRAY_TASK_MAX=100
SLURM_ARRAY_TASK_MIN=1