How to use array jobs in SLURM

Array jobs

Job arrays in SLURM allow you to run many jobs using one and same job script. The job script you submit with the sbatch command does not accept any parameters. Like this:

$ sbatch array_job.slurm

If you have many data files you want to process in multiple jobs, this approach may be a convenient alternative to generating many separate job scripts for each data file and submitting them one by one.

An array job script example, array_job.slurm:

#!/bin/bash
# ============================================
#SBATCH --job-name=my_array_job
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=4gb
#SBATCH --time=0-01:00:00

#SBATCH --array=1-10
# ============================================

./myapplication $SLURM_ARRAY_TASK_ID

This job script will create on auxiliary job in the queue which will generate 10 element jobs of the array from 1 to 10. The job scripts for each element job of the array will be exactly the same. The difference comes from the value of the environmental variable $SLURM_ARRAY_TASK_ID which is set to the number of this element job in the array (index).

That is the first job of the array will execute command line:

 ./myapplication 1

and the 8th element job will run

 ./myapplication 8

and so on. The auxiliary job will stay in the queue until all the elemental jobs are done. The resources requested in the job script are allocated to each elemental job separately, that is, each job gets its own 4 GB of RAM and 1 CPU, as requested in the example.

Mapping task index to a data file

Simple case of numbered data files

If the data files are data-1.dat, ....., data-11.dat, you can use this mapping:

#!/bin/bash
....
# ============================================
DATA="data-${SLURM_ARRAY_TASK_ID}.dat"

./myapplication $DATA

or, if they are data-001.dat, ....., data-111.dat, you can use the printf command for desired formatting:

#!/bin/bash
....
# ============================================
# This command produces number of the same width, padded to 3 characters.  For example, 001.
printf -v NUM %.3d $SLURM_ARRAY_TASK_ID
DATA="data-${NUM}.dat"

./myapplication $DATA

Mapping job index to a line in a file

The command to extract a specific line from a text file, in this case the line number 10:

sed -n 10p filename.txt

With this, we can put data file names into into a text file index.idx, one data file name per line. Then in the job script the data file names can be extracted using the command above from the line specified by the job task index:

#!/bin/bash
....
# ============================================
MAP_FILE="index.idx"
DATA=`sed -n ${SLURM_ARRAY_TASK_ID}p $MAP_FILE`

./myapplication $DATA

Mapping job index to multiple data files in a file

If you have to process several computations from an element job of the job array you are going to run, you job script have to run several over several data files form your data mapping file.

For example, you want to process 1000 jobs using an array of 20 jobs, from 0 to 19, processing 50 data files in each element job. The data mapping file, index.idx, will look like that:

data-0001.dat
data-0002.dat
data-0003.dat
....
data-0998.dat
data-0999.dat
data-1000.dat

Then, the array job script, array.slurm, can be:

#! /bin/bash
# ====================================================
....
#SBATCH --array=0-19

# ====================================================
SIZE=50
INDEX_FILE="index.idx"

IDXBEG=$(( SLURM_ARRAY_TASK_ID * SIZE ))
IDXEND=$(( IDXBEG + SIZE - 1 ))

for IDX in `seq $IDXBEG $IDXEND`; do
        DATA=`sed -n ${IDX}p $INDEX_FILE`

        # add your work here....
        ./myapplication $DATA
done
# ====================================================

More Arrays

Array options

You can indicate a stride for the array elements to be used in element jobs:

#SBATCH --array=1-100:4

This option in your job script sets the stride to 4 and will run 25 jobs with SLURM_ARRAY_TASK_ID having the values of {1, 5, 9, .... 97}.

Another option is to indicate the number of element jobs that can run at the same time. By default all the element jobs can be executed at the same time, provided that resources are available. With this option you can limit the number of jobs to, for example, one:

#SBATCH --array=1-100%1

you can combine it with the stride as well, to limit your 25 jobs to run 5 at a time:

#SBATCH --array=1-100:4%5

Limiting a job array to run 1 job at a time may be a practical way to resubmit the same job a fixed number of times. If you want to run some simulation for 1 000 000 steps, it can be done as an array job of, say 100 element, each element running 10 000 steps, one after another.

Array Environmental variables

For a first element jobs in an --array=1-100 the following variables will be set to

SLURM_JOB_ID=87654321
SLURM_ARRAY_JOB_ID=87654321
SLURM_ARRAY_TASK_ID=1
SLURM_ARRAY_TASK_COUNT=100
SLURM_ARRAY_TASK_MAX=100
SLURM_ARRAY_TASK_MIN=1