How to use array jobs in SLURM

Array jobs

Job arrays in SLURM allow you to run many jobs using one and same job script. The job script you submit with the sbatch command does not accept any parameters. Like this:

$ sbatch array_job.slurm

If you have many data files you want to process in multiple jobs, this approach may be a convenient alternative to generating many separate job scripts for each data file and submitting them one by one.

An array job script example, array_job.slurm:

#!/bin/bash
# ============================================
#SBATCH --job-name=my_array_job
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=4gb
#SBATCH --time=0-01:00:00

#SBATCH --array=1-10
# ============================================

./myapplication $SLURM_ARRAY_TASK_ID

This job script will create on auxiliary job in the queue which will generate 10 element jobs of the array from 1 to 10. The job scripts for each element job of the array will be exactly the same. The difference comes from the value of the environmental variable $SLURM_ARRAY_TASK_ID which is set to the number of this element job in the array (index).

That is the first job of the array will execute command line:

 ./myapplication 1

and the 8th element job will run

 ./myapplication 8

and so on. The auxiliary job will stay in the queue until all the elemental jobs are done. The resources requested in the job script are allocated to each elemental job separately, that is, each job gets its own 4 GB of RAM and 1 CPU, as requested in the example.

Simple case of numbered data files

#!/bin/bash
....
# ============================================
DATA="data-${SLURM_ARRAY_TASK_ID}.dat"

./myapplication $DATA