What is a scheduler?

What's a job?

On computers we are most often familiar with graphical user interfaces (GUIs). There are windows, menus, buttons; we click here and there and the system responds. On compute clusters run by the UofC and Compute Canada the environment is different. To begin with, you control it by typing, not clicking. This is called a command line interface. Most of the compute clusters are controlled by the Linux operating system, rather than MS Windows or MacOS. Furthermore, a program you would like to run may not begin immediately, but may instead be put in a waiting list. The computation will only start when the necessary computational resources become available.

You prepare a small text file called a job script that basically says what program to run, where to get the input, and where to put the output. You submit this job script to a piece of software called the scheduler which decides when and where it will run. Once the job has finished you can retrieve the results of the calculation. Normally there is no interaction between you and the program while the job is running, although you can check on its progress if you wish.

Here's a very simple job script, simple_job.slurm:

#!/bin/bash
#SBATCH --time=00:01:00
echo 'Hello, world!'
sleep 30  

It runs the programs echo and sleep, there is no input, and the output will go to a default location. Lines starting with #SBATCH are directives to the scheduler, telling it things about what the job needs to run. This job, for example, only needs one minute of run time (00:01:00).

The job scheduler

The job scheduler is a piece of software with multiple responsibilities. It must

• maintain a database of jobs,
• enforce policies regarding limits and priorities,
• ensure resources are not overloaded, for example by only assigning each CPU core to one job at a time,
• decide which jobs to run and on which compute nodes,
• launch jobs on those nodes, and
• clean up after each job finishes.

On UofC and Compute Canada clusters, these responsibilities are handled by the Slurm Workload Manager. All the examples and syntax shown on this page are for Slurm.

Slurm Quick Start User Guide:

https://slurm.schedmd.com/quickstart.html

Requesting resources

You use the job script to ask for the resources needed to run your calculation. Among the resources associated with a job are time and number of processors. In the example above, the time requested is one minute and there will be one processor allocated by default since no specific number is given. Please refer to Examples of job scripts for other types of requests such as multiple processors, memory capacity and special processors such as GPUs.

It is important to specify those parameters well. If you ask for less than the calculation needs, the job will be killed for exceeding the requested time or memory limit. If you ask for more than it needs, the job may wait longer than necessary before it starts, and once running it will needlessly prevent others from using those resources.

A basic SLURM job

We can submit the job script simple_job.slurm shown above with sbatch:

[someuser@host ~]$ sbatch simple_job.slurm
Submitted batch job 1234
[someuser@host ~]$ squeue -u $USER
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1234   cpu2019 simple_j someuser  R       0:03      1 zeno001
[someuser@host ~]$ cat slurm-1234.out
Hello, world!

Look at the ST column in the output of squeue to determine the status of your jobs. The two most common states are "PD" for "pending" or "R" for "running". When the job has finished it no longer appears in the squeue output.

Notice that each job is assigned a "job ID", a unique identification number printed when you submit the job --- 1234 in this example. You can have more than one job in the system at a time, and the ID number can be used to distinguish them even if they have the same name. And finally, because we didn't specify anywhere else to put it the output is placed in a file named with the same job ID number, slurm‑1234.out.

You can also specify options to sbatch on the command line. So for example,

[someuser@host ~]$ sbatch --time=00:30:00 simple_job.sh 

will change the time limit of the job to 30 minutes. Any option can be overridden in this way.

Choosing where the output goes

If you want the output file to have a more distinctive name than slurm‑1234.out, you can use --output to change it. The following script sets a job name which will appear in squeue output, and sends the output to a file prefixed with the job name and containing the job ID number.

name_output.slurm:

#!/bin/bash
#SBATCH --time=00:01:00
#SBATCH --job-name=test
#SBATCH --output=test-%J.out
echo 'Hello, world!'

Error output will normally appear in the same file, just as it would if you were typing commands interactively. If you wish you can split the standard error channel (stderr) from the standard output channel (stdout)

by specifying a file name with the ‑e option.