Abaqus

2020-05-31T15:04:23Z

Phillips: /* Licensing considerations */

Abaqus

2020-05-31T15:03:20Z

Phillips: /* Introduction */

2020-05-29T14:48:29Z

Phillips:

[[Category:GROMACS]]
[[Category:Software]]
[[Category:ARC]]
= General =

* '''GROMACS''' home site: http://www.gromacs.org/

* Manuals: http://manual.gromacs.org/documentation/

'''GROMACS''' is a versatile package to perform molecular dynamics,
i.e. simulate the Newtonian equations of motion for systems with hundreds to millions of particles.

It is primarily designed for biochemical molecules like proteins,
lipids and nucleic acids that have a lot of complicated bonded interactions,
but since '''GROMACS''' is extremely fast at calculating the nonbonded interactions (that usually dominate simulations)
many groups are also using it for research on non-biological systems, e.g. polymers.

= Using GROMACS on ARC =

Researchers using '''GROMACS''' on ARC are expected to be generally familiar with its capabilities,
input file types and their formats and the use of checkpoint files to restart simulations.

Like other calculations on ARC systems, '''GROMACS''' is run by submitting an appropriate script for batch scheduling using the '''sbatch''' command.
For more information about submitting jobs, see [[Running jobs]] article.

=== GROMACS modules ===

Currently there are several software modules on ARC that provide different versions of '''GROMACS'''.
The versions differ in the release date as well as the CPU architecture the software is compiled for.

You can see them using the <code>module</code> command:
<pre>
$ module avail gromacs

----------- /global/software/Modules/3.2.10/modulefiles ---------
gromacs/2016.3-gnu
gromacs/2018.0-gnu
gromacs/2019.6-legacy
gromacs/2019.6-skylake
gromacs/5.0.7-gnu
</pre>

The names of the modules give hints on the specific version of '''GROMACS''' they provide access to.
* The '''gnu''' suffix indicates that those versions have been compiled with GNU GCC compiler.
: In these specific cases, GCC 4.8.5.

* '''GROMACS''' '''2019.6''' was compiled using GCC 7.3.0 for two different CPU kinds, the old kind, '''legacy''', and the new kind, '''skylake'''.
The '''legacy''' module should be used on compute nodes before 2019, and the '''skylake''' module is for node from 2019 and up.

* All '''GROMACS''' versions provided by all the modules have support for '''GPU computations''', even though it may not be practical to run it on GPU nodes due to limited resources.

A module has to be loaded before '''GROMACS''' can be used on ARC. Like this:
<pre>
$ gmx --version
bash: gmx: command not found...

$ module load gromacs/2019.6-legacy

$ gmx --version
:-) GROMACS - gmx, 2019.6 (-:

GROMACS is written by:
Emile Apol Rossen Apostolov Paul Bauer Herman J.C. Berendsen
.....
.....
GROMACS version: 2019.6
Precision: single
Memory model: 64 bit
MPI library: none
OpenMP support: enabled (GMX_OPENMP_MAX_THREADS = 64)
GPU support: CUDA
SIMD instructions: SSE4.1
FFT library: fftw-3.3.7-sse2-avx
RDTSCP usage: enabled
TNG support: enabled
Hwloc support: hwloc-1.11.8
Tracing support: disabled
C compiler: /global/software/gcc/gcc-7.3.0/bin/gcc GNU 7.3.0
C compiler flags: -msse4.1 -O3 -DNDEBUG -funroll-all-loops -fexcess-precision=fast
C++ compiler: /global/software/gcc/gcc-7.3.0/bin/g++ GNU 7.3.0
C++ compiler flags: -msse4.1 -std=c++11 -O3 -DNDEBUG -funroll-all-loops -fexcess-precision=fast
CUDA compiler: /global/software/cuda/cuda-10.0.130/bin/nvcc nvcc: NVIDIA (R) Cuda compiler driver;Copyright (c) 2005-2018 NVIDIA Corporation;Built on Sat_Aug_25_21:08:01_CDT_2018;Cuda compilation tools, release 10.0, V10.0.130
CUDA compiler flags:-gencode;arch=compute_30,code=sm_30;-gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_52,code=sm_52;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=compute_75;-use_fast_math;;; ;-msse4.1;-std=c++11;-O3;-DNDEBUG;-funroll-all-loops;-fexcess-precision=fast;
CUDA driver: 9.10
CUDA runtime: N/A
</pre>

=== Running a GROMACS Job ===

To run your simulation on ARC cluster you have to have:
(1) a set of '''GROMACS''' input files and
(2) a SLURM job script '''.slurm'''.

Place your input files for a simulation into a separate directory and
prepare an appropriate '''job script''' for it.
<pre>
$ ls -l

-rw-r--r-- 1 drozmano drozmano 7224622 Jan 29 2014 bilayer.gro
-rw-r--r-- 1 drozmano drozmano 2567 May 21 12:14 bilayer.mdp
-rw-r--r-- 1 drozmano drozmano 87 Apr 29 2016 bilayer.top
drwxr-xr-x 1 drozmano drozmano 200 May 20 16:27 ff
-rw-r--r-- 1 drozmano drozmano 504 May 20 16:30 ff.top
-rwxr-xr-x 1 drozmano drozmano 1171 May 21 13:45 job.slurm
</pre>

Here:
* <code>bilayer.top</code> -- contains the topology of the system.
* <code>bilayer.gro</code> -- contains initial configuration (positions of atoms) of the system.
* <code>bilayer.mdp</code> -- contains parameters of the simulation run ('''GROMACS''' settings).
* <code>ff</code> -- a directory containing external custom force field (models). Not required if only standard models are used.
* <code>ff.top</code> -- a topology file that includes required models from available force fields.
: This file is included by the <code>bilayer.top</code> file.
* <code>job.slurm</code> -- a '''SLURM jobs script''' that is used to submit this calculation to the cluster.

'''On ARC''', you can get '''an example''' with the files shown above with the commands:
<pre>
$ tar xvf /global/software/gromacs/tests-2019/bilayer.tar.bz2

$ cd bilayer
$ ls -l
....
</pre>

At this point you can '''submit your job''' to ARC's scheduler ('''SLURM''').
<pre>
$ sbatch job.slurm

Submitted batch job 5570681
</pre>
You can check the status of the job using the '''job ID''' from the confirmation message above.
<pre>
$ squeue -j 5570681

JOBID USER STATE TIME_LIMIT TIME NODES TASKS CPUS MIN_MEMORY NODELIST
5570681 drozmano RUNNING 5:00:00 0:15 1 1 40 8G fc1
</pre>
The <code>squeue</code> command output may look different in your case depending on your settings.

After the job is over and it does not show in the <code>squeue</code> output
we can check the results.
<pre>
$ ls -l

-rw-r--r-- 1 drozmano drozmano 7224622 Jan 29 2014 bilayer.gro
-rw-r--r-- 1 drozmano drozmano 2567 May 21 12:14 bilayer.mdp
-rw-r--r-- 1 drozmano drozmano 87 Apr 29 2016 bilayer.top
-rw-r--r-- 1 drozmano drozmano 2594772 May 21 14:37 bilayer.tpr
-rw-r--r-- 1 drozmano drozmano 7224622 May 21 14:37 confout.gro
-rw-r--r-- 1 drozmano drozmano 1772 May 21 14:37 ener.edr
drwxr-xr-x 1 drozmano drozmano 200 May 20 16:27 ff
-rw-r--r-- 1 drozmano drozmano 504 May 20 16:30 ff.top
-rwxr-xr-x 1 drozmano drozmano 1172 May 21 14:36 job.slurm
-rw-r--r-- 1 drozmano drozmano 82051 May 21 14:37 md.log
-rw-r--r-- 1 drozmano drozmano 10949 May 21 14:37 mdout.mdp
-rw-r--r-- 1 drozmano drozmano 13162 May 21 14:37 slurm-5570681.out
-rw-r--r-- 1 drozmano drozmano 2514832 May 21 14:37 state.cpt
-rw-r--r-- 1 drozmano drozmano 4309444 May 21 14:37 traj_comp.xtc
</pre>
The '''new files''' here are:
* <code>bilayer.tpr</code> -- Binary input file that is generated by the <code>gmx grompp</code> before the actual simulation.
: This file can be generated before submitting the job, if needed. Here it is created in the job script.

* <code>confout.gro</code> -- a configuration file containing the final atomic positions at the end of the simulation.
* <code>ener.edr</code> -- a file with energy data. Can be used for later analysis.
* <code>md.log</code> -- the main log file for the simulation.
* <code>mdout.mdp</code> -- a file containing all simulation parameters as used by GROMACS. Based on <code>bilayer.mdp</code>.
* <code>state.cpt</code> -- a binary checkpoint file containing the system state at the end of the simulation.
: This file can be used to continue simulations further.
* <code>traj_com.xtc</code> -- a trajectory file containing atomic positions at some time points during the simulation.

* <code>slurm-5570681.out</code> -- The intercept of the output printed on screen during the simulation.
: Done by '''SLURM''' for you. The number in the name is the '''job ID''' of the job.

If '''something is not working''' the way you expected, then the <code>slurm-5570681.out</code> file is the '''first''' place you should examine.

The <code>md.log</code> is the '''next''' thing to look into.

If everything is as expected, then your '''computation is done''' and you can use the results.
'''Success!'''

You may want to check the output file and the main log anyways:
<pre>
# Press "q" to exit the text viewer.
$ less slurm-5570681.out
....

$ less md.log
....
</pre>

=== The job script ===

If you have input files as in the example above, and you use your own computer,
then to run the simulation,
you have to generate a binary input file for the <code>mdrun</code> '''GROMACS''' command first.
<pre>
$ gmx grompp -v -f bilayer.mdp -c bilayer.gro -p bilayer.top -o bilayer.tpr

:-) GROMACS - gmx grompp, 2019.6 (-:

GROMACS is written by:
Emile Apol Rossen Apostolov Paul Bauer Herman J.C. Berendsen
.....
.....
Checking consistency between energy and charge groups...
Calculating fourier grid dimensions for X Y Z
Using a fourier grid of 112x112x52, spacing 0.116 0.116 0.112
Estimate for the relative computational load of the PME mesh part: 0.13
This run will generate roughly 14 Mb of data
writing run input file...

There were 2 notes

$ ls -l
-rw-r--r-- 1 drozmano drozmano 7224622 Jan 29 2014 bilayer.gro
-rw-r--r-- 1 drozmano drozmano 2567 May 21 12:14 bilayer.mdp
-rw-r--r-- 1 drozmano drozmano 87 Apr 29 2016 bilayer.top
-rw-r--r-- 1 drozmano drozmano 2594772 May 21 15:06 bilayer.tpr
drwxr-xr-x 1 drozmano drozmano 200 May 20 16:27 ff
-rw-r--r-- 1 drozmano drozmano 504 May 20 16:30 ff.top
</pre>
At this point you can run the simulation like this:
<pre>
$ gmx mdrun -v -s bilayer.tpr

:-) GROMACS - gmx grompp, 2019.6 (-:

GROMACS is written by:
Emile Apol Rossen Apostolov Paul Bauer Herman J.C. Berendsen
......
... lots-of-output ...
......
Writing final coordinates.
^Mstep 10000, remaining wall clock time: 0 s
Core t (s) Wall t (s) (%)
Time: 1557.342 38.939 3999.4
(ns/day) (hour/ns)
Performance: 22.191 1.082

GROMACS reminds you: "In ..."
</pre>
To run this calculation on a cluster via the '''SLURM''' scheduling system you have to provide a job script
that does '''two things''':
# Provides '''steps that run''' the simulation, and
# Requests all '''necessary computational resources''' that are needed for this simulation.

The '''steps to run''' the simulation you already know:
# Load a desired '''GROMACS''' module to activate the software.
# Generate the binary input, the '''.tpr''' file.
# Run the '''mdrun''' command on it.

The '''computational resources''' for the run include:
* the '''Number of CPUs''',
* the '''Amount of memory''' (RAM), and
* the '''Time''' sufficient to complete the computation.

Below, '''several examples of job scripts''' are given.

These scripts are suitable for use on ARC depending on
* what '''part of the cluster''' the job should run on and
* the '''number of CPUs''' the job is going to use.

==== Single node (modern) job script ====
This script is for jobs that use up to '''one''' full modern node (2019 and later).
These nodes are in the list of default partitions and have 40 CPUs each.

This specific example requests '''40 CPUs on 1 node''' for '''5 hours'''.
It also requests '''16GB of RAM''' on the node.
The simulation runs as '''1 process of 40 threads'''.

'''job-single.slurm''':
<pre>
#!/bin/bash
# ================================================================
#SBATCH --job-name=gro_test

#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=40
#SBATCH --mem=16GB
#SBATCH --time=0-05:00:00
# ================================================================
module purge
module load gromacs/2019.6-skylake
# ================================================================
echo "Starting at `date`."
echo "========================="
# Input files.
MDP=bilayer.mdp
TOP=bilayer.top
GRO=bilayer.gro

# Binary input file to generate.
TPR=bilayer.tpr

# Preprocess the input files.
gmx grompp -v -f $MDP -c $GRO -p $TOP -o $TPR

# Check if preprocessing have gone well.
if ! test -e $TPR; then
echo "ERROR: Could not create a TPR file for the run. Aborting."
exit
fi

# Run the simulation.
OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

gmx mdrun -v -s $TPR
echo "========================="
echo "Done at `date`."
# ================================================================
</pre>

==== Multi node (modern) job script ====
This script is for jobs that use several modern nodes (2019 and later).
These nodes are in the list of default partitions and have 40 CPUs each.

This specific example requests '''80 CPUs on 2 nodes''' (40 CPUs each) for '''5 hours'''.
It also requests '''16GB of RAM''' on each node. The simulation runs as '''80 one-threaded processes'''.

'''job-multi.slurm''':
<pre>
#!/bin/bash
# ================================================================
#SBATCH --job-name=gro_test

#SBATCH --nodes=2
#SBATCH --ntasks-per-node=40
#SBATCH --cpus-per-task=1
#SBATCH --mem=16GB
#SBATCH --time=0-05:00:00
# ================================================================
module purge
module load gromacs/2019.6-skylake
# ================================================================
echo "Starting at `date`."
echo "========================="
# Input files.
MDP=bilayer.mdp
TOP=bilayer.top
GRO=bilayer.gro

# Binary input file to generate.
TPR=bilayer.tpr

# Preprocess the input files.
gmx grompp -v -f $MDP -c $GRO -p $TOP -o $TPR

# Check if preprocessing have gone well.
if ! test -e $TPR; then
echo "ERROR: Could not create a TPR file for the run. Aborting."
exit
fi

# Run the simulation.
OMP_NUM_THREADS=1
mpiexec gmx_mpi mdrun -v -s $TPR

echo "========================="
echo "Done at `date`."
# ================================================================
</pre>

==== Single node (legacy) job script ====

This script is for jobs that use up to '''one''' full legacy node
('''parallel''', '''cpu2013''', '''lattice''' partitions).
The partition has to be specified in the resource request.

This specific example requests '''12 CPUs on 1 node''' in the '''parallel''' legacy partition for '''5 hours'''.
It also requests '''23GB of RAM''' on the node (all available memory on this kind).
The simulation runs as '''1 process of 12 threads'''.

'''job-single-legacy.slurm''':
<pre>
#!/bin/bash
# ================================================================
#SBATCH --job-name=gro_test

#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=12
#SBATCH --mem=23GB
#SBATCH --time=0-12:00:00

#SBATCH --partition=parallel
# ================================================================
module purge
module load gromacs/2019.6-legacy
# ================================================================
echo "Starting at `date`."
echo "========================="
# Input files.
MDP=bilayer.mdp
TOP=bilayer.top
GRO=bilayer.gro

# Binary input file to generate.
TPR=bilayer.tpr

# Preprocess the input files.
gmx grompp -v -f $MDP -c $GRO -p $TOP -o $TPR

# Check if preprocessing have gone well.
if ! test -e $TPR; then
echo "ERROR: Could not create a TPR file for the run. Aborting."
exit
fi

# Run the simulation.
OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

gmx mdrun -v -s $TPR
echo "========================="
echo "Done at `date`."
# ================================================================
</pre>

==== Multi node (legacy) job script ====

This script is for jobs that use several legacy node
('''parallel''', '''cpu2013''', '''lattice''' partitions).
The partition has to be specified in the resource request.

This specific example requests '''48 CPUs on 4 nodes''' in the '''parallel''' legacy partition (12 CPUs each) for '''5 hours'''.
It also requests '''23GB of RAM''' on each node. The simulation runs as '''48 one-threaded processes'''.

'''job-multi.slurm''':
<pre>
#!/bin/bash
# ================================================================
#SBATCH --job-name=gro_test

#SBATCH --nodes=4
#SBATCH --ntasks-per-node=12
#SBATCH --cpus-per-task=1
#SBATCH --mem=23GB
#SBATCH --time=0-05:00:00

#SBATCH --partition=parallel
# ================================================================
module purge
module load gromacs/2019.6-legacy
# ================================================================
echo "Starting at `date`."
echo "========================="
# Input files.
MDP=bilayer.mdp
TOP=bilayer.top
GRO=bilayer.gro

# Binary input file to generate.
TPR=bilayer.tpr

# Preprocess the input files.
gmx grompp -v -f $MDP -c $GRO -p $TOP -o $TPR

# Check if preprocessing have gone well.
if ! test -e $TPR; then
echo "ERROR: Could not create a TPR file for the run. Aborting."
exit
fi

# Run the simulation.
OMP_NUM_THREADS=1
mpiexec gmx_mpi mdrun -v -s $TPR

echo "========================="
echo "Done at `date`."
# ================================================================
</pre>

==== Tip ====

Sometimes it is useful to run the preprocessor to generate the binary input before submitting a job.
In the example above there had been two notes about the setup and you have to make sure that
you are ready to continue with the simulation despite them.

Also, if there are problems with your input you will be able to see them right away without going through job submission.

= Misc =

== Performance ==

Performance measurements for '''GROMACS 2019.6''' on the 2019 compute nodes using
different parallelization options and number of CPUs.

The '''bilayer''' system of ~105000 atoms was simulated for 100000 steps.
<pre>
-------------------------------------------------------------------------------
#CPUs #Nodes Processes Threads Wall Time Performance Efficiency
per node per proc (s) (ns/day) (%)
-------------------------------------------------------------------------------
1 1 1 1 1031.6 0.84 100.0

10 1 1 10 119.0 7.26 86.4
20 1 1 20 65.9 13.11 78.0
40 1 1 40 37.6 22.97 68.4

10 1 10 1 126.4 6.83 81.3
20 1 20 1 78.0 11.08 66.0
40 1 40 1 45.3 19.07 56.8

40 1 20 2 48.3 17.90 53.3
40 1 10 4 43.9 19.68 58.6
36 1 6 6 49.4 17.50 57.9

80 2 10 4 30.5 28.34 42.2
80 2 20 2 32.6 26.53 39.4
80 2 40 1 30.3 28.49 42.4

120 3 40 1 20.8 41.51 41.2
160 4 40 1 19.2 44.95 33.4
-------------------------------------------------------------------------------
</pre>

'''Observations''':
* If you want to run the job on a '''single node''', then use '''1 process''' with as many threads as the number of CPUs you request.

* If you need '''more than 1 node''', then run '''1-threaded MPI processes''' for each CPU you request.

* Going '''beyond 3 nodes''' may not be computationally efficient on ARC.

== Selected GROMACS commands ==
=== gmx ===
<pre>
SYNOPSIS

gmx [-[no]h] [-[no]quiet] [-[no]version] [-[no]copyright] [-nice <int>]
[-[no]backup]

OPTIONS

Other options:

-[no]h (no)
Print help and quit
-[no]quiet (no)
Do not print common startup info or quotes
-[no]version (no)
Print extended version information and quit
-[no]copyright (yes)
Print copyright information on startup
-nice <int> (19)
Set the nicelevel (default depends on command)
-[no]backup (yes)
Write backups if output files exist

Additional help is available on the following topics:
commands List of available commands
selections Selection syntax and usage
To access the help, use 'gmx help <topic>'.
For help on a command, use 'gmx help <command>'.
</pre>

=== gmx grompp ===
Preprocess input files.
<pre>
$ gmx help grompp

SYNOPSIS

gmx grompp [-f [<.mdp>]] [-c [<.gro/.g96/...>]] [-r [<.gro/.g96/...>]]
[-rb [<.gro/.g96/...>]] [-n [<.ndx>]] [-p [<.top>]]
[-t [<.trr/.cpt/...>]] [-e [<.edr>]] [-ref [<.trr/.cpt/...>]]
[-po [<.mdp>]] [-pp [<.top>]] [-o [<.tpr>]] [-imd [<.gro>]]
[-[no]v] [-time <real>] [-[no]rmvsbds] [-maxwarn <int>]
[-[no]zero] [-[no]renum]

OPTIONS

Options to specify input files:

-f [<.mdp>] (grompp.mdp)
grompp input file with MD parameters
-c [<.gro/.g96/...>] (conf.gro)
Structure file: gro g96 pdb brk ent esp tpr
-r [<.gro/.g96/...>] (restraint.gro) (Opt.)
Structure file: gro g96 pdb brk ent esp tpr
-rb [<.gro/.g96/...>] (restraint.gro) (Opt.)
Structure file: gro g96 pdb brk ent esp tpr
-n [<.ndx>] (index.ndx) (Opt.)
Index file
-p [<.top>] (topol.top)
Topology file
-t [<.trr/.cpt/...>] (traj.trr) (Opt.)
Full precision trajectory: trr cpt tng
-e [<.edr>] (ener.edr) (Opt.)
Energy file

Options to specify input/output files:

-ref [<.trr/.cpt/...>] (rotref.trr) (Opt.)
Full precision trajectory: trr cpt tng

Options to specify output files:

-po [<.mdp>] (mdout.mdp)
grompp input file with MD parameters
-pp [<.top>] (processed.top) (Opt.)
Topology file
-o [<.tpr>] (topol.tpr)
Portable xdr run input file
-imd [<.gro>] (imdgroup.gro) (Opt.)
Coordinate file in Gromos-87 format

Other options:

-[no]v (no)
Be loud and noisy
-time <real> (-1)
Take frame at or first after this time.
-[no]rmvsbds (yes)
Remove constant bonded interactions with virtual sites
-maxwarn <int> (0)
Number of allowed warnings during input processing. Not for normal
use and may generate unstable systems
-[no]zero (no)
Set parameters for bonded interactions without defaults to zero
instead of generating an error
-[no]renum (yes)
Renumber atomtypes and minimize number of atomtypes
</pre>

=== gmx mdrun ===

'''gmx mdrun''' is the main computational chemistry engine within '''GROMACS'''.
It performs Molecular Dynamics simulations, but it can also perform
Stochastic Dynamics, Energy Minimization, test particle insertion or
(re)calculation of energies. Normal mode analysis is another option.

<pre>
SYNOPSIS

gmx mdrun [-s [<.tpr>]] [-cpi [<.cpt>]] [-table [<.xvg>]] [-tablep [<.xvg>]]
[-tableb [<.xvg> [...]]] [-rerun [<.xtc/.trr/...>]] [-ei [<.edi>]]
[-multidir [<dir> [...]]] [-awh [<.xvg>]] [-membed [<.dat>]]
[-mp [<.top>]] [-mn [<.ndx>]] [-o [<.trr/.cpt/...>]]
[-x [<.xtc/.tng>]] [-cpo [<.cpt>]] [-c [<.gro/.g96/...>]]
[-e [<.edr>]] [-g [<.log>]] [-dhdl [<.xvg>]] [-field [<.xvg>]]
[-tpi [<.xvg>]] [-tpid [<.xvg>]] [-eo [<.xvg>]] [-devout [<.xvg>]]
[-runav [<.xvg>]] [-px [<.xvg>]] [-pf [<.xvg>]] [-ro [<.xvg>]]
[-ra [<.log>]] [-rs [<.log>]] [-rt [<.log>]] [-mtx [<.mtx>]]
[-if [<.xvg>]] [-swap [<.xvg>]] [-deffnm <string>] [-xvg <enum>]
[-dd <vector>] [-ddorder <enum>] [-npme <int>] [-nt <int>]
[-ntmpi <int>] [-ntomp <int>] [-ntomp_pme <int>] [-pin <enum>]
[-pinoffset <int>] [-pinstride <int>] [-gpu_id <string>]
[-gputasks <string>] [-[no]ddcheck] [-rdd <real>] [-rcon <real>]
[-dlb <enum>] [-dds <real>] [-gcom <int>] [-nb <enum>]
[-nstlist <int>] [-[no]tunepme] [-pme <enum>] [-pmefft <enum>]
[-bonded <enum>] [-[no]v] [-pforce <real>] [-[no]reprod]
[-cpt <real>] [-[no]cpnum] [-[no]append] [-nsteps <int>]
[-maxh <real>] [-replex <int>] [-nex <int>] [-reseed <int>]

OPTIONS

Options to specify input files:

-s [<.tpr>] (topol.tpr)
Portable xdr run input file
-cpi [<.cpt>] (state.cpt) (Opt.)
Checkpoint file
-table [<.xvg>] (table.xvg) (Opt.)
xvgr/xmgr file
-tablep [<.xvg>] (tablep.xvg) (Opt.)
xvgr/xmgr file
-tableb [<.xvg> [...]] (table.xvg) (Opt.)
xvgr/xmgr file
-rerun [<.xtc/.trr/...>] (rerun.xtc) (Opt.)
Trajectory: xtc trr cpt gro g96 pdb tng
-ei [<.edi>] (sam.edi) (Opt.)
ED sampling input
-multidir [<dir> [...]] (rundir) (Opt.)
Run directory
-awh [<.xvg>] (awhinit.xvg) (Opt.)
xvgr/xmgr file
-membed [<.dat>] (membed.dat) (Opt.)
Generic data file
-mp [<.top>] (membed.top) (Opt.)
Topology file
-mn [<.ndx>] (membed.ndx) (Opt.)
Index file

Options to specify output files:

-o [<.trr/.cpt/...>] (traj.trr)
Full precision trajectory: trr cpt tng
-x [<.xtc/.tng>] (traj_comp.xtc) (Opt.)
Compressed trajectory (tng format or portable xdr format)
-cpo [<.cpt>] (state.cpt) (Opt.)
Checkpoint file
-c [<.gro/.g96/...>] (confout.gro)
Structure file: gro g96 pdb brk ent esp
-e [<.edr>] (ener.edr)
Energy file
-g [<.log>] (md.log)
Log file
-dhdl [<.xvg>] (dhdl.xvg) (Opt.)
xvgr/xmgr file
-field [<.xvg>] (field.xvg) (Opt.)
xvgr/xmgr file
-tpi [<.xvg>] (tpi.xvg) (Opt.)
xvgr/xmgr file
-tpid [<.xvg>] (tpidist.xvg) (Opt.)
xvgr/xmgr file
-eo [<.xvg>] (edsam.xvg) (Opt.)
xvgr/xmgr file
-devout [<.xvg>] (deviatie.xvg) (Opt.)
xvgr/xmgr file
-runav [<.xvg>] (runaver.xvg) (Opt.)
xvgr/xmgr file
-px [<.xvg>] (pullx.xvg) (Opt.)
xvgr/xmgr file
-pf [<.xvg>] (pullf.xvg) (Opt.)
xvgr/xmgr file
-ro [<.xvg>] (rotation.xvg) (Opt.)
xvgr/xmgr file
-ra [<.log>] (rotangles.log) (Opt.)
Log file
-rs [<.log>] (rotslabs.log) (Opt.)
Log file
-rt [<.log>] (rottorque.log) (Opt.)
Log file
-mtx [<.mtx>] (nm.mtx) (Opt.)
Hessian matrix
-if [<.xvg>] (imdforces.xvg) (Opt.)
xvgr/xmgr file
-swap [<.xvg>] (swapions.xvg) (Opt.)
xvgr/xmgr file

Other options:

-deffnm <string>
Set the default filename for all file options
-xvg <enum> (xmgrace)
xvg plot formatting: xmgrace, xmgr, none
-dd <vector> (0 0 0)
Domain decomposition grid, 0 is optimize
-ddorder <enum> (interleave)
DD rank order: interleave, pp_pme, cartesian
-npme <int> (-1)
Number of separate ranks to be used for PME, -1 is guess
-nt <int> (0)
Total number of threads to start (0 is guess)
-ntmpi <int> (0)
Number of thread-MPI ranks to start (0 is guess)
-ntomp <int> (0)
Number of OpenMP threads per MPI rank to start (0 is guess)
-ntomp_pme <int> (0)
Number of OpenMP threads per MPI rank to start (0 is -ntomp)
-pin <enum> (auto)
Whether mdrun should try to set thread affinities: auto, on, off
-pinoffset <int> (0)
The lowest logical core number to which mdrun should pin the first
thread
-pinstride <int> (0)
Pinning distance in logical cores for threads, use 0 to minimize
the number of threads per physical core
-gpu_id <string>
List of unique GPU device IDs available to use
-gputasks <string>
List of GPU device IDs, mapping each PP task on each node to a
device
-[no]ddcheck (yes)
Check for all bonded interactions with DD
-rdd <real> (0)
The maximum distance for bonded interactions with DD (nm), 0 is
determine from initial coordinates
-rcon <real> (0)
Maximum distance for P-LINCS (nm), 0 is estimate
-dlb <enum> (auto)
Dynamic load balancing (with DD): auto, no, yes
-dds <real> (0.8)
Fraction in (0,1) by whose reciprocal the initial DD cell size will
be increased in order to provide a margin in which dynamic load
balancing can act while preserving the minimum cell size.
-gcom <int> (-1)
Global communication frequency
-nb <enum> (auto)
Calculate non-bonded interactions on: auto, cpu, gpu
-nstlist <int> (0)
Set nstlist when using a Verlet buffer tolerance (0 is guess)
-[no]tunepme (yes)
Optimize PME load between PP/PME ranks or GPU/CPU (only with the
Verlet cut-off scheme)
-pme <enum> (auto)
Perform PME calculations on: auto, cpu, gpu
-pmefft <enum> (auto)
Perform PME FFT calculations on: auto, cpu, gpu
-bonded <enum> (auto)
Perform bonded calculations on: auto, cpu, gpu
-[no]v (no)
Be loud and noisy
-pforce <real> (-1)
Print all forces larger than this (kJ/mol nm)
-[no]reprod (no)
Try to avoid optimizations that affect binary reproducibility
-cpt <real> (15)
Checkpoint interval (minutes)
-[no]cpnum (no)
Keep and number checkpoint files
-[no]append (yes)
Append to previous output files when continuing from checkpoint
instead of adding the simulation part number to all file names
-nsteps <int> (-2)
Run this number of steps (-1 means infinite, -2 means use mdp
option, smaller is invalid)
-maxh <real> (-1)
Terminate after 0.99 times this time (hours)
-replex <int> (0)
Attempt replica exchange periodically with this period (steps)
-nex <int> (0)
Number of random exchanges to carry out each exchange interval (N^3
is one suggestion). -nex zero or not specified gives neighbor
replica exchange.
-reseed <int> (-1)
Seed for replica exchange, -1 is generate a seed
</pre>
= Support =

Please send any questions regarding using GROMACS on ARC to support@hpc.ucalgary.ca.

Mathematica

2020-05-29T14:47:31Z

Phillips:

2020-05-27T14:50:04Z

Phillips: Added job array section

[[Category:MATLAB]]
[[Category:Software]]
[[Category:ARC]]



= Introduction =

[http://www.mathworks.com/ MATLAB] is a general-purpose high-level programming package for numerical work such as linear algebra, signal processing and other calculations involving matrices or vectors of data. Visualization tools are also included for presentation of results. The basic MATLAB package is extended through add-on components including SIMULINK, and the Image Processing, Optimization, Neural Network, Signal Processing, Statistics and Wavelet Toolboxes, among others.

The main purpose of this page is to show how to use MATLAB on the University of Calgary [[ ARC_Cluster_Guide |ARC (Advanced Research Computing) cluster]]. It is presumed that you already have an account an ARC and have read the material on reserving resources and [[Running jobs|running jobs]] with the Slurm job management system.

= Ways to run MATLAB - License considerations =
At the University of Calgary, Information Technologies has purchased a MATLAB Total Academic Headcount license that allows installation and use of MATLAB on central clusters, such as ARC, as well as on personal workstations throughout the University. Potentially thousands of instances of MATLAB can be run simultaneously, each checking out a license from a central license server. An alternative is to compile MATLAB code into a standalone application. When such an application is run, it does not need to contact the server for a license. This allows researchers to run their calculations on compatible hardware, not necessarily at the University of Calgary, such as on [https://docs.computecanada.ca/wiki/Getting_started Compute Canada clusters (external link)].

For information about installing MATLAB on your own computer, see the Information Technologies Knowledge Base [https://ucalgary.service-now.com/it?id=kb_article&sys_id=787d708213fdbec08246f7b2e144b0d4 article on MATLAB].

= Running MATLAB on the ARC cluster =
Although it is possible to run MATLAB interactively, the expectation is that most calculations with MATLAB will be completed by submitting a batch job script to the Slurm job scheduler with the sbatch command.

For many researchers, the main reason for using ARC for MATLAB-based calculations is to be able to run many instances at the same time. It is recommended in such cases that any parallel processing features be removed from the code and each instance of MATLAB be run on a single CPU core. It is also possible to run MATLAB on multiple cores in an attempt to speed up individual instances, but, this generally results in a less efficient use of the cluster hardware. In the sections that follow, serial and then parallel processing examples are shown.

== Serial MATLAB example==

For the purposes of illustration, suppose the following serial MATLAB code, in a file sawtooth.m, is to be run. If your code does not already have them, add a function statement at the beginning and matching end statement at the end as shown in the example. Other features of this example include calling a function with both numerical and string arguments, incorporating a Slurm environment variable into the MATLAB code and producing graphical output in a non-interactive environment.
<pre>
function sawtooth(nterms,nppcycle,ncycle,pngfilebase)

% MATLAB file example to approximate a sawtooth
% with a truncated Fourier expansion.

% nterms = number of terms in expansion.
% nppcycle = number of points per cycle.
% ncycle = number of complete cycles to plot.
% pngfilebase = base of file name for graph of results.

% 2020-05-14

np=nppcycle*ncycle;

fourbypi=4.0/pi;
y(1:np)=pi/2.0;
x(1:np)=linspace(-pi*ncycle,pi*ncycle,np);

for k=1:nterms
twokm=2*k-1;
y=y-fourbypi*cos(twokm*x)/twokm^2;
end

% Prepare output
% Construct the output file name from the base file name and number of terms
% Also append the Slurm JOBID to keep file names unique from run to run.

job=getenv('SLURM_JOB_ID')
pngfile=strcat(pngfilebase,'_',num2str(nterms),'_',job)
disp(['Writing file: ',pngfile,'.png'])

fig=figure;
plot(x,y);
print(fig,pngfile,'-dpng');

quit
end
</pre>
In preparation to run the sawtooth.m code, create a batch job script, sawtooth.slurm of the form:
<pre>
#!/bin/bash
#SBATCH --time=03:00:00 # Adjust this to match the walltime of your job
#SBATCH --nodes=1 # For serial code, always specify just one node.
#SBATCH --ntasks=1 # For serial code, always specify just one task.
#SBATCH --cpus-per-task=1 # For serial code, always specify just one CPU per task.
#SBATCH --mem=4000m # Adjust to match total memory required, in MB.
#SBATCH --partition=pawson-bf,apophis-bf,razi-bf,single,lattice,parallel,cpu2013,cpu2019

# Sample batch job script for running a MATLAB function with both numerical and string arguments
# 2020-05-14

# Specify the name of the main MATLAB function to be run.
# This would normally be the same as the MATLAB source code file name without a .m suffix).
MAIN="sawtooth"

# Define key parameters for the example calculation.
NTERMS=100
NPPCYCLE=20
NCYCLE=3
PNGFILEBASE="sawtooth"

# Contruct a complete function call to pass to MATLAB
# Note, string arguments should appear to MATLAB enclosed in single quotes
ARGS="($NTERMS,$NPPCYCLE,$NCYCLE,'$PNGFILEBASE')"
MAIN_WITH_ARGS=${MAIN}${ARGS}

echo "Calling MATLAB function: ${MAIN_WITH_ARGS}"

echo "Starting run at $(date)"
echo "Running on compute node $(hostname)"
echo "Running from directory $(pwd)"

# Choose a version of MATLAB by loading a module:
module load matlab/r2019b
echo "Using MATLAB version: $(which matlab)"

# Use -singleCompThread below for serial MATLAB code:
matlab -singleCompThread -batch "${MAIN_WITH_ARGS}" > ${MAIN}_${SLURM_JOB_ID}.out 2>&1

echo "Finished run at $(date)"
</pre>
Note that the above script uses the -batch option on the matlab command line. The MathWorks web page on [https://www.mathworks.com/help/matlab/ref/matlablinux.html running MATLAB on Linux (external link)] starting with Release 2019a of MATLAB, recommends using the -batch option for non-interactive use instead of the similar -r option that is recommended in interactive sessions.

To submit the job to be executed, run:
<pre>
sbatch sawtooth.slurm
</pre>

The job should produce three output files: Slurm script output, MATLAB command output and a PNG file, all tagged with the Slurm Job ID.

== Parallel MATLAB examples==

MATLAB provides several ways of speeding up calculations through parallel processing. These include relying on internal parallelization in which multiple threads are used or by using explicit language features, such as parfor, to start up multiple workers on a compute node. Examples of both approaches are shown below. Using multiple compute nodes for a single MATLAB calculation, which depends on the MATLAB Parallel Server product, is not considered here as there has not been sufficient demand to configure that software on ARC.

For many researchers, submitting many independent serial jobs is a better approach to efficient parallelization than using parallel programming in MATLAB itself. An example of job-based parallelism is also shown below.

=== Thread-based parallel processing ===
First consider an example using multiple cores with MATLAB's built-in thread-based parallelization.

Suppose the following code to calculate eigenvalues of a number of random matrices is in a file eig_thread_test.m . Note the use of the maxNumCompThreads function to control the number of threads (one thread per CPU core). For some years now, MathWorks has marked that function as deprecated, but, it still provides a useful limit to ensure that MATLAB doesn't use more cores than assigned by Slurm.

<pre>
function eig_thread_test(nthreads,matrix_size,nmatrices,results_file)

% Calculate the absolute value of the maximum eigenvalue for each of a number of matrices
% possibly using multiple threads.

% nthreads = number of computational threads to use.
% matrix_size = order of two-dimensional random matrix.
% nmatrices = number of matrices to process.
% results_file = name of file in which to save the maximum eigenvalues

% 2020-05-25

matlab_ncores=feature('numcores')

slurm_ncores_per_task=str2num(getenv('SLURM_CPUS_PER_TASK'))
if(isempty(slurm_ncores_per_task))
slurm_ncores_per_task=1;
disp('SLURM_CPUS_PER_TASK not set')
end

% Set number of computational threads to the minimum of matlab_ncores and slurm_ncores_per_task
% Note Mathworks warns that the maxNumCompThreads function will
% be removed in future versions of MATLAB.

% Use only thread-based parallel processing

intial_matlab_max_ncores = maxNumCompThreads(min([nthreads,slurm_ncores_per_task]));

disp(['Using a maximum of ',num2str(maxNumCompThreads()),' computational threads.'])

tic

for i = 1:nmatrices
e=eig(rand(matrix_size));
eigenvalues(i) = max(abs(e));
end

toc

save(results_file,'eigenvalues','-ascii')

quit
end
</pre>

Here is a job script, eig_thread_test.slurm that can be used to run the eig_thread_test.m code.
The number of threads used for the calculation is controlled by specifying the --cpus-per-task parameter that Slurm uses to control the number of CPU cores assigned to the job.

<pre>
#!/bin/bash

#SBATCH --time=01:00:00 # Adjust this to match the walltime of your job
#SBATCH --nodes=1 # Always specify just one node.
#SBATCH --ntasks=1 # Specify just one task.
#SBATCH --cpus-per-task=8 # The number of threads to use
#SBATCH --mem=4000m # Adjust to match total memory required, in MB.
#SBATCH --partition=pawson-bf,apophis-bf,razi-bf,single,lattice,parallel,cpu2013,cpu2019

# Sample batch job script for running a MATLAB function to test thread-based parallel processing features.
# 2020-05-25

# Specify the name of the main MATLAB function to be run.
# This would normally be the same as the MATLAB source code file name without a .m suffix).
MAIN="eig_thread_test"

# Define key parameters for the example calculation.
NTHREADS=${SLURM_CPUS_PER_TASK}
MATRIX_SIZE=10000
NMATRICES=10
RESULTS_FILE="maximum_eigenvalues_${SLURM_JOB_ID}.txt"

# Contruct a complete function call to pass to MATLAB
# Note, string arguments should appear to MATLAB enclosed in single quotes
ARGS="($NTHREADS,$MATRIX_SIZE,$NMATRICES,'$RESULTS_FILE')"
MAIN_WITH_ARGS=${MAIN}${ARGS}

echo "Calling MATLAB function: ${MAIN_WITH_ARGS}"

echo "Starting run at $(date)"
echo "Running on compute node $(hostname)"
echo "Running from directory $(pwd)"

# Choose a version of MATLAB by loading a module:
module load matlab/r2020a
echo "Using MATLAB version: $(which matlab)"

matlab -batch "${MAIN_WITH_ARGS}" > ${MAIN}_${SLURM_JOB_ID}.out 2>&1

echo "Finished run at $(date)"
</pre>

The above job can be submitted with
<pre>
sbatch eig_thread_test.slurm
</pre>

If assigned to one of the modern partitions (as opposed to the older single, lattice or parallel partitions) the job took about 16 minutes, about 4 times faster than a comparable serial job. Using 8 cores to obtain just a factor of four speed-up is not an efficient use of ARC, but, might be justified in some cases. In some cases, using multiple workers (as discussed in the next section) may be faster than using the same number of cores with thread-based parallelization.

=== Explicit parallel processing using a pool of workers ===

Now consider the more complicated case of creating a pool of workers and using a parfor loop to explicitly parallelize a section of code. Each worker may use one or more cores through MATLAB's internal thread-based parallelization, as in the preceding example. Suppose the following code is in a file eig_parallel_test.m.
<pre>
function eig_parallel_test(nworkers,nthreads,matrix_size,nmatrices,results_file)

% Calculate the absolute value of the maximum eigenvalue for each of a number of matrices
% possibly using multiple threads and multiple MATLAB workers.

% nworkers = number of MATLAB workers to use.
% nthreads = number of threads per worker.
% matrix_size = order of two-dimensional random matrix.
% nmatrices = number of matrices to process.
% results_file = name of file in which to save the maximum eigenvalues

% 2020-05-25

matlab_ncores=feature('numcores')

slurm_ncores_per_task=str2num(getenv('SLURM_CPUS_PER_TASK'))
if(isempty(slurm_ncores_per_task))
slurm_ncores_per_task=1;
disp('SLURM_CPUS_PER_TASK not set')
end

% Set number of computational threads to the minimum of matlab_ncores and slurm_ncores
% Note Mathworks warns that the maxNumCompThreads function will
% be removed in future versions of MATLAB.
% Testing based on remarks at
% https://www.mathworks.com/matlabcentral/answers/158192-maxnumcompthreads-hyperthreading-and-parpool
% shows the maxNumCompThreads has to be called inside the parfor loop.

tic

if ( nworkers > 1 )

% Process with multiple workers
% Check on properties of the local MATLAB cluster.
% One can set properties such as c.NumThreads and c.NumWorkers

parallel.defaultClusterProfile('local')
c = parcluster()
c.NumThreads=nthreads
c.NumWorkers=nworkers

% Create a pool of workers with the current cluster settings.
% Note, testing without the nworkers argument showed a limit of 12 workers even if c.NumWorkers is defined.
parpool(c,nworkers)

ticBytes(gcp);

parfor i = 1:nmatrices
e=eig(rand(matrix_size));
eigenvalues(i) = max(abs(e));
end

tocBytes(gcp)

% Close down the pool.
delete(gcp('nocreate'));

else

% Use only thread-based parallel processing

intial_matlab_max_ncores = maxNumCompThreads(min([nthreads,slurm_ncores_per_task]))

for i = 1:nmatrices
e=eig(rand(matrix_size));
eigenvalues(i) = max(abs(e));
end

end % nworkers test

toc

save(results_file,'eigenvalues','-ascii')

quit
end
</pre>
Of particular note in the preceding example is the section of lines (copied below) that creates a cluster object, c, and modifies the number of threads associated with this object (c.NumThreads=nthreads). In a similar way, one can modify the number of workers (c.NumWorkers=nworkers). Testing showed that if one then used the MATLAB gcp or parpool commands without arguments to create a pool of workers, at most 12 workers were created. However, it was found that by using parpool(c,nworkers), the requested number of workers would be started, even if nworkers > 12.
<pre>
parallel.defaultClusterProfile('local')
c = parcluster()
c.NumThreads=nthreads
c.NumWorkers=nworkers
parpool(c,nworkers)
</pre>

An example Slurm batch job script, eig_parallel_test.slurm, used to test the above code was:
<pre>
#!/bin/bash

#SBATCH --time=03:00:00 # Adjust this to match the walltime of your job
#SBATCH --nodes=1 # Always specify just one node.
#SBATCH --ntasks=1 # Always specify just one task.
#SBATCH --cpus-per-task=8 # Choose --cpus-per-task to match the number of workers * threads per worker
#SBATCH --mem=10000m # Adjust to match total memory required, in MB.
#SBATCH --partition=pawson-bf,apophis-bf,razi-bf,single,lattice,parallel,cpu2013,cpu2019

# Sample batch job script for running a MATLAB function to test parallel processing features
# 2020-05-25

# Specify the name of the main MATLAB function to be run.
# This would normally be the same as the MATLAB source code file name without a .m suffix).
MAIN="eig_parallel_test"

# Define key parameters for the example calculation.
NWORKERS=${SLURM_NTASKS}
NTHREADS=${SLURM_CPUS_PER_TASK}
MATRIX_SIZE=10000
NMATRICES=10
RESULTS_FILE="maximum_eigenvalues_${SLURM_JOB_ID}.txt"

# Contruct a complete function call to pass to MATLAB
# Note, string arguments should appear to MATLAB enclosed in single quotes
ARGS="($NWORKERS,$NTHREADS,$MATRIX_SIZE,$NMATRICES,'$RESULTS_FILE')"
MAIN_WITH_ARGS=${MAIN}${ARGS}

echo "Calling MATLAB function: ${MAIN_WITH_ARGS}"

echo "Starting run at $(date)"
echo "Running on compute node $(hostname)"
echo "Running from directory $(pwd)"

# Choose a version of MATLAB by loading a module:
module load matlab/r2020a
echo "Using MATLAB version: $(which matlab)"

matlab -batch "${MAIN_WITH_ARGS}" > ${MAIN}_${SLURM_JOB_ID}.out 2>&1

echo "Finished run at $(date)"
</pre>
Note that the Slurm SBATCH parameter --cpus-per-task, the total number of cores to use, should be the product of the number of workers and the threads per worker. (It might be argued that one more core should be requested beyond the product of workers and threads, to use for the main MATLAB process, but, for a fully parallelized code, the workers do the great bulk of the calculation and the main MATLAB process uses relatively little CPU time.)

Variations on the above code was tested with many combinations of workers and threads per worker. It was found that if many jobs were started in close succession that some of the jobs failed to start properly.

Also note that in the above example, the number of cores requested is not well matched to the number of workers. If there are 8 workers processing 10 matrices and each worker gets assigned one matrix, there are two left over. The total time for 10 matrices (or for any number of matrices from 9 to 16) would be about double that for 8 matrices. For example, in testing the above code, the time for processing 10 matrices with 8 workers with just one computational thread was 1050 seconds, whereas processing just 8 matrices with 8 workers took only 560 seconds.

=== Job-based parallel processing ===

Slurm provides a feature called job arrays that can sometimes be conveniently used to submit a large number of similar jobs. This can be used effectively when doing a parameter sweep, in which one key value (or several) in the code are changed from run to run. For example, in the eigenvalue calculation code considered in the previous section, one may want to study the effect of changing the matrix size. Alternatively, there may be cases in which the only change from one job to another is the name of an input file that contains the data to be processed.

The key to using the job array feature is to set up the code to depend on a single integer value, the job array index, $SLURM_ARRAY_TASK_ID . When a job is run, the array index variable is replaced by Slurm with a specific value, taken from a corresponding --array argument on the sbatch command line. For example, if the job script shown below is called eig_parallel_array_test.slurm and you run the script as
<pre>
sbatch --array=1000,2000,3000 eig_parallel_array_test.slurm
</pre>
then three jobs will be run, with $SLURM_ARRAY_TASK_ID taking on the values 1000, 2000 and 3000 for the three cases, respectively.

If you had a case in which input files to be processed were data_1.in, data_2.in, ... data_100.in, you could submit 100 jobs to each process one data file with
<pre>
sbatch --array=1-100 script_to_process_one_file.slurm
</pre>

where the job script refers to the files as data_${SLURM_ARRAY_TASK_ID}.in . (Putting the curly brackets around the variable name helps the bash shell from being confused with other text in the file name.)

Testing shows that starting large numbers of jobs in a short period of time, such as can occur when job arrays are used and there are many free compute nodes, can lead to failures. This kind of error appears to be related to delayed file system access when MATLAB tries to write job information to a hidden subdirectory under your home directory. One way to avoid this problem is to introduce a delay between job submissions. This can be done by using a loop of the form:
<pre>
for job in $(seq 1 100)
do
sbatch --array=$job script_to_process_one_file.slurm
done
</pre>

One other caveat regarding the use of job arrays is that the resources requested with the SBATCH directives are for a single instance of the calculation to be completed. So, if the resource requirements are expected to differ significantly from one job to another, you may have to write separate job scripts for each case, rather than using a job array.

Here is an example of a job script to calculate the maximum eigenvalue of a number of random matrices of a given size.
<pre>
#!/bin/bash

#SBATCH --time=01:00:00 # Adjust this to match the walltime of your job
#SBATCH --nodes=1 # Always specify just one node.
#SBATCH --ntasks=1 # Always specify just one task.
#SBATCH --cpus-per-task=1 # For serial code, always specify just one CPU per task.
#SBATCH --mem=4000m # Adjust to match total memory required, in MB.
#SBATCH --partition=pawson-bf,apophis-bf,razi-bf,single,lattice,parallel,cpu2013,cpu2019

# Sample batch job script for running a MATLAB function to illustrate job-based parallel processing with job arrays
# 2020-05-27

# The order of the matrix is specified using the job array index.

# Specify the name of the main MATLAB function to be run.
# This would normally be the same as the MATLAB source code file name without a .m suffix).
MAIN="eig_parallel_test"

# Define key parameters for the example calculation.
NWORKERS=1
NTHREADS=1
MATRIX_SIZE=$SLURM_ARRAY_TASK_ID
NMATRICES=10
RESULTS_FILE="maximum_eigenvalues_${MATRIX_SIZE}_${SLURM_JOB_ID}.txt"

# Contruct a complete function call to pass to MATLAB
# Note, string arguments should appear to MATLAB enclosed in single quotes
ARGS="($NWORKERS,$NTHREADS,$MATRIX_SIZE,$NMATRICES,'$RESULTS_FILE')"
MAIN_WITH_ARGS=${MAIN}${ARGS}

echo "Calling MATLAB function: ${MAIN_WITH_ARGS}"

echo "Starting run at $(date)"
echo "Running on compute node $(hostname)"
echo "Running from directory $(pwd)"

# Choose a version of MATLAB by loading a module:
module load matlab/r2020a
echo "Using MATLAB version: $(which matlab)"

matlab -batch "${MAIN_WITH_ARGS}" > ${MAIN}_${SLURM_JOB_ID}.out 2>&1

echo "Finished run at $(date)"
</pre>

= Standalone Applications =

When running MATLAB code as described in the preceding sections, a connection to the campus MATLAB license server, checking out licenses for MATLAB and any specialized toolboxes needed, is made for each job that is submitted. Currently, with the University of Calgary's Total Academic Headcount license, there are sufficient license tokens to support thousands of simultaneous MATLAB sessions (although ARC usage policy and cluster load will limit individual users to smaller numbers of jobs). However, there may be times at which the license server is slow to respond when large numbers of requests are being handled, or the server may be unavailable temporarily due to network problems. MathWorks offers an alternative way of running MATLAB code that can avoid license server issues by compiling it into a standalone application. A license is required only during the compilation process and not when the code is run. This allows calculations to be run on ARC without concerns regarding the license server. The compiled code can also be run on compatible (64-bit Linux) hardware, not necessarily at the University of Calgary, such as on [https://docs.computecanada.ca/wiki/Getting_started Compute Canada (external link)] clusters.

== Creating a standalone application ==

The MATLAB '''mcc''' command is used to compile source code (.m files) into a standalone executable. There are a couple of important considerations to keep in mind when creating an executable that can be run in a batch-oriented cluster environment. One is that there is no graphical display attached to your session and another is that the number of threads used by the standalone application has to be controlled. There is also an important difference in the way arguments of the main function are handled.

Let's illustrate the process of creating a standalone application for the sawtooth.m code used previously. Unfortunately, if that code is compiled as it is, the resulting compiled application will fail to run properly. The reason is that the compiled code sees all the input arguments as strings instead of interpreting them as numbers. To work around this problem, use a MATLAB function, '''isdeployed''', to determine whether or not the code is being run as a standalone application. Here is a modified version of code, called sawtooth_standalone.m that can be successfully compiled and run as a standalone application.

<pre>
function sawtooth_standalone(nterms,nppcycle,ncycle,pngfilebase)

% MATLAB file example to approximate a sawtooth
% with a truncated Fourier expansion.

% nterms = number of terms in expansion.
% nppcycle = number of points per cycle.
% ncycle = number of complete cycles to plot.
% pngfilebase = base of file name for graph of results.

% 2020-05-21

% Test to see if the code is running as a standalone application
% If it is, convert the arguments intended to be numeric from
% the input strings to numbers

if isdeployed
nterms=str2num(nterms)
nppcycle=str2num(nppcycle)
ncycle=str2num(ncycle)
end

np=nppcycle*ncycle;

fourbypi=4.0/pi;
y(1:np)=pi/2.0;
x(1:np)=linspace(-pi*ncycle,pi*ncycle,np);

for k=1:nterms
twokm=2*k-1;
y=y-fourbypi*cos(twokm*x)/twokm^2;
end

% Prepare output
% Construct the output file name from the base file name and number of terms
% Also append the Slurm JOBID to keep file names unique from run to run.

job=getenv('SLURM_JOB_ID')
pngfile=strcat(pngfilebase,'_',num2str(nterms),'_',job)
disp(['Writing file: ',pngfile,'.png'])

fig=figure;
plot(x,y);
print(fig,pngfile,'-dpng');

quit
end
</pre>

Suppose that the sawtooth_standalone.m file is in a subdirectory src below your current working directory and that the compiled files are going to be written to a subdirectory called deploy. The following commands (at the Linux shell prompt) could be used to compile the code:

<pre>
mkdir deploy
cd src
module load matlab/r2019b
mcc -R -nodisplay \
-R -singleCompThread \
-m -v -w enable \
-d ../deploy \
sawtooth_standalone.m
</pre>

Note the option -singleCompThread has been included in order to limit the executable to just one computational thread.

In the deploy directory, an executable, sawtooth_standalone, will be created along with a script, run_sawtooth_standalone.sh. These two files should be copied to the target machine where the code is to be run.

== Running a standalone application ==

After the standalone executable sawtooth_standalone and corresponding script run_sawtooth_standalone.sh have been transferred to a directory on the target system on which they will be run (whether to a different directory on ARC or to a completely different cluster), a batch job script needs to be created in that same directory. Here is an example batch job script, sawtooth_standalone.slurm, appropriate for the ARC cluster.

<pre>
#!/bin/bash
#SBATCH --time=03:00:00 # Adjust this to match the walltime of your job
#SBATCH --nodes=1 # For serial code, always specify just one node.
#SBATCH --ntasks=1 # For serial code, always specify just one task.
#SBATCH --cpus-per-task=1 # For serial code, always specify just one CPU per task.
#SBATCH --mem=4000m # Adjust to match total memory required, in MB.
#SBATCH --partition=pawson-bf,apophis-bf,razi-bf,single,lattice,parallel,cpu2013,cpu2019

# Sample batch job script for running a compiled MATLAB function
# 2020-05-21

# Specify the name of the compiled MATLAB standalone executable
MAIN="sawtooth_standalone"

# Define key parameters for the example calculation.
NTERMS=100
NPPCYCLE=20
NCYCLE=3
PNGFILEBASE=$MAIN

ARGS="$NTERMS $NPPCYCLE $NCYCLE $PNGFILEBASE"

# Choose the MCR directory according to the compiler version used
MCR=/global/software/matlab/mcr/v97

echo "Starting run at $(date)"
echo "Running on compute node $(hostname)"
echo "Running from directory $(pwd)"

./run_${MAIN}.sh $MCR $ARGS > ${MAIN}_${SLURM_JOB_ID}.out 2>&1

echo "Finished run at $(date)"
</pre>

The job is then submitted with sbatch:

<pre>
sbatch sawtooth_standalone.slurm
</pre>

An important part of the above script is the definition of the variable '''MCR''', which defines the location of the MATLAB Compiler Runtime (MCR) directory. This directory contains files necessary for the standalone application to run. The version of the MCR files specified (v97 in the example, which corresponds to MATLAB R2019b) must match the version of MATLAB used to compile the code.

A list of MATLAB distributions and the corresponding MCR versions is given on the [https://www.mathworks.com/products/compiler/matlab-runtime.html Mathworks web site (external link)]. Some versions installed on ARC are listed below, along with the corresponding installation directory to which the MCR variable should be set if running on ARC. (As of this writing on May 21, 2020, installation of release R2020a has not quite been finished, but, should be ready before the end of the month). If the MCR version you need does not appear in /global/software/matlab/mcr, write to support@hpc.ucalgary.ca to request that it be installed, or use a different version of MATLAB for your compilation.

{| class="WG_Indent3" style="height: 150px; width: 576px" border="2"
| style="text-align: center" | '''MATLAB Release'''
| style="text-align: center" | '''MCR Version'''
| style="text-align: center" | '''MCR directory'''
|-
| style="text-align: center" | R2017a
| style="text-align: center" | 9.2
| class="WG_Inline_Code" style="text-align: center" | /global/software/matlab/mcr/v92
|-
| style="text-align: center" | R2017b
| style="text-align: center" | 9.3
| class="WG_Inline_Code" style="text-align: center" | /global/software/matlab/mcr/v93
|-
| style="text-align: center" | R2018a
| style="text-align: center" | 9.4
| class="WG_Inline_Code" style="text-align: center" | /global/software/matlab/mcr/v94
|-
| style="text-align: center" | R2019b
| style="text-align: center" | 9.7
| class="WG_Inline_Code" style="text-align: center" | /global/software/matlab/mcr/v97
|-
| style="text-align: center" | R2020a
| style="text-align: center" | 9.8
| class="WG_Inline_Code" style="text-align: center" | /global/software/matlab/mcr/v98
|}

MATLAB

2020-05-27T14:04:37Z

Phillips:

[[Category:MATLAB]]
[[Category:Software]]
[[Category:ARC]]



= Introduction =

[http://www.mathworks.com/ MATLAB] is a general-purpose high-level programming package for numerical work such as linear algebra, signal processing and other calculations involving matrices or vectors of data. Visualization tools are also included for presentation of results. The basic MATLAB package is extended through add-on components including SIMULINK, and the Image Processing, Optimization, Neural Network, Signal Processing, Statistics and Wavelet Toolboxes, among others.

The main purpose of this page is to show how to use MATLAB on the University of Calgary [[ ARC_Cluster_Guide |ARC (Advanced Research Computing) cluster]]. It is presumed that you already have an account an ARC and have read the material on reserving resources and [[Running jobs|running jobs]] with the Slurm job management system.

= Ways to run MATLAB - License considerations =
At the University of Calgary, Information Technologies has purchased a MATLAB Total Academic Headcount license that allows installation and use of MATLAB on central clusters, such as ARC, as well as on personal workstations throughout the University. Potentially thousands of instances of MATLAB can be run simultaneously, each checking out a license from a central license server. An alternative is to compile MATLAB code into a standalone application. When such an application is run, it does not need to contact the server for a license. This allows researchers to run their calculations on compatible hardware, not necessarily at the University of Calgary, such as on [https://docs.computecanada.ca/wiki/Getting_started Compute Canada clusters (external link)].

For information about installing MATLAB on your own computer, see the Information Technologies Knowledge Base [https://ucalgary.service-now.com/it?id=kb_article&sys_id=787d708213fdbec08246f7b2e144b0d4 article on MATLAB].

= Running MATLAB on the ARC cluster =
Although it is possible to run MATLAB interactively, the expectation is that most calculations with MATLAB will be completed by submitting a batch job script to the Slurm job scheduler with the sbatch command.

For many researchers, the main reason for using ARC for MATLAB-based calculations is to be able to run many instances at the same time. It is recommended in such cases that any parallel processing features be removed from the code and each instance of MATLAB be run on a single CPU core. It is also possible to run MATLAB on multiple cores in an attempt to speed up individual instances, but, this is generally results in a less efficient use of the cluster hardware. In the sections that follow, serial and then parallel processing examples are shown.

== Serial MATLAB example==

For the purposes of illustration, suppose the following serial MATLAB code, in a file sawtooth.m, is to be run. If your code does not already have them, add a function statement at the beginning and matching end statement at the end as shown in the example. Other features of this example include calling a function with both numerical and string arguments, incorporating a Slurm environment variable into the MATLAB code and producing graphical output in a non-interactive environment.
<pre>
function sawtooth(nterms,nppcycle,ncycle,pngfilebase)

% MATLAB file example to approximate a sawtooth
% with a truncated Fourier expansion.

% nterms = number of terms in expansion.
% nppcycle = number of points per cycle.
% ncycle = number of complete cycles to plot.
% pngfilebase = base of file name for graph of results.

% 2020-05-14

np=nppcycle*ncycle;

fourbypi=4.0/pi;
y(1:np)=pi/2.0;
x(1:np)=linspace(-pi*ncycle,pi*ncycle,np);

for k=1:nterms
twokm=2*k-1;
y=y-fourbypi*cos(twokm*x)/twokm^2;
end

% Prepare output
% Construct the output file name from the base file name and number of terms
% Also append the Slurm JOBID to keep file names unique from run to run.

job=getenv('SLURM_JOB_ID')
pngfile=strcat(pngfilebase,'_',num2str(nterms),'_',job)
disp(['Writing file: ',pngfile,'.png'])

fig=figure;
plot(x,y);
print(fig,pngfile,'-dpng');

quit
end
</pre>
In preparation to run the sawtooth.m code, create a batch job script, sawtooth.slurm of the form:
<pre>
#!/bin/bash
#SBATCH --time=03:00:00 # Adjust this to match the walltime of your job
#SBATCH --nodes=1 # For serial code, always specify just one node.
#SBATCH --ntasks=1 # For serial code, always specify just one task.
#SBATCH --cpus-per-task=1 # For serial code, always specify just one CPU per task.
#SBATCH --mem=4000m # Adjust to match total memory required, in MB.
#SBATCH --partition=pawson-bf,apophis-bf,razi-bf,single,lattice,parallel,cpu2013,cpu2019

# Sample batch job script for running a MATLAB function with both numerical and string arguments
# 2020-05-14

# Specify the name of the main MATLAB function to be run.
# This would normally be the same as the MATLAB source code file name without a .m suffix).
MAIN="sawtooth"

# Define key parameters for the example calculation.
NTERMS=100
NPPCYCLE=20
NCYCLE=3
PNGFILEBASE="sawtooth"

# Contruct a complete function call to pass to MATLAB
# Note, string arguments should appear to MATLAB enclosed in single quotes
ARGS="($NTERMS,$NPPCYCLE,$NCYCLE,'$PNGFILEBASE')"
MAIN_WITH_ARGS=${MAIN}${ARGS}

echo "Calling MATLAB function: ${MAIN_WITH_ARGS}"

echo "Starting run at $(date)"
echo "Running on compute node $(hostname)"
echo "Running from directory $(pwd)"

# Choose a version of MATLAB by loading a module:
module load matlab/r2019b
echo "Using MATLAB version: $(which matlab)"

# Use -singleCompThread below for serial MATLAB code:
matlab -singleCompThread -batch "${MAIN_WITH_ARGS}" > ${MAIN}_${SLURM_JOB_ID}.out 2>&1

echo "Finished run at $(date)"
</pre>
Note that the above script uses the -batch option on the matlab command line. The MathWorks web page on [https://www.mathworks.com/help/matlab/ref/matlablinux.html running MATLAB on Linux (external link)] starting with Release 2019a of MATLAB, recommends using the -batch option for non-interactive use instead of the similar -r option that is recommended in interactive sessions.

To submit the job to be executed, run:
<pre>
sbatch sawtooth.slurm
</pre>

The job should produce three output files: Slurm script output, MATLAB command output and a PNG file, all tagged with the Slurm Job ID.

== Parallel MATLAB examples==

MATLAB provides several ways of speeding up calculations through parallel processing. These include relying on internal parallelization in which multiple threads are used or by using explicit language features, such as parfor, to start up multiple workers on a compute node. Examples of both approaches are shown below. Using multiple compute nodes for a single MATLAB calculation, which depends on the MATLAB Parallel Server product, is not considered here as there has not been sufficient demand to configure that software on ARC.

=== Thread-based parallel processing ===
First consider an example using multiple cores with MATLAB's built-in thread-based parallelization.

Suppose the following code to calculate eigenvalues of a number of random matrices is in a file eig_thread_test.m . Note the use of the maxNumCompThreads function to control the number of threads (one thread per CPU core). For some years now, MathWorks has marked that function as deprecated, but, it still provides a useful limit to ensure that MATLAB doesn't use more cores than assigned by Slurm.

<pre>
function eig_thread_test(nthreads,matrix_size,nmatrices,results_file)

% Calculate the absolute value of the maximum eigenvalue for each of a number of matrices
% possibly using multiple threads.

% nthreads = number of computational threads to use.
% matrix_size = order of two-dimensional random matrix.
% nmatrices = number of matrices to process.
% results_file = name of file in which to save the maximum eigenvalues

% 2020-05-25

matlab_ncores=feature('numcores')

slurm_ncores_per_task=str2num(getenv('SLURM_CPUS_PER_TASK'))
if(isempty(slurm_ncores_per_task))
slurm_ncores_per_task=1;
disp('SLURM_CPUS_PER_TASK not set')
end

% Set number of computational threads to the minimum of matlab_ncores and slurm_ncores_per_task
% Note Mathworks warns that the maxNumCompThreads function will
% be removed in future versions of MATLAB.

% Use only thread-based parallel processing

intial_matlab_max_ncores = maxNumCompThreads(min([nthreads,slurm_ncores_per_task]));

disp(['Using a maximum of ',num2str(maxNumCompThreads()),' computational threads.'])

tic

for i = 1:nmatrices
e=eig(rand(matrix_size));
eigenvalues(i) = max(abs(e));
end

toc

save(results_file,'eigenvalues','-ascii')

quit
end
</pre>

Here is a job script, eig_thread_test.slurm that can be used to run the eig_thread_test.m code.
The number of threads used for the calculation is controlled by specifying the --cpus-per-task parameter that Slurm uses to control the number of CPU cores assigned to the job.

<pre>
#!/bin/bash

#SBATCH --time=01:00:00 # Adjust this to match the walltime of your job
#SBATCH --nodes=1 # Always specify just one node.
#SBATCH --ntasks=1 # Specify just one task.
#SBATCH --cpus-per-task=8 # The number of threads to use
#SBATCH --mem=4000m # Adjust to match total memory required, in MB.
#SBATCH --partition=pawson-bf,apophis-bf,razi-bf,single,lattice,parallel,cpu2013,cpu2019

# Sample batch job script for running a MATLAB function to test thread-based parallel processing features.
# 2020-05-25

# Specify the name of the main MATLAB function to be run.
# This would normally be the same as the MATLAB source code file name without a .m suffix).
MAIN="eig_thread_test"

# Define key parameters for the example calculation.
NTHREADS=${SLURM_CPUS_PER_TASK}
MATRIX_SIZE=10000
NMATRICES=10
RESULTS_FILE="maximum_eigenvalues_${SLURM_JOB_ID}.txt"

# Contruct a complete function call to pass to MATLAB
# Note, string arguments should appear to MATLAB enclosed in single quotes
ARGS="($NTHREADS,$MATRIX_SIZE,$NMATRICES,'$RESULTS_FILE')"
MAIN_WITH_ARGS=${MAIN}${ARGS}

echo "Calling MATLAB function: ${MAIN_WITH_ARGS}"

echo "Starting run at $(date)"
echo "Running on compute node $(hostname)"
echo "Running from directory $(pwd)"

# Choose a version of MATLAB by loading a module:
module load matlab/r2020a
echo "Using MATLAB version: $(which matlab)"

matlab -batch "${MAIN_WITH_ARGS}" > ${MAIN}_${SLURM_JOB_ID}.out 2>&1

echo "Finished run at $(date)"
</pre>

The above job can be submitted with
<pre>
sbatch eig_thread_test.slurm
</pre>

If assigned to one of the modern partitions (as opposed to the older single, lattice or parallel partitions) the job took about 16 minutes, about 4 times faster than a comparable serial job. Using 8 cores to obtain just a factor of four speed-up is not an efficient use of ARC, but, might be justified in some cases. In some cases, using multiple workers (as discussed in the next section) may be faster than using the same number of cores with thread-based parallelization.

=== Explicit parallel processing using a pool of workers ===

Now consider the more complicated case of creating a pool of workers and using a parfor loop to explicitly parallelize a section of code. Each worker may use one or more cores through MATLAB's internal thread-based parallelization, as in the preceding example. Suppose the following code is in a file eig_parallel_test.m.
<pre>
function eig_parallel_test(nworkers,nthreads,matrix_size,nmatrices,results_file)

% Calculate the absolute value of the maximum eigenvalue for each of a number of matrices
% possibly using multiple threads and multiple MATLAB workers.

% nworkers = number of MATLAB workers to use.
% nthreads = number of threads per worker.
% matrix_size = order of two-dimensional random matrix.
% nmatrices = number of matrices to process.
% results_file = name of file in which to save the maximum eigenvalues

% 2020-05-25

matlab_ncores=feature('numcores')

slurm_ncores_per_task=str2num(getenv('SLURM_CPUS_PER_TASK'))
if(isempty(slurm_ncores_per_task))
slurm_ncores_per_task=1;
disp('SLURM_CPUS_PER_TASK not set')
end

% Set number of computational threads to the minimum of matlab_ncores and slurm_ncores
% Note Mathworks warns that the maxNumCompThreads function will
% be removed in future versions of MATLAB.
% Testing based on remarks at
% https://www.mathworks.com/matlabcentral/answers/158192-maxnumcompthreads-hyperthreading-and-parpool
% shows the maxNumCompThreads has to be called inside the parfor loop.

tic

if ( nworkers > 1 )

% Process with multiple workers
% Check on properties of the local MATLAB cluster.
% One can set properties such as c.NumThreads and c.NumWorkers

parallel.defaultClusterProfile('local')
c = parcluster()
c.NumThreads=nthreads
c.NumWorkers=nworkers

% Create a pool of workers with the current cluster settings.
% Note, testing without the nworkers argument showed a limit of 12 workers even if c.NumWorkers is defined.
parpool(c,nworkers)

ticBytes(gcp);

parfor i = 1:nmatrices
e=eig(rand(matrix_size));
eigenvalues(i) = max(abs(e));
end

tocBytes(gcp)

% Close down the pool.
delete(gcp('nocreate'));

else

% Use only thread-based parallel processing

intial_matlab_max_ncores = maxNumCompThreads(min([nthreads,slurm_ncores_per_task]))

for i = 1:nmatrices
e=eig(rand(matrix_size));
eigenvalues(i) = max(abs(e));
end

end % nworkers test

toc

save(results_file,'eigenvalues','-ascii')

quit
end
</pre>
Of particular note in the preceding example is the section of lines (copied below) that creates a cluster object, c, and modifies the number of threads associated with this object (c.NumThreads=nthreads). In a similar way, one can modify the number of workers (c.NumWorkers=nworkers). Testing showed that if one then used the MATLAB gcp or parpool commands without arguments to create a pool of workers, at most 12 workers were created. However, it was found that by using parpool(c,nworkers), the requested number of workers would be started, even if nworkers > 12.
<pre>
parallel.defaultClusterProfile('local')
c = parcluster()
c.NumThreads=nthreads
c.NumWorkers=nworkers
parpool(c,nworkers)
</pre>

An example Slurm batch job script, eig_parallel_test.slurm, used to test the above code was:
<pre>
#!/bin/bash

#SBATCH --time=03:00:00 # Adjust this to match the walltime of your job
#SBATCH --nodes=1 # Always specify just one node.
#SBATCH --ntasks=1 # Always specify just one task.
#SBATCH --cpus-per-task=8 # Choose --cpus-per-task to match the number of workers * threads per worker
#SBATCH --mem=10000m # Adjust to match total memory required, in MB.
#SBATCH --partition=pawson-bf,apophis-bf,razi-bf,single,lattice,parallel,cpu2013,cpu2019

# Sample batch job script for running a MATLAB function to test parallel processing features
# 2020-05-25

# Specify the name of the main MATLAB function to be run.
# This would normally be the same as the MATLAB source code file name without a .m suffix).
MAIN="eig_parallel_test"

# Define key parameters for the example calculation.
NWORKERS=${SLURM_NTASKS}
NTHREADS=${SLURM_CPUS_PER_TASK}
MATRIX_SIZE=10000
NMATRICES=10
RESULTS_FILE="maximum_eigenvalues_${SLURM_JOB_ID}.txt"

# Contruct a complete function call to pass to MATLAB
# Note, string arguments should appear to MATLAB enclosed in single quotes
ARGS="($NWORKERS,$NTHREADS,$MATRIX_SIZE,$NMATRICES,'$RESULTS_FILE')"
MAIN_WITH_ARGS=${MAIN}${ARGS}

echo "Calling MATLAB function: ${MAIN_WITH_ARGS}"

echo "Starting run at $(date)"
echo "Running on compute node $(hostname)"
echo "Running from directory $(pwd)"

# Choose a version of MATLAB by loading a module:
module load matlab/r2020a
echo "Using MATLAB version: $(which matlab)"

matlab -batch "${MAIN_WITH_ARGS}" > ${MAIN}_${SLURM_JOB_ID}.out 2>&1

echo "Finished run at $(date)"
</pre>
Note that the Slurm SBATCH parameter --cpus-per-task, the total number of cores to use, should be the product of the number of workers and the threads per worker. (It might be argued that one more core should be requested beyond the product of workers and threads, to use for the main MATLAB process, but, for a fully parallelized code, the workers do the great bulk of the calculation and the main MATLAB process uses relatively little CPU time.)

Variations on the above code was tested with many combinations of workers and threads per worker. It was found that if many jobs were started in close succession that some of the jobs failed to start properly.

Also note that in the above example, the number of cores requested is not well matched to the number of workers. If there are 8 workers processing 10 matrices and each worker gets assigned one matrix, there are two left over. The total time for 10 matrices (or for any number of matrices from 9 to 16) would be about double that for 8 matrices. For example, in testing the above code, the time for processing 10 matrices with 8 workers with just one computational thread was 1050 seconds, whereas processing just 8 matrices with 8 workers took only 560 seconds.

= Standalone Applications =

When running MATLAB code as described in the preceding section, a connection to the campus MATLAB license server, checking out licenses for MATLAB and any specialized toolboxes needed, is made for each job that is submitted. Currently, with the University of Calgary's Total Academic Headcount license, there are sufficient license tokens to support thousands of simultaneous MATLAB sessions (although ARC usage policy and cluster load will limit individual users to smaller numbers of jobs). However, there may be times at which the license server is slow to respond when large numbers of requests are being handled, or the server may be unavailable temporarily due to network problems. MathWorks offers an alternative way of running MATLAB code that can avoid license server issues by compiling it into a standalone application. A license is required only during the compilation process and not when the code is run. This allows calculations to be run on ARC without concerns regarding the license server. The compiled code can also be run on compatible (64-bit Linux) hardware, not necessarily at the University of Calgary, such as on [https://docs.computecanada.ca/wiki/Getting_started Compute Canada (external link)] clusters.

== Creating a standalone application ==

The MATLAB '''mcc''' command is used to compile source code (.m files) into a standalone executable. There are a couple of important considerations to keep in mind when creating an executable that can be run in a batch-oriented cluster environment. One is that there is no graphical display attached to your session and another is that the number of threads used by the standalone application has to be controlled. There is also an important difference in the way arguments of the main function are handled.

Let's illustrate the process of creating a standalone application for the sawtooth.m code used previously. Unfortunately, if that code is compiled as it is, the resulting compiled application will fail to run properly. The reason is that the compiled code sees all the input arguments as strings instead of interpreting them as numbers. To work around this problem, use a MATLAB function, '''isdeployed''', to determine whether or not the code is being run as a standalone application. Here is a modified version of code, called sawtooth_standalone.m that can be successfully compiled and run as a standalone application.

<pre>
function sawtooth_standalone(nterms,nppcycle,ncycle,pngfilebase)

% MATLAB file example to approximate a sawtooth
% with a truncated Fourier expansion.

% nterms = number of terms in expansion.
% nppcycle = number of points per cycle.
% ncycle = number of complete cycles to plot.
% pngfilebase = base of file name for graph of results.

% 2020-05-21

% Test to see if the code is running as a standalone application
% If it is, convert the arguments intended to be numeric from
% the input strings to numbers

if isdeployed
nterms=str2num(nterms)
nppcycle=str2num(nppcycle)
ncycle=str2num(ncycle)
end

np=nppcycle*ncycle;

fourbypi=4.0/pi;
y(1:np)=pi/2.0;
x(1:np)=linspace(-pi*ncycle,pi*ncycle,np);

for k=1:nterms
twokm=2*k-1;
y=y-fourbypi*cos(twokm*x)/twokm^2;
end

% Prepare output
% Construct the output file name from the base file name and number of terms
% Also append the Slurm JOBID to keep file names unique from run to run.

job=getenv('SLURM_JOB_ID')
pngfile=strcat(pngfilebase,'_',num2str(nterms),'_',job)
disp(['Writing file: ',pngfile,'.png'])

fig=figure;
plot(x,y);
print(fig,pngfile,'-dpng');

quit
end
</pre>

Suppose that the sawtooth_standalone.m file is in a subdirectory src below your current working directory and that the compiled files are going to be written to a subdirectory called deploy. The following commands (at the Linux shell prompt) could be used to compile the code:

<pre>
mkdir deploy
cd src
module load matlab/r2019b
mcc -R -nodisplay \
-R -singleCompThread \
-m -v -w enable \
-d ../deploy \
sawtooth_standalone.m
</pre>

Note the option -singleCompThread has been included in order to limit the executable to just one computational thread.

In the deploy directory, an executable, sawtooth_standalone, will be created along with a script, run_sawtooth_standalone.sh. These two files should be copied to the target machine where the code is to be run.

== Running a standalone application ==

After the standalone executable sawtooth_standalone and corresponding script run_sawtooth_standalone.sh have been transferred to a directory on the target system on which they will be run (whether to a different directory on ARC or to a completely different cluster), a batch job script needs to be created in that same directory. Here is an example batch job script, sawtooth_standalone.slurm, appropriate for the ARC cluster.

<pre>
#!/bin/bash
#SBATCH --time=03:00:00 # Adjust this to match the walltime of your job
#SBATCH --nodes=1 # For serial code, always specify just one node.
#SBATCH --ntasks=1 # For serial code, always specify just one task.
#SBATCH --cpus-per-task=1 # For serial code, always specify just one CPU per task.
#SBATCH --mem=4000m # Adjust to match total memory required, in MB.
#SBATCH --partition=pawson-bf,apophis-bf,razi-bf,single,lattice,parallel,cpu2013,cpu2019

# Sample batch job script for running a compiled MATLAB function
# 2020-05-21

# Specify the name of the compiled MATLAB standalone executable
MAIN="sawtooth_standalone"

# Define key parameters for the example calculation.
NTERMS=100
NPPCYCLE=20
NCYCLE=3
PNGFILEBASE=$MAIN

ARGS="$NTERMS $NPPCYCLE $NCYCLE $PNGFILEBASE"

# Choose the MCR directory according to the compiler version used
MCR=/global/software/matlab/mcr/v97

echo "Starting run at $(date)"
echo "Running on compute node $(hostname)"
echo "Running from directory $(pwd)"

./run_${MAIN}.sh $MCR $ARGS > ${MAIN}_${SLURM_JOB_ID}.out 2>&1

echo "Finished run at $(date)"
</pre>

The job is then submitted with sbatch:

<pre>
sbatch sawtooth_standalone.slurm
</pre>

An important part of the above script is the definition of the variable '''MCR''', which defines the location of the MATLAB Compiler Runtime (MCR) directory. This directory contains files necessary for the standalone application to run. The version of the MCR files specified (v97 in the example, which corresponds to MATLAB R2019b) must match the version of MATLAB used to compile the code.

A list of MATLAB distributions and the corresponding MCR versions is given on the [https://www.mathworks.com/products/compiler/matlab-runtime.html Mathworks web site (external link)]. Some versions installed on ARC are listed below, along with the corresponding installation directory to which the MCR variable should be set if running on ARC. (As of this writing on May 21, 2020, installation of release R2020a has not quite been finished, but, should be ready before the end of the month). If the MCR version you need does not appear in /global/software/matlab/mcr, write to support@hpc.ucalgary.ca to request that it be installed, or use a different version of MATLAB for your compilation.

{| class="WG_Indent3" style="height: 150px; width: 576px" border="2"
| style="text-align: center" | '''MATLAB Release'''
| style="text-align: center" | '''MCR Version'''
| style="text-align: center" | '''MCR directory'''
|-
| style="text-align: center" | R2017a
| style="text-align: center" | 9.2
| class="WG_Inline_Code" style="text-align: center" | /global/software/matlab/mcr/v92
|-
| style="text-align: center" | R2017b
| style="text-align: center" | 9.3
| class="WG_Inline_Code" style="text-align: center" | /global/software/matlab/mcr/v93
|-
| style="text-align: center" | R2018a
| style="text-align: center" | 9.4
| class="WG_Inline_Code" style="text-align: center" | /global/software/matlab/mcr/v94
|-
| style="text-align: center" | R2019b
| style="text-align: center" | 9.7
| class="WG_Inline_Code" style="text-align: center" | /global/software/matlab/mcr/v97
|-
| style="text-align: center" | R2020a
| style="text-align: center" | 9.8
| class="WG_Inline_Code" style="text-align: center" | /global/software/matlab/mcr/v98
|}