Connecting to RCS HPC Systems

Background

OpenFOAM (for "Open-source Field Operation And Manipulation") is a free, open-source, toolkit for creation of computational fluid dynamics (CFD) applications. It includes solver libraries and pre- and post-processing utilities. Common variants of OpenFOAM include those from openfoam.org and openfoam.com .

Typically, researchers will install OpenFOAM on their own computers to learn how to use the software, run simulations that exceed their local hardware capabilities on ARC and then transfer output data back to their own computers for visualization.

There are three main variants of OpenFOAM software that are released as free and open-source software under the GNU General Public License Version 3.

• OpenFOAM Foundation Inc. variant, released by The OpenFOAM Foundation Inc. (since 2012),

and transferred in 2015 to the English company The OpenFOAM Foundation Ltd.

https://openfoam.org/

• OpenFOAM variant by OpenCFD Ltd. (with the name trademarked since 2007) first released as open-source in 2004.

(Note that since 2012, OpenCFD Ltd is an affiliate of ESI Group.)

https://www.openfoam.com/

• FOAM-Extend variant by Wikki Ltd. (since 2009) - a fork of the OpenFOAM® open source library for Computational Fluid Dynamics (CFD).

http://wikki.co.uk/index.php/foam-extend/

http://foam-extend.org

OpenFOAM Apptainer Containers

The more recent versions of OpenFOAM installed on ARC have been built as Apptainer containers based off of the docker containers available on Dockerhub or custom build by ARC's team.

The path to the directory with the container files is /global/software/openfoam/containers. This directory contains three subdirectories, com, extend, and org. These directories are for containerized versions of OpenFOAM provided by OpenCFD Ltd., Wikki Ltd., and OpenFOAM Foundation Inc., respectively.

To see what versions of containerized OpenFOAM are available, simply list the content of the corresponding directory:

$ ls -l /global/software/openfoam/containers/
drwxr-xr-x 2 drozmano drozmano 4096 Jan 20  2023 com
drwxr-xr-x 2 drozmano drozmano 4096 Jan 20  2023 extend
drwxr-xr-x 2 drozmano drozmano 4096 Jan 20  2023 org

$ ls -l /global/software/openfoam/containers/com
-rwxr-xr-x 1 drozmano drozmano 1521639424 Jan 20  2023 openfoam2012.sif
-rwxr-xr-x 1 drozmano drozmano  418091008 Jan 20  2023 openfoam2206.sif

$ ls -l /global/software/openfoam/containers/extend/

$ ls -l /global/software/openfoam/containers/org
-rwxr-xr-x 1 drozmano drozmano 853811200 Oct 16 15:53 openfoam10-paraview56-arc.sif
-rwxr-xr-x 1 drozmano drozmano 894136320 Oct 16 15:53 openfoam11-paraview510-arc.sif
-rwxr-xr-x 1 drozmano drozmano 814657536 Oct 16 15:53 openfoam9-paraview56-arc.sif

Currently, there are no version of OpenFOAM-Extend are provided.

The OpenFOAM.org containers were slightly modified, to simplify their use. This is indicated by the -arc suffix in their filenames. They are based on the official containers from DockerHub.

Running commands from the containers

You can run a command from the container of your choice with the command line like this:

$ apptainer exec <container> <command> [command arguments]

OpenFOAM.org

For example, you want to use the container with OpenFOAM.org v10, openfoam10-paraview56.sif, in the org subdirectory, and you want to see the help page for thebuoyantReactingFoam command. This is how you can do that:

$ $ apptainer exec /global/software/openfoam/containers/org/openfoam10-paraview56-arc.sif buoyantReactingFoam -help

Usage: buoyantReactingFoam [OPTIONS]
options:
  -case <dir>       specify alternate case directory, default is the cwd
....
....

To make it easier to type the long container file name, it can be assigned to an environmental variable, CONTAINER, for example:

$ export CONTAINER=/global/software/openfoam/containers/org/openfoam10-paraview56-arc.sif

$ apptainer exec $CONTAINER buoyantReactingFoam -help
....
....

The variable will persist in your session and can be reused for other commands.

Natively Installed OpenFOAM

Activating a natively installed OpenFOAM

Due to technical specifics of the software, on ARC, no modules provided for OpenFOAM. Instead, one has to source the activation script from OpenFOAM installation, <OFDIR>/etc/bashsrc.

To see available installed version of OpenFOAM.org:

$ ls -ld /global/software/openfoam/OpenFOAM*

drwxr-xr-x 11 drozmano drozmano 4096 Feb 22 15:20 /global/software/openfoam/OpenFOAM-10
drwxr-xr-x 11 drozmano drozmano 4096 Feb 22 16:16 /global/software/openfoam/OpenFOAM-11
drwxr-xr-x 11 drozmano drozmano 4096 Feb 22 14:25 /global/software/openfoam/OpenFOAM-9
drwxr-xr-x 11 drozmano drozmano 4096 Feb 23 16:10 /global/software/openfoam/OpenFOAM-8

To activate the version of choice in your current terminal session (or in your job script):

# Activate
$ module load openmpi/4.1.1-gnu
$ source /global/software/openfoam/OpenFOAM-10/etc/bashrc

# Test if it works.
$ foamVersion 
OpenFOAM-10

$

You have to source a special activation file from the installed OpenFOAM. The path to the activation files is <OpenFOAM_PATH>/etc/bashrc, this is the path that has to be sourced.

The activation is for the current bash session. If you start a new bash session, to use OpenFOAM you have to activate it again. If you want to deactivate your OpenFOAM environment, you have to close the current bash session and start an new one.

Running OpenFOAM batch jobs on ARC

Researchers using OpenFOAM on ARC are expected to be generally familiar with OpenFOAM capabilities, input file format and the use of restart files.

Like other jobs on ARC, OpenFOAM calculations are run by submitting an appropriate script for batch scheduling using the sbatch command. See documentation on running batch jobs for more information.

Several different versions of OpenFOAM have been installed on ARC under /global/software/openfoam, but, some researchers have chosen to install particular versions in their own home directories or take advantage of the wide range of versions installed on Compute Canada clusters (external link).

Here is a sample script that was used to test OpenFOAM on ARC with one of the supplied tutorial cases (damBreakFine (external link)), which used interFoam, modified to use the scotch decomposition option. The job script and input files can be copied and run on ARC with:

cp -R /global/software/openfoam/examples/damBreak/damBreakVeryFine_scotch_template dambreak
cd dambreak
sbatch dambreak.slurm

The version ("6.x") of OpenFOAM used is from openfoam.org and was built with GNU 4.8.5 compilers and OpenMPI version 2.1.3. OpenFOAM build options used were WM_LABEL_SIZE=64 and FOAMY_HEX_MESH=yes.

#!/bin/bash

#SBATCH --nodes=2
#SBATCH --ntasks-per-node=40    # number of MPI processes per node - adjust according to the partition
#SBATCH --mem=0                 # Use --mem=0 to request all the available memory on a node  
#SBATCH --time=05:00:00         # Maximum run time in hh:mm:ss, or d-hh:mm
#SBATCH --partition=pawson-bf,apophis-bf,razi-bf,cpu2019

# Check on some basics:

echo "Running on host: $(hostname)"
echo "Current working directory is: $(pwd)"
echo "Starting job at $(date)"

# Initialize OpenFOAM environment.
module load openmpi/2.1.3-gnu
export OMPI_MCA_mpi_cuda_support=0

source /global/software/openfoam/6x_20181025_gcc485_mpi_213gnu/OpenFOAM-6/etc/bashrc FOAMY_HEX_MESH=yes

export FOAM_RUN=$PWD

echo "Working in $PWD"

CORES=$SLURM_NTASKS
echo "Running on $CORES cores."

echo "Make a new decomposeParDict file"
DATE=$(date)

cat > system/decomposeParDict <<EOF
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    location    "system";
    object      decomposeParDict;
}

// decomposeParDict created at ${DATE}.

numberOfSubdomains $CORES;

method          scotch;

EOF

echo "Forcing new decomposition"

decomposePar -force

echo "Using mpiexec: $(which mpiexec)"

FOAM=$(which interFoam)
echo "About to run $FOAM at $(date)"

mpiexec $FOAM -parallel > dambreakveryfine_scotch_arc_${CORES}cores_${SLURM_JOB_ID}.out

echo "Finished at $(date)"

echo "Running reconstructPar at $(date)."
reconstructPar -newTimes
echo "Finished reconstructPar at $(date)."
echo "Manually delete processor directories if reconstruction succeeded. "

OpenFOAM can produce large numbers of files per run when many processors (CPU cores) are used. The reconstructPar command should be used to consolidate the per-processor files into a single directory per time step. As noted in a comment in the above script, after you have verified that the reconstruction has succeeded you shoulld delete the processor directories.

Support

Please send any questions regarding using OpenFOAM on ARC to support@hpc.ucalgary.ca.

Links

ARC Software pages