OpenFOAM

Background

OpenFOAM (for "Open-source Field Operation And Manipulation") is a free, open-source, toolkit for creation of computational fluid dynamics (CFD) applications. It includes solver libraries and pre- and post-processing utilities. Common variants of OpenFOAM include those from openfoam.org and openfoam.com .

Typically, researchers will install OpenFOAM on their own computers to learn how to use the software, run simulations that exceed their local hardware capabilities on ARC and then transfer output data back to their own computers for visualization.

There are three main variants of OpenFOAM software that are released as free and open-source software under the GNU General Public License Version 3.

• OpenFOAM Foundation Inc. variant, released by The OpenFOAM Foundation Inc. (since 2012),

and transferred in 2015 to the English company The OpenFOAM Foundation Ltd.

https://openfoam.org/

• OpenFOAM variant by OpenCFD Ltd. (with the name trademarked since 2007) first released as open-source in 2004.

(Note that since 2012, OpenCFD Ltd is an affiliate of ESI Group.)

https://www.openfoam.com/

• FOAM-Extend variant by Wikki Ltd. (since 2009) - a fork of the OpenFOAM® open source library for Computational Fluid Dynamics (CFD).

http://wikki.co.uk/index.php/foam-extend/

http://foam-extend.org

OpenFOAM Apptainer Containers

The more recent versions of OpenFOAM installed on ARC have been built as Apptainer containers based off of the docker containers available on Dockerhub or custom build by ARC's team.

The path to the directory with the container files is /global/software/openfoam/containers. This directory contains three subdirectories, com, extend, and org. These directories are for containerized versions of OpenFOAM provided by OpenCFD Ltd., Wikki Ltd., and OpenFOAM Foundation Inc., respectively.

To see what versions of containerized OpenFOAM are available, simply list the content of the corresponding directory:

$ ls -l /global/software/openfoam/containers/
drwxr-xr-x 2 drozmano drozmano 4096 Jan 20  2023 com
drwxr-xr-x 2 drozmano drozmano 4096 Jan 20  2023 extend
drwxr-xr-x 2 drozmano drozmano 4096 Jan 20  2023 org

$ ls -l /global/software/openfoam/containers/com
-rwxr-xr-x 1 drozmano drozmano 1521639424 Jan 20  2023 openfoam2012.sif
-rwxr-xr-x 1 drozmano drozmano  418091008 Jan 20  2023 openfoam2206.sif

$ ls -l /global/software/openfoam/containers/extend/

$ ls -l /global/software/openfoam/containers/org
-rwxr-xr-x 1 drozmano drozmano 853823488 Jan 20  2023 openfoam10-paraview56.sif
-rwxr-xr-x 1 drozmano drozmano 836984832 Jan 20  2023 openfoam9-paraview56.sif

Currently, there are no version of OpenFOAM-Extend are provided.

Running commands from the containers

You can run a command from the container of your choice with the command line like this:

$ apptainer exec <container> <command> [command arguments]

For example, you want to use the container with OpenFOAM.org v10, openfoam10-paraview56.sif in the org subdirectory, and you want to check the version of the recon-all command. This is how you can do that:

$ apptainer exec /global/software/freesurfer/containers/freesurfer-7.3.2.sif  recon-all --version

or help:

$ apptainer exec /global/software/freesurfer/containers/freesurfer-7.3.2.sif  recon-all --help

To make it easier to type the long container file name, it can be assigned to an environmental variable, CONTAINER, for example:

$ export CONTAINER=/global/software/freesurfer/containers/freesurfer-7.3.2.sif

$ apptainer exec $CONTAINER recon-all --version
freesurfer-linux-centos8_x86_64-7.3.2-20220804-6354275

The variable will persist in your session and can be reused for other commands.

Running OpenFOAM batch jobs on ARC

Researchers using OpenFOAM on ARC are expected to be generally familiar with OpenFOAM capabilities, input file format and the use of restart files.

Like other jobs on ARC, OpenFOAM calculations are run by submitting an appropriate script for batch scheduling using the sbatch command. See documentation on running batch jobs for more information.

Several different versions of OpenFOAM have been installed on ARC under /global/software/openfoam, but, some researchers have chosen to install particular versions in their own home directories or take advantage of the wide range of versions installed on Compute Canada clusters (external link).

Here is a sample script that was used to test OpenFOAM on ARC with one of the supplied tutorial cases (damBreakFine (external link)), which used interFoam, modified to use the scotch decomposition option. The job script and input files can be copied and run on ARC with:

cp -R /global/software/openfoam/examples/damBreak/damBreakVeryFine_scotch_template dambreak
cd dambreak
sbatch dambreak.slurm

The version ("6.x") of OpenFOAM used is from openfoam.org and was built with GNU 4.8.5 compilers and OpenMPI version 2.1.3. OpenFOAM build options used were WM_LABEL_SIZE=64 and FOAMY_HEX_MESH=yes.

#!/bin/bash

#SBATCH --nodes=2
#SBATCH --ntasks-per-node=40    # number of MPI processes per node - adjust according to the partition
#SBATCH --mem=0                 # Use --mem=0 to request all the available memory on a node  
#SBATCH --time=05:00:00         # Maximum run time in hh:mm:ss, or d-hh:mm
#SBATCH --partition=pawson-bf,apophis-bf,razi-bf,cpu2019

# Check on some basics:

echo "Running on host: $(hostname)"
echo "Current working directory is: $(pwd)"
echo "Starting job at $(date)"

# Initialize OpenFOAM environment.
module load openmpi/2.1.3-gnu
export OMPI_MCA_mpi_cuda_support=0

source /global/software/openfoam/6x_20181025_gcc485_mpi_213gnu/OpenFOAM-6/etc/bashrc FOAMY_HEX_MESH=yes

export FOAM_RUN=$PWD

echo "Working in $PWD"

CORES=$SLURM_NTASKS
echo "Running on $CORES cores."

echo "Make a new decomposeParDict file"
DATE=$(date)

cat > system/decomposeParDict <<EOF
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    location    "system";
    object      decomposeParDict;
}

// decomposeParDict created at ${DATE}.

numberOfSubdomains $CORES;

method          scotch;

EOF

echo "Forcing new decomposition"

decomposePar -force

echo "Using mpiexec: $(which mpiexec)"

FOAM=$(which interFoam)
echo "About to run $FOAM at $(date)"

mpiexec $FOAM -parallel > dambreakveryfine_scotch_arc_${CORES}cores_${SLURM_JOB_ID}.out

echo "Finished at $(date)"

echo "Running reconstructPar at $(date)."
reconstructPar -newTimes
echo "Finished reconstructPar at $(date)."
echo "Manually delete processor directories if reconstruction succeeded. "

OpenFOAM can produce large numbers of files per run when many processors (CPU cores) are used. The reconstructPar command should be used to consolidate the per-processor files into a single directory per time step. As noted in a comment in the above script, after you have verified that the reconstruction has succeeded you shoulld delete the processor directories.

Support

Please send any questions regarding using OpenFOAM on ARC to support@hpc.ucalgary.ca.