Nektar++

General

• Web site: https://www.nektar.info/

• Speed Comparison among Nektar++ Solvers

https://www.nektar.info/speed-comparison-among-nektar-solvers/

Nektar++ is a tensor product based finite element package designed to allow one to construct efficient classical low polynomial order h-type solvers (where h is the size of the finite element) as well as higher p-order piecewise polynomial order solvers.

The framework comes with a number of solvers and also allows one to construct a variety of new solvers.

Nektar++ on ARC

Limitations

Currently, only a container version of Nektar++ is provided on ARC.

Due to the technical incompatibility, Nektar++ 5.3.0 does not seem to be working on the cpu2019 partition when more than 1 node is allocated for the job. It does seem to work on the cpu2021, cpu2022, and cpu2023 partitions.

Available containers

$ ls -l /global/software/nektar/containers 

-rwxr-xr-x 1 drozmano drozmano 412758016 Apr 12 14:36 nektar-5.3.0.sif

Testing

Check the version:

$ apptainer exec /global/software/nektar/containers/nektar-5.3.0.sif IncNavierStokesSolver --version

Nektar++ version 5.3.0

Check the help for a solver:

$ apptainer exec /global/software/nektar/containers/nektar-5.3.0.sif IncNavierStokesSolver --help
Allowed options:
  -v [ --verbose ]            be verbose
  -V [ --version ]            print version information
  -h [ --help ]               print this help message
  -I [ --solverinfo ] arg     override a SOLVERINFO property
  -P [ --parameter ] arg      override a parameter
  --npx arg                   number of procs in X-dir
  --npy arg                   number of procs in Y-dir
  --npz arg                   number of procs in Z-dir
  --nsz arg                   number of slices in Z-dir
  --npt arg                   number of procs in T-dir (parareal)
  --part-only arg             only partition mesh into N partitions.
  --part-only-overlapping arg only partition mesh into N overlapping 
                              partitions.
  --part-info                 Output partition information
  -f [ --forceoutput ]        Disables backups files and forces output to be 
                              written without any checks
  --writeoptfile              write an optimisation file
  --useoptfile arg            use an optimisation file
  -i [ --io-format ] arg      Default input/output format (e.g. Xml, Hdf5)
  --set-start-chknumber arg   Set the starting number of the checkpoint file.
  --set-start-time arg        Set the starting time of the simulation.
  --use-hdf5-node-comm        Use a per-node communicator for HDF5 
                              partitioning.
  --use-ptscotch              Use PtScotch for parallel mesh partitioning.
  --use-scotch                Use Scotch for mesh partitioning.

Example

The test case

A test case is available in /global/software/nektar/tests/test3d.

$ cd
$ cp -r /global/software/nektar/tests/test3d .

$ cd test3d
$ ls -l

-rw-r--r-- 1 username username 44194196 Apr 12 15:01 cylinder_3d_mesh_gmsh_oriented.xml
-rw-r--r-- 1 username username     8617 Apr 12 15:01 cylinder_full3d_conditions.xml
-rw-r--r-- 1 username username      768 Apr 12 15:01 job.slurm

$ sbatch job.slurm
....

The job script

A job script to run on 4 full nodes of the cpu2022 partition, job.slurm:

#! /bin/bash
# -------------------------------------------------------------------------------------------
#SBATCH --job-name=nektar-test

#SBATCH --nodes=4
#SBATCH --ntasks-per-node=52
#SBATCH --cpus-per-task=1
#SBATCH --mem=128gb
#SBATCH --time=0-24:00:00

#SBATCH --paritition=cpu2022
# -------------------------------------------------------------------------------------------
module load openmpi/4.1.1-gnu

CONTAINER="/global/software/nektar/containers/nektar-5.3.0.sif"

OPTS="--verbose --io-format Hdf5 --use-hdf5-node-comm"

mpiexec singularity exec -B /work $CONTAINER IncNavierStokesSolver cylinder_3d_mesh_gmsh_oriented.xml cylinder_full3d_conditions.xml $OPTS

# -------------------------------------------------------------------------------------------