Nektar++
General
- Web site: https://www.nektar.info/
- Speed Comparison among Nektar++ Solvers
Nektar++ is a tensor product based finite element package designed to allow one to construct efficient classical low polynomial order h-type solvers
(where h is the size of the finite element) as well as higher p-order piecewise polynomial order solvers.
The framework comes with a number of solvers and also allows one to construct a variety of new solvers.
Nektar++ on ARC
Limitations
Currently, only a container version of Nektar++ is provided on ARC.
Due to the technical incompatibility, Nektar++ 5.3.0 does not seem to be working on the cpu2019 partition when more than 1 node is allocated for the job.
It does seem to work on the cpu2021, cpu2022, and cpu2023 partitions.
Available containers
$ ls -l /global/software/nektar/containers -rwxr-xr-x 1 drozmano drozmano 412758016 Apr 12 14:36 nektar-5.3.0.sif
Testing
Check the version:
$ apptainer exec /global/software/nektar/containers/nektar-5.3.0.sif IncNavierStokesSolver --version Nektar++ version 5.3.0
Check the help for a solver:
$ apptainer exec /global/software/nektar/containers/nektar-5.3.0.sif IncNavierStokesSolver --help
Allowed options:
-v [ --verbose ] be verbose
-V [ --version ] print version information
-h [ --help ] print this help message
-I [ --solverinfo ] arg override a SOLVERINFO property
-P [ --parameter ] arg override a parameter
--npx arg number of procs in X-dir
--npy arg number of procs in Y-dir
--npz arg number of procs in Z-dir
--nsz arg number of slices in Z-dir
--npt arg number of procs in T-dir (parareal)
--part-only arg only partition mesh into N partitions.
--part-only-overlapping arg only partition mesh into N overlapping
partitions.
--part-info Output partition information
-f [ --forceoutput ] Disables backups files and forces output to be
written without any checks
--writeoptfile write an optimisation file
--useoptfile arg use an optimisation file
-i [ --io-format ] arg Default input/output format (e.g. Xml, Hdf5)
--set-start-chknumber arg Set the starting number of the checkpoint file.
--set-start-time arg Set the starting time of the simulation.
--use-hdf5-node-comm Use a per-node communicator for HDF5
partitioning.
--use-ptscotch Use PtScotch for parallel mesh partitioning.
--use-scotch Use Scotch for mesh partitioning.
Example
The test case
A test case is available in /global/software/nektar/tests/test3d.
$ cd $ cp -r /global/software/nektar/tests/test3d . $ cd test3d $ ls -l -rw-r--r-- 1 username username 44194196 Apr 12 15:01 cylinder_3d_mesh_gmsh_oriented.xml -rw-r--r-- 1 username username 8617 Apr 12 15:01 cylinder_full3d_conditions.xml -rw-r--r-- 1 username username 768 Apr 12 15:01 job.slurm $ sbatch job.slurm ....
The job script
A job script to run on 4 full nodes of the cpu2022 partition,
job.slurm:
#! /bin/bash
# -------------------------------------------------------------------------------------------
#SBATCH --job-name=nektar-test
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=52
#SBATCH --cpus-per-task=1
#SBATCH --mem=128gb
#SBATCH --time=0-24:00:00
#SBATCH --paritition=cpu2022
# -------------------------------------------------------------------------------------------
module load openmpi/4.1.1-gnu
CONTAINER="/global/software/nektar/containers/nektar-5.3.0.sif"
OPTS="--verbose --io-format Hdf5 --use-hdf5-node-comm"
mpiexec singularity exec -B /work $CONTAINER IncNavierStokesSolver cylinder_3d_mesh_gmsh_oriented.xml cylinder_full3d_conditions.xml $OPTS
# -------------------------------------------------------------------------------------------