Freesurfer on ARC
Background
- Official site: https://surfer.nmr.mgh.harvard.edu/
Freesurfer is a collection of neuroimaging software that can be used to process Magnetic Resonance Imaging files like DICOMs and NIFTIs. Detailed documentation can be found on the Freesurfer website. Most simple workflows that are scalable on ARC use recon-all to perform a reconstruction of the MRI image from the collection of slices. This is done by parallelizing over scans or subjects.
Freesurfer Apptainer Containers
The more recent versions of Freesurfer installed on ARC have been built as Apptainer containers based off of the official public Freesurfer docker release: [1]
Currently, on ARC there are containers for several versions of Freesurfer available. The containers have integrated license files. These containers cannot be used for commercial or clinical purposes. We strongly encourage you to submit a request for your own license so that you have registered with the makers of Freesurfer: [2]
The full path to the directory with the container files is /global/software/freesurfer/containers.
To see what versions of containerized Freesurfer are available, use
$ ls -l /global/software/freesurfer/containers/ -rwxr-xr-x 1 drozmano drozmano 4358946816 Oct 3 15:32 freesurfer-7.1.1.sif -rwxr-xr-x 1 drozmano drozmano 6402535424 Oct 3 15:31 freesurfer-7.3.2.sif -rwxr-xr-x 1 drozmano drozmano 9342550016 Oct 3 15:31 freesurfer-7.4.1.sif
Running commands from the containers
You can run a command from the container of your choice with the command line like this:
$ apptainer exec <container> <command> [command arguments]
For example, you want to use the container with Freesurfer 7.3.2, freesurfer-7.3.2.sif,
and you want to check the version of the recon-all command.
This is how you can do that:
$ apptainer exec /global/software/freesurfer/containers/freesurfer-7.3.2.sif recon-all --version freesurfer-linux-centos8_x86_64-7.3.2-20220804-6354275
or help:
$ apptainer exec /global/software/freesurfer/containers/freesurfer-7.3.2.sif recon-all --help | less USAGE: recon-all Required Arguments: -subjid <subjid> -<process directive> Fully-Automated Directive: -all : performs all stages of cortical reconstruction -autorecon-all : same as -all Manual-Intervention Workflow Directives: -autorecon1 : process stages 1-5 (see below) -autorecon2 : process stages 6-23 .... ....
Press "q" to exit.
To make it easier to type the long container file name, it can be assigned to an environmental variable, CONTAINER, for example:
$ export CONTAINER=/global/software/freesurfer/containers/freesurfer-7.3.2.sif $ apptainer exec $CONTAINER recon-all --version freesurfer-linux-centos8_x86_64-7.3.2-20220804-6354275
The variable will persist in your session and can be reused for other commands.
Freesurfer Interactive Job
The main command that can be called from an interactive job is singularity shell:
[userrname@arc ~]$ salloc --partition=single --time=2:0:0 --nodes=1 --ntasks=1 --cpus-per-task=8 --mem=16gb
salloc: Pending job allocation 8430460
salloc: job 8430460 queued and waiting for resources
salloc: job 8430460 has been allocated resources
salloc: Granted job allocation 8430460
salloc: Waiting for resource configuration
salloc: Nodes cn056 are ready for job
[username@cn056 ~]$ CONTAINER=/global/software/freesurfer/containers/freesurfer-7.3.2.sif
[username@cn056 ~]$ apptainer shell $CONTAINER
Apptainer> recon-all --version
freesurfer-linux-centos8_x86_64-7.3.2-20220804-6354275
Once you have a prompt inside the container you can run any non-graphical command that is part of freesurfer. We do not have instructions for running freeview yet.
Examples
As MRI reconstruction is memory intensive in general and highly serial for a single scan when using Freesurfer, we encourage users to submit jobs in a non-interactive mode.
This wiki page will include a simple example for running one reconstruction and a more complex example for submitting many reconstructions as a job array.
Single Scan Script
For the simple case, we will use a specification of the time, memory, and CPU requirements and leave off a partition specification.
Here we are assuming that any partition on the default partition list (mostly cpu2019-like hardware as of October 2021) will suffice.
This is a sample recon-all command.
You can replace it with any Freesurfer command that is appropriate to your work.
Job script, single.slurm:
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=16G
#SBATCH --time=24:0:0
CONTAINER=/global/software/freesurfer/containers/freesurfer-7.1.1.sif
#These paths don't need to be in your home dir
subject=subject/specific/dir/name
dicom_file=file_name_to_process.dcm
subjects=~/path/to/parent/subjects/dir
subject=~/path/to/output/dir
apptainer exec -B /scratch,/work,/bulk $CONTAINER recon-all -cw256 \
-subject ${subjects_dir}/${subject} \
-i ${subjects_dir}/${subject}/${dicom_file} \
-all -sd ${output_dir}/
Please note the -B /scratch,/work,/bulk option.
It makes sure that the container also mounts those file systems, in case you want to access / write files outside of your home directory.
The "\" character at the end of the command line allows to continue the command on the next line. It is done for readability, when a command becomes too long.
!!! You have to replace the dummy file paths in this example with actual paths to the directories with your data. !!!
Multiple Scans Reconstruction in Parallel with a Job Array
This script supposes that you have a fixed output directory, a fixed parent directory where the source data lives, a file consisting of a list of child directory names to be processed, and that the same file name should be passed as input for each child directory. The specifics of your research data management scheme will be particular to your project and the corresponding changes that will be required will depend on how you have organized your work. However, this is primarily a matter
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=16G
#SBATCH --time=24:0:0
#SBATCH --array=0-99
CONTAINER=/global/software/freesurfer/containers/freesurfer-7.3.2.sif
((i=SLURM_ARRAY_TASK_ID-1))
###This section collects the paths relevant to a specific task (i.e. processing a particular image and will change from workflow to workflow.
#check existence of subjects directory list file
subjects_file=/path/to/list/of/specific/subject/directories
if [ ! -e "${subjects_file}" ]; then
>&2 echo "error: subjects_file does not exist"
exit 1
fi
# read the subjects from the subjects file
IFS=$'\n'; subjects=( $(cat "${subjects_file}") );
#populate the parent directory for data, the child directory, the file to process, and the output directory
subject=${subjects[${i}]}
dicom_file=file_name_to_process.dcm
subjects=/path/to/parent/subjects/dir
output_dir=/path/to/output/dir
#This is the actual processing command with task specific variables filled in
apptainer exec -B /scratch,/work,/bulk $CONTAINER recon-all -cw256 \
-subject ${subjects_dir}/${subject} \
-i ${subjects_dir}/${subject}/${dicom_file} \
-all -sd ${output_dir}/
!!! You have to replace the dummy file paths in this example with actual paths to the directories with your data. !!!
If you want to know more about using array jobs on ARC you can also check this article: How to use array jobs in SLURM.