Bioinformatics applications - Revision history

Lleung: Added categories and navbox

2023-09-20T22:34:06Z

Added categories and navbox

← Older revision		Revision as of 22:34, 20 September 2023
Line 141:		Line 141:
	Contact: Heng Li <lh3@sanger.ac.uk>		Contact: Heng Li <lh3@sanger.ac.uk>
	Usage: bwa <command> [options]		Usage: bwa <command> [options]

			[[Category:Guides]]
			[[Category:How-Tos]]
			{{Navbox Guides}}

Tannistha.nandi: /* Strategy to speed up the fastqc runtime */

2021-06-11T21:05:57Z

Strategy to speed up the fastqc runtime

← Older revision		Revision as of 21:05, 11 June 2021
Line 96:		Line 96:

	The core and memory allocation need be adjusted to run fastqc in a multi-threaded mode. The compute resource request will be based on the following factors:		The core and memory allocation need be adjusted to run fastqc in a multi-threaded mode. The compute resource request will be based on the following factors:
	# Number of ~~fastq~~ files		# Number of input files
	# Number of cores		# Number of cores
	# Memory /RAM		# Memory /RAM

Tannistha.nandi: /* Strategy to speed up the fastqc runtime */

2021-06-11T21:04:29Z

Strategy to speed up the fastqc runtime

← Older revision		Revision as of 21:04, 11 June 2021
Line 69:		Line 69:
	\| CPU efficiency \|\| 99.28%		\| CPU efficiency \|\| 99.28%
	\|-		\|-
	\| Run time \|\| ~~00:~~46~~:13~~		\| Run time \|\| 46 mins
	\|-		\|-
	\| Memory utilized\|\| ~~285.46 MB~~		\| Memory utilized\|\| 286MB
	\|}		\|}

	==== Strategy to speed up the fastqc runtime ====		==== Strategy to speed up the fastqc runtime ====
	We will use the fastqc runtime flag --threads or -t in short. The flag -t specifies the number of files which can be processed simultaneously. Generally, a single thread will run on 1 core, two threads will run on 2 cores and so on.		We will use the fastqc runtime flag --threads or -t in short. The flag -t specifies the number of files which can be processed simultaneously. Generally, a single thread will run on 1 core, two threads will run on 2 cores and so on.
	‘Fastqc’ can accept multiple input files. One strategy would be to split up the ~~33GB~~ data file into smaller chunks. Here we have used a python-based tool called ‘fastqsplitter 1.2.0’ (https://pypi.org/project/fastqsplitter/) to split up the fastq file into 10 chunks.		‘Fastqc’ can accept multiple input files. One strategy would be to split up the 35GB data file into smaller chunks. Here we have used a python-based tool called ‘fastqsplitter 1.2.0’ (https://pypi.org/project/fastqsplitter/) to split up the fastq file into 10 chunks.

	[tannistha.nandi@arc ~]$ salloc –mem=6G -t 01:00:00 -c 10 -N 1 -n 1 -p apophis-bf		[tannistha.nandi@arc ~]$ salloc –mem=6G -t 01:00:00 -c 10 -N 1 -n 1 -p apophis-bf

Tannistha.nandi: /* 1. FastQC */

2021-06-11T20:24:28Z

1. FastQC

← Older revision		Revision as of 20:24, 11 June 2021
Line 17:		Line 17:
	------------------------------------------------------------------------------------------------------------------------------------		------------------------------------------------------------------------------------------------------------------------------------

	=== 1. FastQC ===		=== FastQC ===

	FastQC assess the quality of your sequencing data. It is available as a module on the ARC cluster. To run the installed version of fastqc, load the biobuilds/2017.11 module as below:		FastQC assess the quality of your sequencing data. It is available as a module on the ARC cluster. To run the installed version of fastqc, load the biobuilds/2017.11 module as below:
Line 129:		Line 129:
	------------------------------------------------------------------------------------------------------------------------------------		------------------------------------------------------------------------------------------------------------------------------------

	=== 2. BWA (Burrows-Wheeler Aligner)===		=== BWA (Burrows-Wheeler Aligner)===

	BWA is a popular software for mapping sequencing reads to a reference genome. It is available as a module on the ARC cluster. To run the installed version of bwa, load the biobuilds/2017.11 module as below:		BWA is a popular software for mapping sequencing reads to a reference genome. It is available as a module on the ARC cluster. To run the installed version of bwa, load the biobuilds/2017.11 module as below:

Tannistha.nandi: /* Strategy to speed up the fastqc runtime */

2021-06-11T20:24:01Z

Strategy to speed up the fastqc runtime

← Older revision		Revision as of 20:24, 11 June 2021
Line 74:		Line 74:
	\|}		\|}

	=== Strategy to speed up the fastqc runtime ===		==== Strategy to speed up the fastqc runtime ====
	We will use the fastqc runtime flag --threads or -t in short. The flag -t specifies the number of files which can be processed simultaneously. Generally, a single thread will run on 1 core, two threads will run on 2 cores and so on.		We will use the fastqc runtime flag --threads or -t in short. The flag -t specifies the number of files which can be processed simultaneously. Generally, a single thread will run on 1 core, two threads will run on 2 cores and so on.
	‘Fastqc’ can accept multiple input files. One strategy would be to split up the 33GB data file into smaller chunks. Here we have used a python-based tool called ‘fastqsplitter 1.2.0’ (https://pypi.org/project/fastqsplitter/) to split up the fastq file into 10 chunks.		‘Fastqc’ can accept multiple input files. One strategy would be to split up the 33GB data file into smaller chunks. Here we have used a python-based tool called ‘fastqsplitter 1.2.0’ (https://pypi.org/project/fastqsplitter/) to split up the fastq file into 10 chunks.

Tannistha.nandi: /* Performance evaluation */

2021-06-08T22:14:14Z

Performance evaluation

← Older revision		Revision as of 22:14, 8 June 2021
Line 124:		Line 124:
	It is one of the recommended approaches to reduce the runtime for a given input size. In this example, the multi-threaded feature improved the run time by 10X over a single threaded job. Another point to take note of is that the 4X increased memory utilization of the multi-threaded process. The RAM usage trend is based on the input file size and the way memory allocation is handled by the application.		It is one of the recommended approaches to reduce the runtime for a given input size. In this example, the multi-threaded feature improved the run time by 10X over a single threaded job. Another point to take note of is that the 4X increased memory utilization of the multi-threaded process. The RAM usage trend is based on the input file size and the way memory allocation is handled by the application.

	[[File:Fastqc.png\|center]]		[[File:Fastqc performance.png\|center]]

	------------------------------------------------------------------------------------------------------------------------------------		------------------------------------------------------------------------------------------------------------------------------------

Tannistha.nandi: /* 1. FastQC */

2021-06-08T22:11:37Z

1. FastQC

← Older revision		Revision as of 22:11, 8 June 2021
Line 63:		Line 63:
	\| # files \|\| 1		\| # files \|\| 1
	\|-		\|-
	\| File size \|\| ~~33GB~~		\| File size \|\| 35GB
	\|-		\|-
	\| # cores \|\| 1		\| # cores \|\| 1
Line 119:		Line 119:
	Submitted batch job 9658870		Submitted batch job 9658870

	The runtime of multi-threaded fastqc job (job id 9658870) on the same input data size (~~33G~~) is ~5 minutes. Thus, the run time of fastqc analysis on the same input size can be reduced significantly using its multi-threaded functionality.		The runtime of multi-threaded fastqc job (job id 9658870) on the same input data size (35G) is ~5 minutes. Thus, the run time of fastqc analysis on the same input size can be reduced significantly using its multi-threaded functionality.

	==== Performance evaluation ====		==== Performance evaluation ====

Tannistha.nandi: /* 1. FastQC */

2021-06-08T22:09:01Z

1. FastQC

← Older revision		Revision as of 22:09, 8 June 2021
Line 31:		Line 31:
	fastqc [-o output dir] [--(no)extract] [-f fastq\|bam\|sam] [-c contaminant file] seqfile1..seqfileN		fastqc [-o output dir] [--(no)extract] [-f fastq\|bam\|sam] [-c contaminant file] seqfile1..seqfileN

	In this example, we will work with the following <big>'''~~33GB~~'''</big> of genome sequencing data in fastq file format.		In this example, we will work with the following <big>'''35GB'''</big> of genome sequencing data in fastq file format.
	[tannistha.nandi@arc ~]$ ls -l		[tannistha.nandi@arc ~]$ ls -l
	-rw-rw-r-- 1 tannistha.nandi tannistha.nandi ~~33GB~~ Jan 24. 2018 SN4570284.fq.gz		-rw-rw-r-- 1 tannistha.nandi tannistha.nandi 35GB Jan 24. 2018 SN4570284.fq.gz

	We will submit the fastqc job on a backfill partition called apophis-bf where each compute node has 40 cores with 185GB of RAM. The job script below requests for <big>'''1'''</big> core (--cpus-per-task) for <big>'''1'''</big> hour, along with <big>'''300 MB'''</big> of RAM on the apophis-bf partition. The RAM estimate of 300MB is based on the fastqc user guide that states each thread will be allocated 250MB of memory. On the ARC cluster, the CPUs and cores refer to the same thing. When the job scheduler SLURM will allocate these resources on a compute node, the job will run the first command to load the biobuilds/2017.11 module, which will set the path to fastqc. The next line will launch and run the fastqc application on the data file.		We will submit the fastqc job on a backfill partition called apophis-bf where each compute node has 40 cores with 185GB of RAM. The job script below requests for <big>'''1'''</big> core (--cpus-per-task) for <big>'''1'''</big> hour, along with <big>'''300 MB'''</big> of RAM on the apophis-bf partition. The RAM estimate of 300MB is based on the fastqc user guide that states each thread will be allocated 250MB of memory. On the ARC cluster, the CPUs and cores refer to the same thing. When the job scheduler SLURM will allocate these resources on a compute node, the job will run the first command to load the biobuilds/2017.11 module, which will set the path to fastqc. The next line will launch and run the fastqc application on the data file.

Tannistha.nandi: /* Strategy to speed up the fastqc runtime */

2021-06-08T21:49:24Z

Strategy to speed up the fastqc runtime

← Older revision		Revision as of 21:49, 8 June 2021
Line 115:		Line 115:
	#<------------------------Run python script--------------------------->		#<------------------------Run python script--------------------------->
	fastqc '''--threads 10''' -o output_dir --noextract -f fastq SN4570284_S0.fq.gz SN4570284_S1.fq.gz SN4570284_1_S2.fq.gz SN4570284_S3.fq.gz SN4570284_S4.fq.gz SN4570284_S5.fq.gz SN4570284_1_S6.fq.gz SN4570284_S7.fq.gz SN4570284_S8.fq.gz SN4570284_S9.fq.gz		fastqc '''--threads 10''' -o output_dir --noextract -f fastq SN4570284_S0.fq.gz SN4570284_S1.fq.gz SN4570284_1_S2.fq.gz SN4570284_S3.fq.gz SN4570284_S4.fq.gz SN4570284_S5.fq.gz SN4570284_1_S6.fq.gz SN4570284_S7.fq.gz SN4570284_S8.fq.gz SN4570284_S9.fq.gz
			Save the above script as 'fastqc-multicore.sh' and submit it to SLURM using a sbatch command as below:

	[tannistha.nandi@arc ~]$ sbatch fastqc-multicore.sh		[tannistha.nandi@arc ~]$ sbatch fastqc-multicore.sh
	Submitted batch job 9658870		Submitted batch job 9658870

Tannistha.nandi: /* Strategy to speed up the fastqc runtime */

2021-06-08T21:47:59Z

Strategy to speed up the fastqc runtime

← Older revision		Revision as of 21:47, 8 June 2021
Line 75:		Line 75:

	=== Strategy to speed up the fastqc runtime ===		=== Strategy to speed up the fastqc runtime ===
	I will use the fastqc runtime flag --threads or -t in short. The flag -t specifies the number of files which can be processed simultaneously. Generally, a single thread will run on 1 core, two threads will run on 2 cores and so on.		We will use the fastqc runtime flag --threads or -t in short. The flag -t specifies the number of files which can be processed simultaneously. Generally, a single thread will run on 1 core, two threads will run on 2 cores and so on.
	‘Fastqc’ can accept multiple input files. One strategy would be to split up the 33GB data file into smaller chunks. Here I have used a python-based tool called ‘fastqsplitter 1.2.0’ (https://pypi.org/project/fastqsplitter/) to split up the fastq file into 10 chunks.		‘Fastqc’ can accept multiple input files. One strategy would be to split up the 33GB data file into smaller chunks. Here we have used a python-based tool called ‘fastqsplitter 1.2.0’ (https://pypi.org/project/fastqsplitter/) to split up the fastq file into 10 chunks.

	[tannistha.nandi@arc ~]$ salloc –mem=6G -t 01:00:00 -c 10 -N 1 -n 1 -p apophis-bf		[tannistha.nandi@arc ~]$ salloc –mem=6G -t 01:00:00 -c 10 -N 1 -n 1 -p apophis-bf
Line 101:		Line 101:
	# Input file size		# Input file size

	There are 10 fastq files, ~3GB each. A total of 10 cores can be requested, one fastq file per core. Accordingly, we need to allocate memory for all 10 cores. A single core fastqc job on a file size of 3GB used ~187 MB of RAM (based on a separate job called 'fastqc-S1'). Thus, I scaled up the memory request for 10 cores to a total of 2000MB of RAM. The multi-threaded job script is saved to a file called ‘fastqc-multicore.sh’.		There are 10 fastq files, ~3GB each. A total of 10 cores can be requested, one fastq file per core. Accordingly, we need to allocate memory for all 10 cores. A single core fastqc job on a file size of 3GB used ~187 MB of RAM (based on a separate job called 'fastqc-S1'). Thus, we scaled up the memory request for 10 cores to a total of 2000MB of RAM. The multi-threaded job script is saved to a file called ‘fastqc-multicore.sh’.
	#!/bin/bash		#!/bin/bash
	#<------------------------Request for Resources----------------------->		#<------------------------Request for Resources----------------------->