Softwares used in the prealignment module

Fastp

Trim .fastq files by removing adapter sequences and other unwanted sequences. Adapter sequences are specified in units.tsv under the adapter column.

Rule

rule fastp_pe:
    input:
        sample=lambda wildcards: [
            get_fastq_file(units, wildcards, "fastq1"),
            get_fastq_file(units, wildcards, "fastq2"),
        ],
    output:
        trimmed=temp(
            [
                "prealignment/fastp_pe/{sample}_{type}_{flowcell}_{lane}_{barcode}_fastq1.fastq.gz",
                "prealignment/fastp_pe/{sample}_{type}_{flowcell}_{lane}_{barcode}_fastq2.fastq.gz",
            ]
        ),
        html="prealignment/fastp_pe/{sample}_{type}_{flowcell}_{lane}_{barcode}_fastp.html",
        json="prealignment/fastp_pe/{sample}_{type}_{flowcell}_{lane}_{barcode}_fastp.json",
    params:
        adapters=lambda wildcards: " --adapter_sequence {} --adapter_sequence_r2 {} ".format(
            *get_fastq_adapter(units, wildcards).split(",")
        ),
        extra=config.get("fastp_pe", {}).get("extra", ""),
    log:
        "prealignment/fastp_pe/{sample}_{type}_{flowcell}_{lane}_{barcode}_fastq.fastq.gz.log",
    benchmark:
        repeat(
            "prealignment/fastp_pe/{sample}_{type}_{flowcell}_{lane}_{barcode}_fastq.fastq.gz.benchmark.tsv",
            config.get("fastp_pe", {}).get("benchmark_repeats", 1),
        )
    resources:
        mem_mb=config.get("fastp_pe", {}).get("mem_mb", config["default_resources"]["mem_mb"]),
        mem_per_cpu=config.get("fastp_pe", {}).get("mem_per_cpu", config["default_resources"]["mem_per_cpu"]),
        partition=config.get("fastp_pe", {}).get("partition", config["default_resources"]["partition"]),
        threads=config.get("fastp_pe", {}).get("threads", config["default_resources"]["threads"]),
        time=config.get("fastp_pe", {}).get("time", config["default_resources"]["time"]),
    threads: config.get("fastp_pe", {}).get("threads", config["default_resources"]["threads"])
    container:
        config.get("fastp_pe", {}).get("container", config["default_container"])
    message:
        "{rule}: trim fastq files {input.sample} using fastp,\n\t\t with adapters: {params.adapters}"
    wrapper:
        "0.78.0/bio/fastp"

input / output files

Rule parameters	Key	Value	Description
input	sample	Fastq files specified in `units.tsv` obtained by get_fastq_file defined in the hydra-genetics module	Untrimmed `.fastq files` from the same sample
output	trimmed	`"prealignment/fastp_pe/{sample}_{type}_{flowcell}_{lane}_{barcode}_fastq1.fastq.gz"`	Trimmed `.fastq files` (read1 and read2) from the same sample
	trimmed	`"prealignment/fastp_pe/{sample}_{type}_{flowcell}_{lane}_{barcode}_fastq2.fastq.gz"`
	html	`"prealignment/fastp_pe/{sample}_{type}_{flowcell}_{lane}_{barcode}_fastp.html"`	html QC report
	json	`"prealignment/fastp_pe/{sample}_{type}_{flowcell}_{lane}_{barcode}_fastp.json"`	json QC report

Configuration

Software settings (`config.yaml`)

Key	Type	Description
benchmark_repeats	integer	set number of times benchmark should be repeated
container	string	name or path to docker/singularity container
extra	string	parameters that should be forwarded NOTE: Fastp has these options active as default: --trim_poly_g --qualified_quality_phred 15 --unqualified_percent_limit 40 --n_base_limit 5 --length_required 15

Resources settings (`resources.yaml`)

Key	Type	Description
mem_mb	integer	max memory in MB to be available
mem_per_cpu	integer	memory in MB used per cpu
partition	string	partition to use on cluster
threads	integer	number of threads to be available RECOMMENDATION: Use multiple threads for decreased run time. NOTE: If multiple threads is used it is also best to increase memory (mem_mb) NOTE: If multiple threads is used the read order of the fastq files will differ between runs
time	string	max execution time

Fastq merging

Merge .fastq files generated for example on different lanes by simply concatenating them using cat

Rule

rule merged:
    input:
        fastq=merged_input,
    output:
        fastq=temp("prealignment/merged/{sample}_{type}_{read}.fastq.gz"),
    log:
        "prealignment/merged/{sample}_{type}_{read}.fastq.gz.log",
    benchmark:
        repeat(
            "prealignment/merged/{sample}_{type}_{read}.fastq.gz.benchmark.tsv",
            config.get("merged", {}).get("benchmark_repeats", 1),
        )
    resources:
        mem_mb=config.get("merged", {}).get("mem_mb", config["default_resources"]["mem_mb"]),
        mem_per_cpu=config.get("merged", {}).get("mem_per_cpu", config["default_resources"]["mem_per_cpu"]),
        partition=config.get("merged", {}).get("partition", config["default_resources"]["partition"]),
        threads=config.get("merged", {}).get("threads", config["default_resources"]["threads"]),
        time=config.get("merged", {}).get("time", config["default_resources"]["time"]),
    threads: config.get("merged", {}).get("threads", config["default_resources"]["threads"])
    container:
        config.get("merged", {}).get("container", config["default_container"])
    message:
        "{rule}: merge fastq files {input}"
    shell:
        "cat {input.fastq} > {output.fastq} 2> {log}"

input / output files

Rule parameters	Key	Value	Description
input	fastq	`merged_input`	Trimmed `.fastq files` (read1 or read2) from the same sample Files obtained by merged_input in defined in `common.smk`
output	fastq	`"prealignment/merged/{sample}_{type}_{read}.fastq.gz"`	Merged `.fastq files` (read1 or read2) from the same sample

Configuration

Software settings (`config.yaml`)

Key	Type	Description
benchmark_repeats	integer	set number of times benchmark should be repeated
container	string	name or path to docker/singularity container

Resources settings (`resources.yaml`)

Key	Type	Description
mem_mb	integer	max memory in MB to be available
mem_per_cpu	integer	memory in MB used per cpu
partition	string	partition to use on cluster
threads	integer	number of threads to be available
time	string	max execution time

pbmarkdup

Mark or remove duplicates in unmapped CCS PACBIO reads from amplified an library.

Rule

rule pbmarkdup:
    input:
        bam=get_pbmarkdup_input,
    output:
        bam="prealignment/pbmarkdup/{sample}_{type}_{processing_unit}_{barcode}.bam",
    params:
        log_level=config.get("pbmarkdup", {}).get("log_level", "WARN"),
        extra=config.get("pbmarkdup", {}).get("extra", ""),
    log:
        "prealignment/pbmarkdup/{sample}_{type}_{processing_unit}_{barcode}.bam.log",
    benchmark:
        repeat(
            "prealignment/pbmarkdup/{sample}_{type}_{processing_unit}_{barcode}.bam.benchmark.tsv",
            config.get("pbmarkdup", {}).get("benchmark_repeats", 1),
        )
    threads: config.get("pbmarkdup", {}).get("threads", config["default_resources"]["threads"])
    resources:
        mem_mb=config.get("pbmarkdup", {}).get("mem_mb", config["default_resources"]["mem_mb"]),
        mem_per_cpu=config.get("pbmarkdup", {}).get("mem_per_cpu", config["default_resources"]["mem_per_cpu"]),
        partition=config.get("pbmarkdup", {}).get("partition", config["default_resources"]["partition"]),
        threads=config.get("pbmarkdup", {}).get("threads", config["default_resources"]["threads"]),
        time=config.get("pbmarkdup", {}).get("time", config["default_resources"]["time"]),
    container:
        config.get("pbmarkdup", {}).get("container", config["default_container"])
    message:
        "{rule}: mark duplicates in {input.bam}"
    shell:
        "pbmarkdup --num-threads {threads} "
        "{params.extra} "
        "{input.bam} "
        "{output.bam} --log-level {params.log_level} &> {log}"

input / output files

Rule parameters	Key	Value	Description
input	bam	`get_pbmarkdup_input`	bam file of unaligned pacbio reads
output	bam	`"prealignment/pbmarkdup/{sample}_{type}_{processing_unit}_{barcode}.bam"`	duplicate marked bam file

Configuration

Software settings (`config.yaml`)

Key	Type	Description
benchmark_repeats	integer	set number of times benchmark should be repeated
container	string	name or path to docker/singularity container
extra	string	parameters that should be forwarded
log_level	string	set the logging level for pbmarkdup

Resources settings (`resources.yaml`)

Key	Type	Description
mem_mb	integer	max memory in MB to be available
mem_per_cpu	integer	memory in MB used per cpu
partition	string	partition to use on cluster
threads	integer	number of threads to be available
time	string	max execution time

Sortmerna

Filter out ribosomal RNA (rRNA) from RNA data

Rule

rule sortmerna:
    input:
        fq1="prealignment/merged/{sample}_{type}_fastq1.fastq.gz",
        fq2="prealignment/merged/{sample}_{type}_fastq2.fastq.gz",
        ref=config.get("sortmerna", {}).get("fasta", ""),
        idx=config.get("sortmerna", {}).get("index", ""),
    output:
        align=temp("prealignment/sortmerna/{sample}_{type}.rrna.fq.gz"),
        kvdb=temp(directory("prealignment/sortmerna/{sample}_{type}/kvdb")),
        other=temp("prealignment/sortmerna/{sample}_{type}.fq.gz"),
        out=temp("prealignment/sortmerna/{sample}_{type}.rrna.log"),
        readb=temp(directory("prealignment/sortmerna/{sample}_{type}/readb")),
    params:
        extra=config.get("sortmerna", {}).get("extra", ""),
        ref=get_sortmerna_refs,
    log:
        "prealignment/sortmerna/{sample}_{type}.rrna.fq.gz.log",
    benchmark:
        repeat(
            "prealignment/sortmerna/{sample}_{type}.rrna.fq.gz.benchmark.tsv",
            config.get("sortmerna", {}).get("benchmark_repeats", 1),
        )
    resources:
        mem_mb=config.get("sortmerna", {}).get("mem_mb", config["default_resources"]["mem_mb"]),
        mem_per_cpu=config.get("sortmerna", {}).get("mem_per_cpu", config["default_resources"]["mem_per_cpu"]),
        partition=config.get("sortmerna", {}).get("partition", config["default_resources"]["partition"]),
        threads=config.get("sortmerna", {}).get("threads", config["default_resources"]["threads"]),
        time=config.get("sortmerna", {}).get("time", config["default_resources"]["time"]),
    threads: config.get("sortmerna", {}).get("threads", config["default_resources"]["threads"])
    container:
        config.get("sortmerna", {}).get("container", config["default_container"])
    message:
        "{rule}: identify ribosomal rna in {input.fq1} and {input.fq2}"
    shell:
        "sortmerna "
        "--fastx "
        "--threads {threads} "
        "--ref {params.ref} "
        "--idx-dir {input.idx} "
        "--reads {input.fq1} "
        "--reads {input.fq2} "
        "--workdir prealignment/sortmerna/{wildcards.sample}_{wildcards.type} "
        "--aligned prealignment/sortmerna/{wildcards.sample}_{wildcards.type}.rrna "
        "--other prealignment/sortmerna/{wildcards.sample}_{wildcards.type} &> {log}"

input / output files

Rule parameters	Key	Value	Description
input	fq1	`"prealignment/merged/{sample}_{type}_fastq1.fastq.gz"`	Unfiltered merged `.fastq` files from read 1 of the same sample
	fq2	`"prealignment/merged/{sample}_{type}_fastq2.fastq.gz"`	Unfiltered merged `.fastq` files from read 2 of the same sample
	ref	`config.get("sortmerna", {}).get("fasta", "")`	Fasta reference genome
	idx	`config.get("sortmerna", {}).get("index", "")`	Sortmera index directory
output	align	`"prealignment/sortmerna/{sample}_{type}.rrna.fq.gz"`	Fastq with reads that align to ribosomal rna
	kvdb	`"prealignment/sortmerna/{sample}_{type}/kvdb"`	Workdir kvd with key-value datastore for alignment results
	other	`"prealignment/sortmerna/{sample}_{type}.fq.gz"`	rRNA filtered merged .fastq file
	out	`"prealignment/sortmerna/{sample}_{type}.rrna.log"`	Workdir readb with temporary read info
	readb	`"prealignment/sortmerna/{sample}_{type}/readb"`	Sortmeras ribosomal log file

Configuration

Software settings (`config.yaml`)

Key	Type	Description
benchmark_repeats	integer	set number of times benchmark should be repeated
container	string	name or path to docker/singularity container
extra	string	parameters that should be forwarded
fasta	array	list of fasta files containing ribosomal rna sequences
index	string	path to index directory

Resources settings (`resources.yaml`)

Key	Type	Description
mem_mb	integer	max memory in MB to be available
mem_per_cpu	integer	memory in MB used per cpu
partition	string	partition to use on cluster
threads	integer	number of threads to be available RECOMMENDATION: Use multiple threads for decreased run time. NOTE: If multiple threads is used it is also best to increase memory (mem_mb)
time	string	max execution time

seqtk_downsample

Downsamples fastq files to the specified number of reads

Rule

rule seqtk_subsample:
    input:
        fastq="prealignment/fastp_pe/{sample}_{type}_{flowcell}_{lane}_{barcode}_{read}.fastq.gz",
        fastp_json="prealignment/fastp_pe/{sample}_{type}_{flowcell}_{lane}_{barcode}_fastp.json",
    output:
        fastq=temp("prealignment/seqtk_subsample/{sample}_{type}_{flowcell}_{lane}_{barcode}_{read}.ds.fastq.gz"),
    params:
        extra=config.get("seqtk_subsample", {}).get("extra", "-2"),
        nr_reads_per_fastq=lambda wildcards: get_nr_reads_per_fastq(
            config.get("seqtk_subsample", {}).get("nr_reads", 1000000000), units, wildcards
        ),
        seed=config.get("seqtk_subsample", {}).get("seed", "-s100"),
        fastp_read_field=lambda wildcards: "read1_after_filtering" if wildcards.read == "fastq1" else "read2_after_filtering",
    log:
        "prealignment/seqtk_subsample/{sample}_{type}_{flowcell}_{lane}_{barcode}_{read}.ds.fastq.log",
    benchmark:
        repeat(
            "prealignment/seqtk_subsample/{sample}_{type}_{flowcell}_{lane}_{barcode}_{read}.ds.fastq.benchmark.tsv",
            config.get("seqtk_subsample", {}).get("benchmark_repeats", 1),
        )
    threads: config.get("seqtk_subsample", {}).get("threads", config["default_resources"]["threads"])
    resources:
        mem_mb=config.get("seqtk_subsample", {}).get("mem_mb", config["default_resources"]["mem_mb"]),
        mem_per_cpu=config.get("seqtk_subsample", {}).get("mem_per_cpu", config["default_resources"]["mem_per_cpu"]),
        partition=config.get("seqtk_subsample", {}).get("partition", config["default_resources"]["partition"]),
        threads=config.get("seqtk_subsample", {}).get("threads", config["default_resources"]["threads"]),
        time=config.get("seqtk_subsample", {}).get("time", config["default_resources"]["time"]),
    container:
        config.get("seqtk_subsample", {}).get("container", config["default_container"])
    message:
        "{rule}: downsample {input.fastq} if it has more reads than the target, otherwise copy it unchanged"
    shell:
        "actual_reads=$(grep -A1 \"{params.fastp_read_field}\" {input.fastp_json} | grep total_reads | grep -oE '[0-9]+'); "
        'if [ "$actual_reads" -le {params.nr_reads_per_fastq} ]; then '
        "cp {input.fastq} {output.fastq}; "
        "else "
        "seqtk sample {params.seed} {params.extra} {input.fastq} {params.nr_reads_per_fastq} | gzip > {output.fastq}; "
        "fi &> {log}"

input / output files

Rule parameters	Key	Value	Description
input	fastq	`"prealignment/fastp_pe/{sample}_{type}_{flowcell}_{lane}_{barcode}_{read}.fastq.gz"`	fastq file to be downsampled
output	fastq	`"prealignment/seqtk_subsample/{sample}_{type}_{flowcell}_{lane}_{barcode}_{read}.ds.fastq.gz"`	downsampled fastq file

Configuration

Software settings (`config.yaml`)

Key	Type	Description
benchmark_repeats	integer	set number of times benchmark should be repeated
container	string	name or path to docker/singularity container
extra	string	parameters that should be forwarded
nr_reads_per_fastq	integer	Number of maximum reads in the subsampled fastq file defined by nr_reads in the config and then divided by number of lanes
seed	string	seed for the random generator (must be the same for R1 and R2)

Resources settings (`resources.yaml`)

Key	Type	Description
mem_mb	integer	max memory in MB to be available
mem_per_cpu	integer	memory in MB used per cpu
partition	string	partition to use on cluster
threads	integer	number of threads to be available
time	string	max execution time

Softwares used in the prealignment module

Fastp

Rule

input / output files

Configuration

Software settings (config.yaml)

Resources settings (resources.yaml)

Fastq merging

Rule

input / output files

Configuration

Software settings (config.yaml)

Resources settings (resources.yaml)

pbmarkdup

Rule

input / output files

Configuration

Software settings (config.yaml)

Resources settings (resources.yaml)

Sortmerna

Rule

input / output files

Configuration

Software settings (config.yaml)

Resources settings (resources.yaml)

seqtk_downsample

Rule

input / output files

Configuration

Software settings (config.yaml)

Resources settings (resources.yaml)

Software settings (`config.yaml`)

Resources settings (`resources.yaml`)

Software settings (`config.yaml`)

Resources settings (`resources.yaml`)

Software settings (`config.yaml`)

Resources settings (`resources.yaml`)

Software settings (`config.yaml`)

Resources settings (`resources.yaml`)

Software settings (`config.yaml`)

Resources settings (`resources.yaml`)