Nextflow DSL2 pipeline for fungal genome prediction and annotation on SLURM HPC. Covers the full funannotate workflow: genome cleaning → repeat masking → RNA-seq discovery → download → train → predict → (optional) annotate.
The pipeline is decomposed into process modules and subworkflows:
modules/funannotate/
setup/ SETUP_TAXONDB
genome/ GENOME_CLEAN, GENOME_CLEAN_BATCH, MASKREPEAT_TANTAN_RUN
rnaseq/ SRA_QUERY_BATCH, COLLECT_SRA_QUERY, SRA_FETCH, SRA_FETCH_SE,
WRITE_EMPTY_READS, RNASEQ_PREPARE, FUNANNOTATE_TRAIN
predict/ GENEMARK_RUN, FUNANNOTATE_PREDICT, FUNANNOTATE_UPDATE,
BACKFILL_ABINITIO_PARAMS
function/ ANTISMASH_RUN, INTERPROSCAN_RUN, SIGNALP_RUN,
FUNANNOTATE_ANNOTATE
subworkflows/local/
FUNANNOTATE_GENOME_PREP.nf genome cleaning + repeat masking
FUNANNOTATE_RNASEQ.nf SRA query → fetch → trinity → train
FUNANNOTATE_PREDICTION.nf GeneMark → funannotate predict (ab-initio reuse + staleness)
FUNANNOTATE_ANNOTATION.nf antiSMASH + InterProScan + SignalP + update + annotate
workflows/funannotate.nf orchestrates the 4 subworkflows
Launch via the single entry point nextflow/main.nf:
nextflow run nextflow/main.nf \
-c nextflow/nextflow.config \
-profile funannotate --pipeline funannotate \
-resumeOr via the SLURM wrapper (recommended for production):
sbatch nextflow/run_funannotate.shGeneMark's license forbids redistribution, so it can never ship in a
container — this matters because the pipeline's training/predict steps are
migrating from a host environment-module install onto a rust-optimized
Singularity container (Trinity/PASA/EVM all confirmed working there).
GeneMark had to come out of funannotate predict's internal call and become
its own step: GENEMARK_RUN (modules/funannotate/predict/GENEMARK_RUN/),
always host-module-only, running between FUNANNOTATE_TRAIN and
FUNANNOTATE_PREDICT:
RNASEQ_PREPARE (species rep only, once) → Trinity-GG assembly
│
▼
FUNANNOTATE_TRAIN (per strain) → reuses shared Trinity, runs PASA
│ → training/funannotate_train.pasa.gff3
│ → training/transcript.alignments.bam
▼
GENEMARK_RUN (per strain, ES mode) → gmes_petap.pl --ES (fresh) or
│ --predict_with <mod> (fast reuse,
│ genuinely training-free, not just a
│ faster seeded retrain)
│ → <out>.genemark.gtf
▼
FUNANNOTATE_PREDICT → --genemark_gtf <out>.genemark.gtf
(predict's own internal GeneMark call never runs;
confirmed via real testing, not just source reading)
Design/validation record: nextflow/docs/GENEMARK_RUN_DESIGN.md (full
design, Fable-reviewed before implementation) and
analysis/genemark_run_validation/ (real end-to-end test results).
Current status (2026-08-12):
- ES mode: built, wired into
FUNANNOTATE_PREDICTION.nf, real end-to-end validated (fresh training and fast--predict_withreuse both produce results matching the original internal-GeneMark baseline; realpredict --genemark_gtfconsumption confirmed equivalent, 11,202 vs. 11,198 genes). Shipped,run_genemark=trueby default. - ET mode (RNA-seq-informed, via
training/transcript.alignments.bam): built and wired, real end-to-end validated (10,780 genes through the actual wired module, matching the standalone recipe's 10,776). Falls back to ES automatically for genomes with no RNA-seq. Set--genemark_mode ETto enable (defaultES). T-022 closed,todo/TODO_REGISTRY.md. - Ab-initio parameter reuse (species-representative
.modsharing,--predict_with) composes with the pipeline's existing ANI-driven representative/sibling reuse system (FUNANNOTATE_PREDICTION.nf,BACKFILL_ABINITIO_PARAMS) — a representative's freshly-trained.modgets backfilled togene_prediction_shared_abinitio/<species>/, and eligible siblings get the fast training-free reuse path automatically. - GeneMark's contribution to the final annotated gene set is highly
genome-dependent — from negligible (0.4%) to a genome's majority (68%,
on a small/fragmented assembly) — see finding F-008
(
.living/findings/funannotate-genemark-contribution.md). This is whyGENEMARK_RUNexists rather than relying on--auto-skip-genemark's graceful degradation once predict moves to the container. - No RNA-seq for a genome does not skip
GENEMARK_RUN— ES mode is pure genome self-training with no RNA-seq dependency, so it runs unconditionally for every genome reaching the predict stage.
samples.csv
│
▼
SETUP_TAXONDB Download NCBI taxdump (storeDir; once per deployment)
│
▼
GENOME_CLEAN Decompress genome, run NCBI FCS-GX contamination purge,
│ filter short contigs → input_clean_genomes/<asmid>.fa.gz
│ (gzip-compressed to save space; storeDir; skipped if a clean
│ .fa.gz — or legacy uncompressed .fa — already exists)
▼
MASKREPEAT_TANTAN_RUN Soft-mask repeats with tantan via funannotate mask
│ → input_clean_genomes/<asmid>.masked.fasta.gz (gzip-compressed)
│ (storeDir; skipped if masked file exists; skippable with
│ --run_repeatmasker false, falls back to clean .fa[.gz])
▼
SRA_QUERY_BATCH Batched NCBI SRA survey — up to sra_query_batch_size species
│ per SLURM job (default 100), max 10 concurrent jobs.
│ Runs esearch/efetch runinfo query per species; no downloading.
│ Species already cached in rnaseq_reads/sra_query/ are reused
│ (file copy, no NCBI call); only uncached species are queried.
│ Primary query: PAIRED[Layout]; records up to 5 candidates.
│ SE fallback: if no PE hits found AND --enable_single_end true,
│ a second SINGLE[Layout] query runs and records up to
│ --max_rnaseq_se_runs candidates.
│ Output: rnaseq_reads/sra_query/<species_tag>.sra_query.csv
│ Columns: species_tag, taxonid, sra_accession, spots, platform, layout
│ (publishDir copy, overwrite: false; delete CSV to re-query)
│ (only runs when --run_sra_fetch true)
▼
COLLECT_SRA_QUERY Merges all per-species query CSVs into a single named manifest
│ written alongside the input samples file:
│ <stem>.rnaseq_sra.csv (e.g. samples.rnaseq_sra.csv)
│ Columns: species_tag, taxonid, sra_accession, spots, platform, layout
│ ── stop here with --stop_after_sra_query true ──
▼
├── (species with PE accessions not overridden to SE)
│ ▼
│ SRA_FETCH Download up to --max_rnaseq_runs PE accessions (read from
│ │ the per-species query CSV — no NCBI call at this stage).
│ │ Per-accession: parallel-fastq-dump → enforce read-pair
│ │ length (EBI FTP fallback on mismatch) → bbnorm (target=30,
│ │ ecc=t) → fastp QC trim.
│ │ Writes rnaseq_reads/<species_tag>_norm_{R1,R2}.fastq.gz
│ │ and a zero-byte _norm_SE.fastq.gz stub.
│ │ (storeDir; skipped if all three files already exist)
│ │ Accessions with blacklist action SE_trinity are skipped
│ │ here — they route to SRA_FETCH_SE instead.
│
├── (species with SE_trinity blacklist overrides OR SINGLE-layout hits
│ when --enable_single_end true; PE always takes priority)
│ ▼
│ SRA_FETCH_SE Download up to --max_rnaseq_se_runs SE accessions.
│ │ SE_trinity: pfd --split-files, take _1 only (real SE data).
│ │ SINGLE layout: pfd gives ACC.fastq.gz or ACC_1.fastq.gz.
│ │ EBI FTP fallback on pfd failure.
│ │ Per-accession: fix_fastq_header_trinity → bbnorm (target=30,
│ │ ecc=f) → fastp QC trim.
│ │ Writes zero-byte _norm_{R1,R2}.fastq.gz PE stubs and
│ │ rnaseq_reads/<species_tag>_norm_SE.fastq.gz.
│ │ (storeDir; skipped if all three files already exist)
│
└── (species with no usable SRA hits)
▼
WRITE_EMPTY_READS Writes zero-byte placeholder files without a SLURM
download job:
rnaseq_reads/<species_tag>_norm_{R1,R2,SE}.fastq.gz
(storeDir; skipped if placeholder files already exist)
│
│ ── stop here with --stop_after_sra_fetch true ──
▼
RNASEQ_PREPARE Run funannotate train --stop_after_trinity on the
│ representative (first) assembly per species.
│ PE mode: --left_norm / --right_norm + --jaccard_clip
│ SE mode: --single_norm (no --jaccard_clip)
│ Archives Trinity-GG FASTA to rnaseq_data/<species_tag>.trinity-GG.fasta
│ (storeDir). All other strains of the same species reuse this.
▼
FUNANNOTATE_TRAIN Run funannotate train on every assembly (4 branches):
│ - Shared Trinity + PE reads: PASA only with --left_norm/--right_norm
│ - Shared Trinity + SE reads: PASA only with --single_norm
│ - No Trinity + PE reads: full train with --left_norm/--right_norm
│ - No Trinity + SE reads: full train with --single_norm
│ Removes large intermediates (hisat2/, trinity_gg/) after completion.
│ Skipped at channel level if funannotate_train.pasa.gff3 already exists.
▼
FUNANNOTATE_PREDICT Run funannotate predict (Augustus + EvidenceModeler).
│ Uses training data linked from genome_annotation/<out>/training/.
│ Skipped if predict_results/<out>.gbk (or .gbk.gz) already exists.
│ Output: genome_annotation/<out>/predict_results/
▼
(optional post-predict steps, run in parallel)
├── ANTISMASH_RUN antiSMASH BGC prediction
├── INTERPROSCAN_RUN InterProScan 5 XML
└── SIGNALP_RUN SignalP 6 (GPU)
│
▼
FUNANNOTATE_ANNOTATE funannotate annotate — functional annotation, GO terms,
BUSCO evidence; optionally incorporates antiSMASH and
InterProScan results.
Output: genome_annotation/<out>/annotate_results/
RNA-seq is handled per species (all strains of a species share the same reads). A species uses one of two strategies. They are alternatives: you can use a different one for each species in the same run, but you normally do not combine them for one species.
| A. Public reads from NCBI SRA (default) | B. Your own reads | |
|---|---|---|
| You do | nothing | put the files in rnaseq_reads/ (below) |
| Pipeline does | queries SRA, downloads and normalizes up to --max_rnaseq_runs runs |
uses your files as they are |
| Result | rnaseq_reads/<species_tag>_norm_{R1,R2,SE}.fastq.gz and rnaseq_reads/sra_query/<species_tag>.sra_query.csv |
the same three _norm_ files |
The species tag is SPECIES with spaces changed to _.
How the pipeline decides, per species
- Existing
rnaseq_reads/<species_tag>_norm_*.fastq.gzare reused and nothing is downloaded:SRA_FETCH,SRA_FETCH_SEandWRITE_EMPTY_READSall usestoreDir "${launchDir}/rnaseq_reads". - A cached
rnaseq_reads/sra_query/<species_tag>.sra_query.csvskips the SRA search when it lists at least one run. A header-only CSV counts as cached only when a matching<species_tag>.sra_query.keysidecar exists, because a header-only CSV may come from a failed query. Otherwise the species is queried again. - With neither, the pipeline searches SRA and downloads (strategy A).
Strategy B: supply your own reads
rnaseq_reads/<species_tag>_norm_R1.fastq.gz # paired-end read 1 (libraries joined)
rnaseq_reads/<species_tag>_norm_R2.fastq.gz # paired-end read 2, same order as R1
rnaseq_reads/<species_tag>_norm_SE.fastq.gz # single-end reads, or an empty file
rnaseq_reads/sra_query/<species_tag>.sra_query.csv # header-only (see below)
- Use real files, not links, and trim adapters first. The pipeline does not trim or normalize reads you supply.
- Because of rule 1, SRA_FETCH never downloads for such a species. To also avoid the
NCBI query, run with
--skip_sra_query trueand give every species a CSV. A header-only CSV (species_tag,taxonid,sra_accession,spots,platform,layout) is then routed toWRITE_EMPTY_READS, which finds your files and does nothing.--skip_sra_query trueskips any species with no CSV, so those get no RNA-seq at all. Without--skip_sra_query, add the.sra_query.keysidecar as well. Its text isv2|<taxids>(the species' taxon IDs, most frequent first,+-joined, as built bysraQueryTaxids()inFUNANNOTATE_RNASEQ.nf).
--run_sra_fetch false is a whole-run switch. It turns off SRA, Trinity and
training for every species, including those with your own reads, which then go
straight to prediction as ab-initio. Your own reads are only used when it is true
(the default).
The SRA workflow is split into a lightweight query phase (SRA_QUERY_BATCH) and a heavy download phase (SRA_FETCH / SRA_FETCH_SE). This avoids allocating large SLURM jobs for species that have no RNA-seq data, and makes the discovery results inspectable before committing to downloads.
- Groups up to
--sra_query_batch_sizeunique species (default 100) per SLURM job; at most 10 batch jobs run concurrently (maxForks 10). - Per species within each batch: sends a single
esearch | efetch -format runinfocall to NCBI SRA. Species already cached inrnaseq_reads/sra_query/are reused (file copy only — no NCBI call). Only uncached species hit the network. - Primary (PE) query filter criteria:
txid<taxonid>[Organism:noexp]— exact taxon, no descendantsRNA-Seq[Strategy],PAIRED[Layout]- Read length 75–300 bp
- Illumina or BGI platform
- ≥250,000 read pairs
- Records up to 5 candidates sorted by spot count descending.
- SE fallback (only when
--enable_single_end trueand no PE hits found): Re-runs the query withSINGLE[Layout](Illumina only) and records up to--max_rnaseq_se_runs(default 3) candidates withlayout=SINGLE. - Output CSV columns:
species_tag, taxonid, sra_accession, spots, platform, layout - Results published to
rnaseq_reads/sra_query/<species_tag>.sra_query.csv(publishDir overwrite: false). Delete the per-species CSV to force re-query. - Resources: 1 CPU, 4 GB RAM, 4h per batch job (short queue); up to 2 retries.
- Backward compatibility: old cached CSVs with 5 columns (no
layout) are handled gracefully — the pipeline treats the missing column asPAIRED.
- Concatenates all per-species CSVs into a single project-level manifest.
- Written alongside the input samples file as
<stem>.rnaseq_sra.csv(e.g.,samples.rnaseq_sra.csvwhen--samples samples.csv). - Re-generated each run from the cached per-species CSVs — always reflects the current set of queried species.
- Use
--stop_after_sra_query trueto halt here and inspect the manifest before committing to downloads. Useful for large initial runs or when checking RNA-seq coverage for a new clade.
The channel performs a three-way branch based on the per-species CSV and the
rnaseq_blacklist.csv override file:
- PE accessions present (not overridden) → SRA_FETCH: downloads up to
--max_rnaseq_runspaired-end accessions. Per-accession:parallel-fastq-dump → enforce_seqpair_readlen (EBI FTP fallback on mismatch) → bbnorm (target=30, ecc=t) → fastp. Writes_norm_{R1,R2}.fastq.gzplus a zero-byte_norm_SE.fastq.gzstub. PE always wins — if a species has PE accessions, SE is never fetched. - SE_trinity override OR SINGLE layout (if enabled) → SRA_FETCH_SE: downloads
up to
--max_rnaseq_se_runssingle-end reads. Per-accession:parallel-fastq-dump → fix_fastq_header_trinity → bbnorm (target=30, ecc=f) → fastp. Writes_norm_SE.fastq.gzplus zero-byte_norm_{R1,R2}.fastq.gzPE stubs. - No usable accessions → WRITE_EMPTY_READS: writes zero-byte placeholder files
for all three slots (
_norm_R1,_norm_R2,_norm_SE) in a trivial job.
All three branches write to rnaseq_reads/<species_tag>_norm_{R1,R2,SE}.fastq.gz
and use storeDir, so the result is cached identically regardless of path taken.
The file rnaseq_blacklist.csv (project root) allows per-accession overrides that
change how SRA_FETCH or SRA_FETCH_SE handles a specific run.
Format: CSV with a header row; column 1 is the SRA accession, column 4 is the action.
accession,taxonid,species_tag,action
SRR17583173,,Hemileia_vastatrix,SE_trinity
ERR123456,,Botrytis_cinerea,skip
SRR654321,,Neurospora_crassa,rename_headers
| Action | Effect |
|---|---|
skip |
Exclude the accession entirely from all processing |
rename_headers |
Download normally; replace FASTQ headers with sequential integers before pairing (fixes BGI/MGISEQ block-splitting desync; applied automatically for BGISEQ platform) |
SE_trinity |
Treat this accession as single-end despite SRA metadata saying PAIRED. SRA_FETCH skips it; SRA_FETCH_SE downloads _1.fastq.gz only and discards any _2. Bypasses --enable_single_end — the entry takes effect regardless of that flag. Useful for mislabeled runs where pfd produces mismatched R1/R2. |
Routing priority:
- A species with at least one usable PE accession (not
skip/ notSE_trinity) → SRA_FETCH. AnySE_trinityentries for that species are silently skipped — PE wins. - A species where all PE accessions are
skip/SE_trinity→ SRA_FETCH_SE (always, regardless of--enable_single_end). - A species with only
SINGLE-layout accessions and--enable_single_end true→ SRA_FETCH_SE. - No usable accessions → WRITE_EMPTY_READS.
On the first run with --run_sra_fetch true, SRA_QUERY_BATCH runs for every
unique species_tag in the filtered sample set. Species are collated into batches of
sra_query_batch_size (default 100); each batch is a single SLURM job. Within each
batch, uncached species are queried sequentially and results written to:
rnaseq_reads/sra_query/<species_tag>.sra_query.csv
The header row is always written. If no qualifying accessions are found, the file contains only the header (0 data rows). Failed queries (network error, timeout) are retried up to 3 times with exponential backoff before an empty CSV is written.
On subsequent runs, SRA_QUERY_BATCH still runs for each batch, but species whose
per-species CSV already exists in rnaseq_reads/sra_query/ are served from the cache
(file copy, no NCBI call). Only species without a cached CSV query NCBI.
The channel branch that drives SRA_FETCH vs WRITE_EMPTY_READS is evaluated from
the cached CSV at channel-evaluation time (.readLines().size() > 1). If SRA_FETCH
has also already completed for a species (its _norm_R1/R2.fastq.gz files exist in
rnaseq_reads/), it too is skipped via its own storeDir check.
| Scope | Action |
|---|---|
| Single species | rm rnaseq_reads/sra_query/<species_tag>.sra_query.csv then re-run |
| All species | rm -r rnaseq_reads/sra_query/ then re-run |
| Force re-download (keep query cache) | rm rnaseq_reads/<species_tag>_norm_*.fastq.gz then re-run |
| Full SRA reset | Remove both rnaseq_reads/sra_query/ and rnaseq_reads/<tag>_norm_*.fastq.gz |
| Re-route PE species to SE (SE_trinity) | Add entry to rnaseq_blacklist.csv, then rm rnaseq_reads/<tag>_norm_*.fastq.gz and re-run. storeDir will block re-fetch unless all three _norm_* files are removed. |
| Enable SE for PE-absent species | Set --enable_single_end true, delete the per-species sra_query CSV (to trigger SE fallback query), then re-run. |
Note:
rnaseq_reads/is undercleanup = truescope in the Nextflow work directory, but the actual output files live outside the work directory (storeDir), so they are not removed by Nextflow cleanup. Delete them manually when needed.
genome_annotation/
<Species_Strain>/
predict_results/ ← primary output used by BFD.nf
<name>.gbk GBK (may be compressed to <name>.gbk.gz to save space;
both forms count as complete — see Skip/cache behavior)
<name>.proteins.fa
<name>.cds-transcripts.fa
<name>.gff3
<name>.scaffolds.fa
predict_misc/
trnascan.no-overlaps.gff3
ab_initio_parameters/
training/
funannotate_train.pasa.gff3
*.bam, *.bai
annotate_results/ ← produced only if --run_annotate true
update_results/ ← produced only if --run_update true
antismash_local/ ← produced only if --run_antismash true
input_clean_genomes/
<asmid>.fa FCS-GX cleaned genome
<asmid>.masked.fasta tantan soft-masked genome
clean/ FCS-GX intermediates (.purge.fasta.gz, .fcs_gx-taxonomy.tsv.gz)
rnaseq_reads/
sra_query/
<species_tag>.sra_query.csv per-species SRA survey results (cached)
columns: species_tag, taxonid, sra_accession, spots, platform, layout
<species_tag>_norm_R1.fastq.gz PE: bbnorm+fastp normalized R1 (or 0-byte stub if SE/no data)
<species_tag>_norm_R2.fastq.gz PE: bbnorm+fastp normalized R2 (or 0-byte stub if SE/no data)
<species_tag>_norm_SE.fastq.gz SE: bbnorm+fastp normalized SE reads (or 0-byte stub if PE/no data)
<samples_stem>.rnaseq_sra.csv merged SRA survey manifest (alongside samples.csv)
columns: species_tag, taxonid, sra_accession, spots, platform, layout
rnaseq_data/
<species_tag>.trinity-GG.fasta shared Trinity assembly per species
logs/nextflow/
funannotate_trace.txt
funannotate_report.html
funannotate_timeline.html
| Parameter | Default | Description |
|---|---|---|
--samples |
samples.csv |
Master species/genome table |
--target |
genome_annotation/ |
Output root for funannotate folders |
--source |
/bigdata/.../NCBI_ASM |
Directory of <asmid>/<asmid>_genomic.fna.gz inputs |
--taxon |
"" (all) |
Restrict to RANK:VALUE, e.g. PHYLUM:Ascomycota |
--n_test |
0 (all) |
Limit to first N samples after taxon filter |
--suppress |
"" |
File of ASMIDs to skip (one per line, first comma-delimited field) |
--only_clean |
false |
Stop after GENOME_CLEAN |
--run_repeatmasker |
true |
Run tantan soft-masking |
--run_sra_fetch |
true |
Run SRA_QUERY_BATCH + SRA_FETCH/SRA_FETCH_SE (both phases) |
--stop_after_sra_query |
false |
Halt after COLLECT_SRA_QUERY; produces <stem>.rnaseq_sra.csv only |
--stop_after_sra_fetch |
false |
Halt after SRA_FETCH/SRA_FETCH_SE; skip train/predict |
--sra_query_batch_size |
100 |
Species per SRA_QUERY_BATCH job; reduce if batches time out |
--max_rnaseq_runs |
2 |
Max PE accessions downloaded per species (SRA_FETCH; SRA_QUERY_BATCH surveys up to 5) |
--enable_single_end |
false |
Enable SE fallback: query SINGLE-layout SRA accessions when no PE found; required for SINGLE-layout species to be fetched. SE_trinity blacklist overrides bypass this flag. |
--max_rnaseq_se_runs |
3 |
Max SE accessions downloaded per species (SRA_FETCH_SE) |
--run_annotate |
false |
Run funannotate annotate after predict |
--run_update |
false |
Run funannotate update (requires --run_sra_fetch true) |
--run_antismash |
false |
Run antiSMASH before annotate |
--run_interpro |
false |
Run InterProScan 5 before annotate |
--run_signalp |
false |
Run SignalP 6 GPU before annotate |
--pasa_mysql |
true |
Use per-task MariaDB for PASA (recommended) |
--max_intronlen |
3000 |
Max intron length passed to funannotate |
--min_intronlen |
10 |
Min intron length passed to funannotate |
--min_contig_len |
2000 |
Contigs shorter than this are dropped after FCS-GX |
--debug |
false |
Print extra channel and task diagnostics |
| Process | Queue | CPUs | Memory | Notes |
|---|---|---|---|---|
SETUP_TAXONDB |
short | 1 | 1 GB | Once per deployment; storeDir-cached |
GENOME_CLEAN |
highmem | 16 | 500 GB | FCS-GX loads /dev/shm/gxdb on h04/h05/h06 |
MASKREPEAT_TANTAN_RUN |
short | 2 | 16 GB | |
SRA_QUERY_BATCH |
short | 1 | 4 GB | Batched esearch/efetch; up to 10 concurrent; up to 2 retries per batch |
COLLECT_SRA_QUERY |
short | 1 | 1 GB | Merges per-species CSVs; fast |
WRITE_EMPTY_READS |
short | 1 | 1 GB | Zero-byte placeholder; no-data species only |
SRA_FETCH |
short → epyc | 24 → 32 | 48 → 192 GB | PE species only; retry bumps memory and queue |
SRA_FETCH_SE |
short → epyc | 24 → 32 | 48 → 192 GB | SE species only (SE_trinity or SINGLE layout); same retry profile as SRA_FETCH |
RNASEQ_PREPARE |
epyc | 16 | 96 GB (+48 GB/retry) | Trinity-GG assembly; up to 3 retries |
FUNANNOTATE_TRAIN |
epyc | 8 → 24 | 96 → 192 GB | PASA alignment; up to 3 retries |
FUNANNOTATE_PREDICT |
epyc | 16 | 32 GB | |
SIGNALP_RUN |
exfab | 16 | 64 GB | Requires --gres=gpu:1 |
INTERPROSCAN_RUN |
epyc | 8 | 32 GB | |
ANTISMASH_RUN |
epyc | 8 | 24 GB | |
FUNANNOTATE_ANNOTATE |
epyc | 16 | 64 GB |
All expensive one-time steps use storeDir, which means Nextflow skips the process
if all declared output files already exist on disk — even without -resume:
| Step | Cache location | Skip condition |
|---|---|---|
SETUP_TAXONDB |
taxondb/ |
taxondb/names.dmp exists |
GENOME_CLEAN |
input_clean_genomes/ |
<asmid>.fa.gz (or legacy <asmid>.fa) exists |
MASKREPEAT_TANTAN_RUN |
input_clean_genomes/ |
<asmid>.masked.fasta.gz (or legacy <asmid>.masked.fasta) exists |
SRA_QUERY_BATCH |
rnaseq_reads/sra_query/ |
per-species CSV exists (checked inside the batch script; publishDir overwrite: false) |
WRITE_EMPTY_READS |
rnaseq_reads/ |
all three <tag>_norm_{R1,R2,SE}.fastq.gz exist (0-byte) |
SRA_FETCH |
rnaseq_reads/ |
all three <tag>_norm_{R1,R2,SE}.fastq.gz exist |
SRA_FETCH_SE |
rnaseq_reads/ |
all three <tag>_norm_{R1,R2,SE}.fastq.gz exist |
RNASEQ_PREPARE |
rnaseq_data/ |
<tag>.trinity-GG.fasta exists |
FUNANNOTATE_TRAIN and FUNANNOTATE_PREDICT skip via channel-level file-existence
checks (funannotate_train.pasa.gff3 and predict_results/<out>.gbk), so they also
skip gracefully when re-running over partially completed datasets.
To save space, cleaned and repeat-masked genomes in input_clean_genomes/ are stored
gzip-compressed: GENOME_CLEAN/GENOME_CLEAN_BATCH emit <asmid>.fa.gz and
MASKREPEAT_TANTAN_RUN emits <asmid>.masked.fasta.gz. The genome path threaded
downstream (genome_fa) may therefore point at a .gz file.
- Resolution: all existence/skip gates use the
genomeFile()helper, which accepts either the compressed (.gz) or a legacy uncompressed form, preferring the compressed one. So a folder compressed after a prior run is not re-cleaned/re-masked, and genomes left uncompressed from older runs still work. - Consumers decompress on the fly:
MASKREPEAT_TANTAN_RUN,RNASEQ_PREPARE,FUNANNOTATE_TRAIN, andFUNANNOTATE_PREDICTeach inflate a.gzgenome to a localgenome_input.fain the task work dir before callingfunannotate ... -i(funannotate cannot read a gzipped FASTA via-i). The work-dir copy is discarded with the task. - EarlGrey pipeline: the separate
nextflow/earlgrey_mask.nfcurated repeat-masking pipeline gets the same treatment — it resolves clean genomes via its owngenomeFile(), inflates.fa.gzbefore EarlGrey/RepeatMasker, and now delivers its soft-masked output compressed asinput_clean_genomes/<asmid>.masked.fasta.gz(which the tantanMASKREPEAT_TANTAN_RUNstoreDir andFUNANNOTATE_PREDICTboth accept). Theselect_repeat_representatives.pyexistence check also accepts.fa.gz.
To save space, a step's GenBank output may be stored gzip-compressed (.gbk.gz)
instead of plain .gbk. All completion/skip gates accept either form (via the
gbkResult() helper), so compressing a finished folder does not force a re-run.
This applies to predict (predict_results/<out>.gbk), update
(update_results/<out>.gbk), and annotate (annotate_results/<out>.gbk), plus the
in-script skip guards in FUNANNOTATE_TRAIN/FUNANNOTATE_PREDICT and the RNA-seq
staleness test below. ANTISMASH_RUN transparently inflates a gzipped predict GBK
to a local copy before running.
Caveats before compressing:
funannotate annotate -i <dir>andfunannotate update -i <dir>readpredict_results/<out>.gbkinternally and cannot consume a.gbk.gz. Only compresspredict_results/<out>.gbkfor genomes that are fully done through any annotate/update you intend to run. With the defaults (run_annotate/run_update/run_antismashallfalse) predict is the terminal artifact, so this is safe.update_resultsandannotate_resultsGBKs are terminal and always safe to compress.- Skip/staleness checks compare RNA-seq mtimes against the GBK mtime, so preserve the
original mtime when compressing (
gzipkeeps it by default; usegzip -nto be safe) to avoid spuriously triggering a stale re-predict. - Post-run assertions (verifying funannotate just wrote output) still expect plain
.gbk, since funannotate always writes uncompressed — compression is a separate housekeeping step applied after a genome completes.
The pipeline also detects when RNA-seq data has been refreshed after a genome was already predicted, and automatically re-runs training and prediction for those genomes.
What "stale" means: A prediction is considered stale when any of the following is
newer than genome_annotation/<out>/predict_results/<out>.gbk (or its .gbk.gz):
rnaseq_reads/<species_tag>_norm_R1.fastq.gz(PE reads), orrnaseq_reads/<species_tag>_norm_SE.fastq.gz(SE reads), orrnaseq_data/<species_tag>.trinity-GG.fasta(shared Trinity assembly).
In both cases the reads used to build the current prediction are no longer the most recent ones on disk.
How the pipeline responds:
-
At the channel level, any assembly whose prediction is stale is moved from the "training complete" bucket into the "needs training" bucket, even though
funannotate_train.pasa.gff3already exists. This forcesFUNANNOTATE_TRAINto run for that assembly. -
Inside the
FUNANNOTATE_TRAINscript, before the normal "training already complete; skipping" guard, the script checks whether the R1 file is newer than the existing GBK (-nttest). If it is, the oldtraining/directory is deleted so funannotate runs a clean re-train rather than silently re-using stale PASA output. -
At the predict channel filter, the same staleness test overrides the normal "GBK exists; skip predict" guard, so
FUNANNOTATE_PREDICTre-runs after the fresh training finishes.
This means a normal pipeline re-run (nextflow run funannotate.nf -resume) will
automatically pick up any new RNA-seq reads without any manual intervention.
SRA_QUERY_BATCH uses publishDir (not storeDir) — the SLURM job always runs,
but the per-species cache check inside the script means only uncached species incur
an NCBI query. COLLECT_SRA_QUERY also always re-merges the per-species CSV cache
into a fresh <stem>.rnaseq_sra.csv, so the manifest always reflects the current
run's species set.
# Genome cleaning
miniconda3, AAFTF, taxonkit
# Repeat masking + prediction + annotation
funannotate
# SRA discovery (SRA_QUERY_BATCH only)
ncbi_edirect
# SRA download + normalisation (SRA_FETCH and SRA_FETCH_SE)
sratoolkit, parallel-fastq-dump, fastp, BBTools, workspace/scratch
# EBI FTP fallback (loaded on demand inside SRA_FETCH / SRA_FETCH_SE)
aria2
# Optional post-predict annotation
signalp/6-gpu — GPU signal peptide prediction
interproscan — domain annotation
antismash — BGC prediction
singularity — MariaDB container for PASA (if pasa_mysql=true)
funannotate.nf produces genome_annotation/<Species_Strain>/predict_results/.
BFD.nf reads from that directory via its SETUP_INPUT step, which creates
symlinks in input/pep/, input/cds/, input/gff3/, input/dna/, and input/trna/.
Run funannotate.nf first, then BFD.nf.
scripts/check_rnaseq_training.py is a standalone audit tool that scans the project
and reports every genome_annotation/ folder that needs a (re-)training run.
Run it before submitting a pipeline job to get a plain-English summary of what is missing or stale, without actually executing anything:
python scripts/check_rnaseq_training.py --tsvFor every entry in samples.csv that has a non-zero
rnaseq_reads/<species_tag>_norm_R1.fastq.gz file and a corresponding
genome_annotation/<out>/ folder, the script evaluates two conditions:
| Status | Meaning |
|---|---|
MISSING_TRAINING |
genome_annotation/<out>/training/ directory does not exist — training has never run for this genome |
NEW_READS |
The training/ directory exists but the R1 file is newer than the directory — RNA-seq reads were refreshed after training last ran |
OK |
Training exists and is up to date (shown only with --report-ok) |
The script derives both the <out> folder name and the <species_tag> file prefix
using the same rules as the Nextflow pipeline:
The STRAIN field from samples.csv is sanitised before the name is assembled:
- Leading/trailing whitespace stripped; quotes (
'") removed. - Only the first
;-delimited token is kept (some entries list synonyms after a semicolon). - Colons (
:) replaced with a space — e.g.CBS:123→CBS 123.
Then:
<out>={SPECIES}_{STRAIN}with spaces →_and special characters ([ ] * ? { }) →_(mirrors themake_outGroovy closure infunannotate.nf)<species_tag>={SPECIES}with spaces →_
Matching is driven by samples.csv rather than directory-name prefix scanning, so
species like Aspergillus sp. and Aspergillus sp. 2663 — which share a common
prefix but are distinct organisms — are never confused.
--project-dir DIR Project root (default: current directory)
--tsv Print a TSV header line before output
--report-ok Also print rows with status OK
Exit code equals the number of folders that need action (0 = nothing to do), making the script safe to use in shell conditionals or CI checks.