Updates for QC, tranches and optional EXIT-RIF gvcf dataset

[abhi18av]

DRAFT PR do not merge yet.

Updates

• Move the FASTQC/MULTIQC checks to the QUALITY_CHECK_WF stage to catch data corruption earlier 👉 fdb959b
• Update the readme to point to the website 👉 66cc26d
• Update the containers to reuse the same conda.yml file 👉 03fa87e
• Add the optional GVCF file from the reference EXIT-RIF dataset 👉 aa6ed01

NOTE: This added the requirement for git lfs install since the file is not downloaded properly without it. Normal Git repositories can't have large files without git lfs. For now, I've sourced that file via http but it can also be downloaded as part of this repo if git-lfs conda package is installed.

• 🚧 Use the tranches file for computing the best set of annotations (MOVED TO A DIFFERENT PR) Tranches optimization #95
• Tweak for the pipeline logic regression due to the updated CSV format 👉 95d450e
• Remove the dead code for TB_PROFILER_LOAD_LIBRARY (initially needed for previous version of tb-profiler) 👉 82cd641

Updated tasks after the meeting on 22-03-2022

• Confirm if the gzip file is corrupted or not within the QC_CHECK workflow; confirm with samples sent via Lennert if FASTQC catches

  • The results of direct gzip -t $fq -v ( bad quality ERR779852_1.fastq.gz 🔴 )

    (fastqc-env) PS /home/abhinav/projects/xbs-nf-dataset/bad-fastq-file-check> foreach ($fq in $listOfFastqs) {
    >> gzip -t $fq -v
    >> }
    ERR751371_1.fastq.gz:
     OK
    ERR751371_2.fastq.gz:
     OK
    ERR779852_1.fastq.gz:
    
    gzip: ERR779852_1.fastq.gz: unexpected end of file
    ERR779852_2.fastq.gz:
     OK
    
    

  • Corresponding results of fastqc $fq (fails ✅ for the bad quality ERR779852_1.fastq.gz and passes for all others)

    
    Approx 85% complete for ERR779852_1.fastq.gz
    Approx 90% complete for ERR779852_1.fastq.gz
    Approx 95% complete for ERR779852_1.fastq.gz
    Failed to process file ERR779852_1.fastq.gz
    uk.ac.babraham.FastQC.Sequence.SequenceFormatException: Ran out of data in the middle of a fastq entry.  Your file is probably truncated
        at uk.ac.babraham.FastQC.Sequence.FastQFile.readNext(FastQFile.java:179)
        at uk.ac.babraham.FastQC.Sequence.FastQFile.next(FastQFile.java:125)
        at uk.ac.babraham.FastQC.Analysis.AnalysisRunner.run(AnalysisRunner.java:77)
        at java.base/java.lang.Thread.run(Thread.java:834)
    
    

• Test without the optional EXIT-RIF GVCF file 👉 Update the parameters and mechanism to use optional input files #94

[abhi18av]

@TimHHH , for some reason the changes introduced in GATK_VARIANTS_TO_TABLE 7f62d81 are causing an issue with the SNPSITES process -> Warning: No SNPs were detected so there is nothing to output.

Full error message below

Error executing process > 'MERGE_WF:PHYLOGENY_ANALYSIS__EXCOMPLEX:SNPSITES (joint)'

Caused by:
  Process `MERGE_WF:PHYLOGENY_ANALYSIS__EXCOMPLEX:SNPSITES (joint)` terminated with an error exit status (1)

Command executed:

  snp-sites -o joint.variable.ExDR.ExComplex.fa joint.ExDR.ExComplex.fa

Command exit status:
  1

Command output:
  (empty)

Command error:
  Warning: No SNPs were detected so there is nothing to output.

Work dir:
  /home/biosharpou/xbs-nf-runs/xbs-nf/work/1f/dd422d5f82f6428aa557b3a7cf27d5

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

[TimHHH]

@abhi18av I am not seeing any issue with the updated sed code, at least when I run it manually on some standard datasets. Could you have a look if the input VCF for these processes has any content? e.g. zcat joint.filtered_SNP.ExDR.IncComplex.vcf.gz | grep -v "#" should give some lines of data. Alternatively could you try again with a larger dataset (include EXIT-RIF)?