DRAGEN Joint Detection DO NOT MERGE

[davidbenjamin]

Initial implementation of DRAGEN joint detection. Functional equivalence with respect to DRAGEN joint detection is actually worse than before due to many outstanding questions.

We currently have no idea how joint detection is supposed to interaction with BQD and FRD. I have tried a few guesses and none have worked (see below for functional equivalence results of the particular guess used in this PR). The interplay of joint detection with BQD and FRD is complicated for several reasons.

Naively one would simply define the BDQ and FRD likelihoods on entire haplotypes rather than alleles at one locus. Unresolved difficulties with this include:

• BQD and FRD are defined with respect to one particular variant position. How would we define them for a haplotype that has no particular locus?
• BQD involves the base qualities at one particular variant locus, how would this be defined for an entire haplotype?
• The above is especially thorny for haplotypes that exhibit multiple variants.
• The FRD prior is only defined for individual events, not haplotypes.
• The BQD and FRD models use reads that overlap a variant site, but it is not clear how to use reads that only partially intersect a haplotype.
• BQD and FRD likelihoods are only defined for homozygous haplotypes, but heterozygous combinations of haplotypes contribute to homozygous genotypes all loci where the distinct haplotypes agree.

Clearly, generalizing BQD and FRD to entire haplotypes is not straightforward. Nor does it suffice to produce "raw" genotype likelihoods using the joint detection approach and then apply BQD and FRD on variant loci afterwards. Some difficulties with this include:

• BQD and FRD require the read-allele likelihoods matrix. Where are these likelihoods supposed to come from? The pre-joint-detection unrigorous "marginalization" where to each allele we assign the maximum likelihood over all haplotypes supporting that allele? Some read-allele likelihoods matrix derived from the read-haplotype likelihoods matrix?
• The drawbacks of the faulty "marginalization" actually become more severe with joint detection since genotyping multiple alleles together in a single determined span produces more haplotypes, which in turn increases the risk of the read-allele likelihoods cherry-picking from too many different haplotypes for different reads.
• The BQD and FRD models produce likelihoods on an absolute scale that is only meaningful relative to genotyping likelihoods from the pre-joint-detection approach. They do not inherently "play nicely" with the posterior probabilities produced by joint detection.
• BQD and FRD as currently implemented in the GATK modify likelihoods before applying a prior, whereas joint detection yields posterior probabilities. Are we supposed to somehow un-apply the prior to joint detection likelihoods, apply BQD and FRD, then re-apply the prior? It is not clear.

[davidbenjamin]

Here is the functional equivalence report for this branch. It is slighlty less functionally-equivalent than the GATK master branch.
jd-report.pdf

[davidbenjamin]

@eitanbanks @droazen @jamesemery @ldgauthier Shelving this pending guidance on implementation.