improve clustering performance for divergent sequences

[xvazquezc]

Hi there,

MASH is good for very similar sequences but might not be the best for long evolutionary distances. I saw blastp is the alternative but it's currently encoded to run one sequence at a time in a loop. Commands from BLAST+ don't scale well past 2-4 threads, so it would be better to make the function multithreaded. Alternatively, this could be done through GNU Parallel or DIAMOND as they should be easy in-place replacement and they scale pretty well. Another option could be to use MMSeqs2.

What do you think?

[Mayne941]

Hi, yes absolutely, we do recommend BLASTp mode for datasets with dissimilar genomes, and the current way we call it makes datasets ~>1000 genomes intractable. It's on our development roadmap to make this more efficient, it's in theory a quick fix but might be a bit fiddly, as the function uses a dictionary for seen pairs of sequences that get referenced and appended to each run through the loop. For this reason I think a pure Python solution is most straightforward, i.e. set up a multiprocessing queue to get around the GIL.

We also have a roadmap item to look at MMSeqs2 as this was recommended by a reviewer. Have bumped both these items up higher up the roadmap, will update when we've made progress :)