Speed up top-k retrieval on filtered conjunctions. #13994
+175
−24
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
A while back we added an optimized bulk scorer that implements block-max AND, this yielded a good speedup on nightly benchmarks, see annotation `FP` at https://benchmarks.mikemccandless.com/AndHighHigh.html. With this PR, filtered conjunctions now also run through this optimized bulk scorer by doing two things: - It flattens inner conjunctions. This makes queries initially written as something like `+(+term1 +term2) #filter` rewritten to `+term1 +term2 #filter`. - It evaluates queries that have a mix of MUST and FILTER clauses evaluated through `BlockMaxConjunctionBulkScorer` by treating FILTER clauses as scoring clauses that produce a score of 0.
|
I ran luceneutil on wikibigall and the new filtered tasks from wikinightly.tasks (https://github.com/mikemccand/luceneutil/blob/main/tasks/wikinightly.tasks#L355-L390): |
benwtrent
approved these changes
Nov 14, 2024
Comment on lines
566
to
589
| for (BooleanClause clause : clauses) { | ||
| if (clause.isRequired() && clause.query() instanceof BooleanQuery innerQuery) { | ||
| if (innerQuery.getMinimumNumberShouldMatch() == 0 | ||
| && innerQuery.getClauses(Occur.SHOULD).isEmpty()) { | ||
| actuallyRewritten = true; | ||
| for (BooleanClause innerClause : innerQuery) { | ||
| Occur occur = innerClause.occur(); | ||
| if (occur == Occur.FILTER | ||
| || occur == Occur.MUST_NOT | ||
| || clause.occur() == Occur.MUST) { | ||
| builder.add(innerClause); | ||
| } else { | ||
| assert clause.occur() == Occur.FILTER && occur == Occur.MUST; | ||
| // In this case we need to change the occur of the inner query from MUST to FILTER. | ||
| builder.add(innerClause.query(), Occur.FILTER); | ||
| } | ||
| } | ||
| } else { | ||
| builder.add(clause); | ||
| } | ||
| } else { | ||
| builder.add(clause); | ||
| } | ||
| } |
There was a problem hiding this comment.
Holy smokes I had to read this like 5 times to fully grok what it is doing.
Could you name occur to innerClauseOccur and change clause to outerClause?
|
@jpountz don't forget CHANGES ;) but this lgtm |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A while back we added an optimized bulk scorer that implements block-max AND, this yielded a good speedup on nightly benchmarks, see annotation
FPat https://benchmarks.mikemccandless.com/AndHighHigh.html. With this PR, filtered conjunctions now also run through this optimized bulk scorer by doing two things:+(+term1 +term2) #filterrewritten to+term1 +term2 #filter.BlockMaxConjunctionBulkScorerby treating FILTER clauses as scoring clauses that produce a score of 0.