Optimization when posting list are saturated. #2745
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fulmicoton
requested review from
PSeitz and
trinity-1686a
and removed request for
trinity-1686a
PSeitz
reviewed
Nov 25, 2025
| #[derive(Default, Copy, Clone, Debug)] | ||
| struct AllAndEmptyScorerCounts { | ||
| all_count: usize, | ||
| empty_count: usize, |
Collaborator
There was a problem hiding this comment.
Suggested change
| empty_count: usize, | |
| num_empty_scorer: usize, |
PSeitz
reviewed
Nov 25, 2025
|
|
||
| #[derive(Default, Copy, Clone, Debug)] | ||
| struct AllAndEmptyScorerCounts { | ||
| all_count: usize, |
Collaborator
There was a problem hiding this comment.
Suggested change
| all_count: usize, | |
| num_all_scorer: usize, |
PSeitz
reviewed
Nov 25, 2025
src/query/boolean_query/mod.rs
Outdated
| index_writer.add_document(doc!(text_field=>"hello happy"))?; | ||
| index_writer.commit()?; | ||
| let searcher = index.reader()?.searcher(); | ||
| let term_a: Box<dyn Query> = Box::new(TermQuery::new( |
Collaborator
There was a problem hiding this comment.
Suggested change
| let term_a: Box<dyn Query> = Box::new(TermQuery::new( | |
| let hit_all_term: Box<dyn Query> = Box::new(TermQuery::new( |
PSeitz
reviewed
Nov 25, 2025
src/query/boolean_query/mod.rs
Outdated
| Term::from_field_text(text_field, "hello"), | ||
| IndexRecordOption::Basic, | ||
| )); | ||
| let term_b: Box<dyn Query> = Box::new(TermQuery::new( |
Collaborator
There was a problem hiding this comment.
Suggested change
| let term_b: Box<dyn Query> = Box::new(TermQuery::new( | |
| let hit_some_term: Box<dyn Query> = Box::new(TermQuery::new( |
PSeitz
reviewed
Nov 25, 2025
src/query/boolean_query/mod.rs
Outdated
| Term::from_field_text(text_field, "happy"), | ||
| IndexRecordOption::Basic, | ||
| )); | ||
| let term_c: Box<dyn Query> = Box::new(TermQuery::new( |
Collaborator
There was a problem hiding this comment.
Suggested change
| let term_c: Box<dyn Query> = Box::new(TermQuery::new( | |
| let hit_none_term: Box<dyn Query> = Box::new(TermQuery::new( |
PSeitz
reviewed
Nov 25, 2025
src/query/term_query/term_weight.rs
Outdated
| mut boost: Score, | ||
| ) -> crate::Result<SpecializedScorer> { | ||
| if !self.scoring_enabled { | ||
| boost = 1.0f32; |
If a posting list doc freq is the segment reader's max_doc, and if scoring does not matter, we can replace it by a AllScorer. In turn, in a boolean query, we can dismiss all scorers and empty scorers, to accelerate the request.
fulmicoton-dd
force-pushed
the
paul.masurel/optimization-all-empty-scorer
branch
from
a7212d5 to
dc4c218
Compare
PSeitz
reviewed
Nov 25, 2025
src/query/term_query/term_weight.rs
Outdated
| scoring_enabled: bool, | ||
| } | ||
|
|
||
| pub(crate) enum SpecializedScorer { |
Collaborator
There was a problem hiding this comment.
SpecializedTermScorer?
We already have SpecializedScorer in boolean_weight.rs, that's confusing
Collaborator
Author
There was a problem hiding this comment.
I renamed it to something explicit, removed it from the pub(crate) api and made it private.
PSeitz
reviewed
Nov 25, 2025
| let positive_scorer = match (should_scorers, must_scorers) { | ||
| (ShouldScorersCombinationMethod::Ignored, must_scorers) => { | ||
| let boxed_scorer: Box<dyn Scorer> = if must_scorers.is_empty() { | ||
| if must_special_scorer_counts.all_count + should_special_scorer_counts.all_count |
Collaborator
There was a problem hiding this comment.
A comment would be good here
PSeitz
reviewed
Nov 25, 2025
| (CombinationMethod::Ignored, Some(must_scorers)) => { | ||
| SpecializedScorer::Other(intersect_scorers(must_scorers, num_docs)) | ||
|
|
||
| let positive_scorer = match (should_scorers, must_scorers) { |
Collaborator
There was a problem hiding this comment.
Suggested change
| let positive_scorer = match (should_scorers, must_scorers) { | |
| let include_scorer = match (should_scorers, must_scorers) { |
fulmicoton-dd
force-pushed
the
paul.masurel/optimization-all-empty-scorer
branch
3 times, most recently
from
195c5f1 to
814ec2f
Compare
fulmicoton-dd
force-pushed
the
paul.masurel/optimization-all-empty-scorer
branch
from
814ec2f to
e864338
Compare
PSeitz
approved these changes
Nov 26, 2025
fulmicoton-dd
added a commit
that referenced
this pull request
Dec 1, 2025
* Optimization when posting list are saturated. If a posting list doc freq is the segment reader's max_doc, and if scoring does not matter, we can replace it by a AllScorer. In turn, in a boolean query, we can dismiss all scorers and empty scorers, to accelerate the request. * Added range query optimization * CR comment * CR comments * CR comment --------- Co-authored-by: Paul Masurel <[email protected]>
fulmicoton-dd
added a commit
that referenced
this pull request
Dec 1, 2025
* Optimization when posting list are saturated. If a posting list doc freq is the segment reader's max_doc, and if scoring does not matter, we can replace it by a AllScorer. In turn, in a boolean query, we can dismiss all scorers and empty scorers, to accelerate the request. * Added range query optimization * CR comment * CR comments * CR comment --------- Co-authored-by: Paul Masurel <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If a posting list doc freq is the segment reader's max_doc, and if scoring does not matter, we can replace it by a AllScorer.
In turn, in a boolean query, we can dismiss all scorers and empty scorers, to accelerate the request.