Add rare_terms bucket aggregation #9826
Conversation
…ve aggregations/pipeline-agg.md; Add aggregations/pipeline/index.md. Individual aggregation files added in other PRs. Signed-off-by: Dave Welsch <[email protected]>
Signed-off-by: Dave Welsch <[email protected]>
dwelsch-esi
requested review from
kolchfa-aws,
Naarcha-AWS,
AMoo-Miki,
natebower,
dlvenable and
epugh
as code owners
|
Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Editorial review -> Merged. Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer. When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). The doc reviewer will arrange for an editorial review. |
|
@sandeshkr419 Could you review this PR? Thanks! |
_aggregations/bucket/terms.md
Outdated
| @@ -59,6 +59,9 @@ GET opensearch_dashboards_sample_data_logs/_search | |||
| The values are returned with the key `key`. | |||
| `doc_count` specifies the number of documents in each bucket. By default, the buckets are sorted in descending order of `doc-count`. | |||
|
|
|||
| It is possible to use `terms` to search for infrequent values by ordering returned values by ascending count ( `"order": {"count": "asc")` ). We strongly discourage this practice since doing so can cause large unknown errors if multiple shards are involved. We recommend using `rare_terms` instead. | |||
There was a problem hiding this comment.
Let's add some more clarification here:
The issue arises because a term that is globally infrequent might not appear as infrequent on every individual shard, or it might be entirely absent from the top (or, in this case, bottom) results returned by some shards. Conversely, a term might seem infrequent on one shard but be quite common on another. In both the scenarios, the rare term might be missed in individual shard results, which can lead to incorrect overall results.
There was a problem hiding this comment.
Added this clarification, thanks!
_aggregations/pipeline/index.md
Outdated
|
|
||
| A sibling aggregation must be a multi-bucket aggregation (have multiple grouped values for a certain field) and the metric must be a numeric value. | ||
|
|
||
| `min_bucket`, `max_bucket`, `sum_bucket`, and `avg_bucket` are common sibling aggregations. |
There was a problem hiding this comment.
I don't think OpenSearch has documentation to these sibling aggregations. Might be good to list the available sibling aggregations out in a tabular format with small description.
Same applies for parent aggregations as well.
This folder wil probably have all pipeline aggregators: https://github.com/opensearch-project/OpenSearch/tree/main/server/src/main/java/org/opensearch/search/aggregations/pipeline
For each aggregator builder classes, look for the NAME property, which should tell you that the corresponding name is one of the pipeline aggregation. Example: https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/search/aggregations/pipeline/AvgBucketPipelineAggregationBuilder.java#L48 tells that avg_bucket is one of the pipeline aggregation.
If you think there is a chance to improve java docs on the same, please open up an issue in core listing out the aggregations, where you think we need to improve on java docs.
There was a problem hiding this comment.
If the plan is to include all parent and all sibling aggregations in detail like #9796, then probably a table of links might be a good follow-up instead of a tabular format.
There was a problem hiding this comment.
Sure, will do in a separate PR. There's a PR up for these aggregations #9599. Once it's merged, I'll add links to the corresponding pages.
_aggregations/pipeline/index.md
Outdated
|
|
||
| The following parameters are optional: | ||
|
|
||
| - `gap_policy`: Real-world data can contain gaps or null values. You can specify the policy to deal with such missing data with the `gap_policy` property. You can either set the `gap_policy` property to `skip` to skip the missing data and continue from the next available value, or `insert_zeros` to replace the missing values with zero and continue running. |
There was a problem hiding this comment.
Lets just say this is used to handle data gaps and then link it to bottom section of the page where you talk about data gaps in detail.
It feels too much detail here when it is explained in more detail in another section.
There was a problem hiding this comment.
This is the old version of the index page. Just merged the new version, where this is implemented.
|
I see pipeline aggregations is also rewritten (which looks very neat now, kudos) in the same PR. |
Signed-off-by: Fanit Kolchina <[email protected]>
Signed-off-by: Fanit Kolchina <[email protected]>
|
Thank you, @sandeshkr419! I addressed your comments. |
There was a problem hiding this comment.
@kolchfa-aws Please see my comments and changes and let me know if you have any questions. Thanks!
Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]>
Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]>
kolchfa-aws
added
6 - Done but waiting to merge
Signed-off-by: Fanit Kolchina <[email protected]>
github-project-automation
bot
moved this from In Progress
to Done
in Performance Roadmap
* Refactor pipeline aggregations to match other aggregation types. Remove aggregations/pipeline-agg.md; Add aggregations/pipeline/index.md. Individual aggregation files added in other PRs. Signed-off-by: Dave Welsch <[email protected]> * Add rare_terms bucket aggregation. Signed-off-by: Dave Welsch <[email protected]> * Doc review Signed-off-by: Fanit Kolchina <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> * Remove unnecessary redirects Signed-off-by: Fanit Kolchina <[email protected]> --------- Signed-off-by: Dave Welsch <[email protected]> Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Co-authored-by: Fanit Kolchina <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Co-authored-by: Nathan Bower <[email protected]> (cherry picked from commit 0364466) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Description
Add rare_terms bucket aggregation.
Issues Resolved
Version
Frontend features
Checklist
For more information on following Developer Certificate of Origin and signing off your commits, please check here.