[BUG] Composite DateAggregation silently drops deprecated interval parameter, which is an acceptable param on server side, client side for DateAggregation

[asimmahmood1]

What is the bug?

CompositeDateAggregation does not honor internval parameter, which is an acceptable param on server side.

How can one reproduce the bug?

This looks like a client-side issue with the opensearch-java 3.x library.
The interval parameter for date_histogram was deprecated in favor of fixed_interval and calendar_interval. While the OpenSearch server still accepts the deprecated interval parameter (which is why queries work in Dashboard), the opensearch-java 3.x client's CompositeDateHistogramAggregationSource class does not include the deprecated interval field during JSON deserialization, causing the parameter to be silently dropped and resulting in the "Invalid interval specified" error. The workaround is to update aggregation queries to use calendar_interval: "month" (for calendar-aware intervals like month/week) or fixed_interval (for fixed durations like "30d").

Since it is a opensource component and not a managed service one we recommend opening a GitHub issue as well with the deep dive at https://github.com/opensearch-project/opensearch-java to request backward compatibility support for the deprecated interval parameter in composite date histogram sources, or to improve the error messaging when deprecated parameters are used.

REST client:

Aggregation Query :{"NAME-composite":{"composite":{"sources":[{"NAME-sources":{"date_histogram":{"field": "screening.DERIVED_UNIFIED_TIMESTAMP","interval":"month"}}}]}}}

Error :Invalid interval specified, must be non-null and non-empty"

So on the server side we do accept the interval field, so should only require client side change:

[ec2-user@ip-172-31-61-197 ~]$ curl -X POST "localhost:9200/big5/_search" -H "Content-Type: application/json" -d'
{
  "size": 0,
  "aggs": {
    "date_composite": {
      "composite": {
        "sources": [
          {
            "date_bucket": {
              "date_histogram": {
                "field": "@timestamp",
                "fixed_interval": "1d"
              }
            }
          }
        ]
      }
    }
  }
}'
{"took":42,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":10000,"relation":"gte"},"max_score":null,"hits":[]},"aggregations":{"date_composite":{"after_key":{"date_bucket":1673308800000},"buckets":[{"key":{"date_bucket":1672531200000},"doc_count":5800000},{"key":{"date_bucket":1672790400000},"doc_count":5800000},{"key":{"date_bucket":1673049600000},"doc_count":5800000},{"key":{"date_bucket":1673308800000},"doc_count":5800000}]}}}[ec2-user@ip-172-31-61-197 ~]$
[ec2-user@ip-172-31-61-197 ~]$ curl -X POST "localhost:9200/big5/_search" -H "Content-Type: application/json" -d'
{
  "size": 0,
  "aggs": {
    "date_composite": {
      "composite": {
        "sources": [
          {
            "date_bucket": {
              "date_histogram": {
                "field": "@timestamp",
                "interval": "1d"
              }
            }
          }
        ]
      }
    }
  }
}'
{"took":11,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":10000,"relation":"gte"},"max_score":null,"hits":[]},"aggregations":{"date_composite":{"after_key":{"date_bucket":1673308800000},"buckets":[{"key":{"date_bucket":1672531200000},"doc_count":5800000},{"key":{"date_bucket":1672790400000},"doc_count":5800000},{"key":{"date_bucket":1673049600000},"doc_count":5800000},{"key":{"date_bucket":1673308800000},"doc_count":5800000}]}}}[ec2-user@ip-172-31-61-197 ~]$

What is the expected behavior?

Either implement interval param or throw an exception that is unsupported.

What is your host/environment?

n/a

Do you have any screenshots?

No

Do you have any additional context?

Should OS java client support deprecated params?

Reported by a customer.

[kkhatua]

@asimmahmood1 are the parameters missing in the api-specs repo?

[asimmahmood1]

Yes, the interval param is missing in the spec:

https://github.com/opensearch-project/opensearch-api-specification/blob/6a051ee87c22dbc87dd6bb30ca69c791426ca67e/spec/schemas/_common.aggregations.yaml#L2042-L2056

So fix should be straight forward.

Also, the param deprecation was marked in ES7/OS1 but we didn’t remove in 2.0 or 3.0. Same issue will occur with AOSS as well, so it’s better for Cx to switch to fixed_interval and calendar_interval due to: (from commit)

The date_histogram accepts an interval which can be either a calendar interval (DST-aware, leap seconds, arbitrary length of months, etc) or fixed interval (strict multiples of SI units). Unfortunately this is inferred by first trying to parse as a calendar interval, then falling back to fixed if that fails. This leads to confusing arrangement where 1d == calendar, but 2d == fixed. And if you want a day of fixed time, you have to specify 24h (e.g. the next smallest unit). This arrangement is very error-prone for users.

We can add that documentation as well: https://docs.opensearch.org/latest/aggregations/bucket/date-histogram/#choose-the-right-interval