[Feature] Support Incremental Mode for metrics streaming

[wu-sheng]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

#8720 is being planned as a part of the OSPP and BanyanDB milestone. OAP need a kernel-level change to adopt this server-side feature.

Use case

Currently, OAP is using this workflow, Receiver -> L1 aggregation -> OAP in-cluster hashing calling -> L2 aggregation -> cache + DB merging + persistent.

By leveraging #8720, once the server-side merging functions are supported (even partially), the workflow could be evolved to Receiver -> L1 aggregation -> no cache and pending persistent.

All these supported metrics would not need to do

1. Serialization and Deserialization for in-cluster aggregation
2. No RPCs among OAP nodes
3. No L2 aggregation
4. No memory cost(no cache mode) before persistent after the L2 aggregation

L1 aggregation would deliver incremental metrics for persistence directly.

---

As OAP could run in the incremental mode, we will face the impacts as there was no latest data.

1. Alerting context will be missed
2. Total mode exporting will not be supported

We could use periodical reading to retrieve metrics for active metrics entities detected by the workflows. <2> will not be supported.

---

Besides the above-mentioned things, we should consider adding configurations to control the OAP mode

1. Run db-side-merging for metrics supported by BanyanDB. DB metadata query should support to indicate the supported function lists.
2. Run original workflow for not supported metrics, even the db-side-merging is supported.
3. Run duplicate writing for every metrics. If the metric functions are supported in <1>,

Related issues

No response

Are you willing to submit a pull request to implement this on your own?

• Yes I am willing to submit a pull request on my own!

Code of Conduct

• I agree to follow this project's Code of Conduct

[wu-sheng]

@hanahmily We could implement most of the logic before the DB side is ready. The only dependencies

1. APIs about whether the metric function is supported
2. APIs about creating db-side-merging metric

[wu-sheng]

Metrics should consider having @DBMergingCapability(function="avg") annotation to indicate this is a metric could support DB server side merging. But the OAP would rely on the server API to check whether the current function is provided at the DB side.

[wu-sheng]

@wankai123 @hanahmily I am moving this to 10.2. Although I am not sure whether we will be ready for this, this is so expected for BanyanDB for sure.