Targets in the same group are assigned to the same OTel

[mike9421]

Component(s)

target allocator

Is your feature request related to a problem? Please describe.

otel-allocator allocates multiple container endpoints of the same pod to different OTel. After the relabel operation, different OTel will collect the same metrics. Since I used cumulativeToDelta, the final metric data will be doubled.

Describe the solution you'd like

For example, the same pod detects the following targets:

• 10.22.14.119 --> assigned to OTel-0

• 10.22.14.119:8000 --> assigned to OTel-1

• 10.22.14.119:8080 --> assigned to OTel-2

The address will be changed to 10.22.14.119:8000 after the relabel operation, and all the above OTel will scrape the metric of 10.22.14.119:8000, which causes metric data errors.

Describe alternatives you've considered

After the otel - allocator performs the relabel operation, it unifies and deduplicates the items.
The selection of labels for deduplication is as follows:

• Select the label of the target where the original __address__ completely matches the __address__ after relabeling.

• If no such target exists, the labels of the first target will be used by default, while the metrics with the _meta prefix from the last target will overwrite the labels. This ensures consistency with the behavior of the OTel Prometheus receiver, for detail, see prometheus and prometheus receiver

The following are examples.

Additional context

No response

[swiatekm]

If I understand correctly, you'd like the target allocator to only keep one target with the same __address__? I don't think we can do that, as it's completely valid to scrape the same target multiple times with Prometheus, attaching different labels each time. As far as I know, neither Prometheus nor prometheusreceiver do any target merging based on __address__, and the links you provided do not show this.