How does ARC work with retries and multiple endpoints?

[tronboto]

Question

I'm curious how Adaptive Request Concurrency works in combination with retries and sinks that support multiple endpoints (e.g. elasticsearch). In particular, how would each of the following configurations affect ARC?

• 1 endpoint that points to a single IP address (likely a load balancer)
• 1 endpoint that points to many IP addresses (multiple A records)
• Many endpoints

It would be nice if the behaviour was documented but all I was able to find was this, which doesn't really cover what I'm after: https://vector.dev/blog/adaptive-request-concurrency/

Vector Config

No response

Vector Logs

No response

[pront]

For the elasticsearch sink, ARC operates per configured endpoint:

• One endpoint, one IP: one ARC controller, including when that IP is a load balancer.
• One endpoint, several DNS addresses: still one ARC controller. DNS resolution and HTTP connection reuse determine which address is used; the resolved IPs do not get separate ARC limits.
• Several endpoints: each endpoint has its own controller, and Vector balances requests across them. The total number of requests in flight can therefore exceed a single endpoint's limit.

Retries go back through the balancer, so a retry can use a different endpoint. Each attempt contributes to the controller for the endpoint it uses; retry backoff is outside that endpoint's response-time measurement.

The relevant implementation is distributed_service. This describes the Elasticsearch sink; the retry and ARC layers are not arranged identically in every sink.

Please don't hesitate to followup in this thread if you have further questions.