Service profile retry and retryBudget have no effect

[stephaneey]

--------------^ Click "Preview" for a better view!
Hello,

Tried to ask on Slack but it is really a m... and couldn't find any related question.

I'm trying to figure out the retry mechanism of serviceProfiles. I built a sample app that purposely throws exceptions (about ~50%). I expected Service Profiles to increase the success rate, however, whatever I try, I see absolutely no difference. The topology I'm using is the following:
traffic generator calling {addition, multiplication, division, substraction, percentage) and percentage calling both multiplication and division. I am purposely generating division by zero exceptions. So, my division service returns a 500 error code whenever a division by zero is performed.
Since the traffic generator is a mere console app and not a web service, I cannot create a route between this guy and the others (unless it is yet possible?). As a console app, it doesn't have any openapi nor protobuf. Tried to use the auto-creation of ServiceProfile against it but it wasn't able to find any route. So, I created a ServiceProfile on the Percentage (itself calling the division service that returns the 500 exception) service which is as follows:

spec:
  retryBudget:
    minRetriesPerSecond: 10
    retryRatio: 0.5
    ttl: 10s
  routes:
  - condition:
      method: POST
      pathRegex: /percentage
    isRetryable: true
    name: POST /percentage

I tried every kind of settings for the retryBudget as well as only enabling isRetryable. Whatever I try, my stats between percentage & division always remain the same:

PS C:\Users\steph\Source\Repos\AKSDemos> while($true){linkerd -n rest-calculator routes deploy/percentage --to svc/division -o wide;sleep 2}

ROUTE             SERVICE   EFFECTIVE_SUCCESS   EFFECTIVE_RPS   ACTUAL_SUCCESS   ACTUAL_RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99
POST /division   division              47.20%          7.5rps           47.20%       7.5rps           1ms           4ms          13ms
[DEFAULT]        division                   -               -                -            -             -             -             -

ROUTE             SERVICE   EFFECTIVE_SUCCESS   EFFECTIVE_RPS   ACTUAL_SUCCESS   ACTUAL_RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99
POST /division   division              51.13%          7.4rps           51.13%       7.4rps           1ms           6ms          13ms
[DEFAULT]        division                   -               -                -            -             -             -             -

ROUTE             SERVICE   EFFECTIVE_SUCCESS   EFFECTIVE_RPS   ACTUAL_SUCCESS   ACTUAL_RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99
POST /division   division              51.13%          7.4rps           51.13%       7.4rps           1ms           6ms          13ms
[DEFAULT]        division                   -               -                -            -             -             -             -

We also clearly see that there is no difference between EFFECTIVE_SUCCESS and ACTUAL_SUCCESS
I must probably be missing something obvious but I don't see what.
Thanks
Best Regards

---

[grampelberg]

Retries happen on the client side. Have you injected your load generator?

[olix0r]

The problem is that we don't currently support retrying post requests. From the docs

For routes that are idempotent and don't have bodies, you can edit the service profile and add isRetryable to the retryable route

POST requests are non-idempotent and necessarily have payloads.

It's a reasonable feature request, but we'd ideally like someone in the community to take on fleshing out the design of the feature.

[stephaneey]

Thanks, in the meantime, I indeed got the confirmation. Didn't realize that POST requests with a body were not subject to retry.

[mbelang]

@olix0r Was there any design submitted yet or any kind of progress regarding supporting POSTs

[alexklibisz]

FWIW, this makes retries unusable for gRPC services, which use POST for every request.

[adleong]

@alexklibisz that's correct. See #3985 for the tracking issue.