Linkerd Proxies rejecting outbound traffic.

[shubhamsharmadvlpr]

Hi Team,

We recently faced an issue while upgrading linkerd from 2.13 --> 2.15 (edge-24.7)

it was already tested on uat and smaller prod envs but when doing it on one of main clusters running ~3k pods with sidecar. pods started to fail outbound request.

we observed following behavior.

• linkerd check was showing success
• Destination/policy containers were failing readiness probe, turns out simple health api was taking 30+ sec causing readiness to fail.
• huge spike in network inbound/outbound traffic for destination pods
• inbound requests were working fine on pods
• outbound calls including external ones were failing from pods that have proxy enabled.

error in proxy when hitting google.com ip.

[ 185.187529s] INFO ThreadId(01) inbound:server{port=8090}:rescue{client.addr=10.1.224.14:43588}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.1.224.14:43588: server: 10.0.217.144:8090: server 10.0.217.144:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable error.sources=[server 10.0.217.144:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable, service unavailable]

scaling linkerd to 2x pods did reduce latency of apis ~ 10 sec. still situation did not improve and caused service degradation.
when rollback did not help either, we removed proxies from some deploments with high replica count and skipped outboud ports 0-65000.

linkerd traffic became stable after 30-45 mins and latency came down to milliseconds.

some questions

1. as destination already had enough resources(cpu/mem) why was it throttling and having high latencies
2. why existing proxies started failing outbound requests, as per docs even if control plane is down proxies should function normally. https://linkerd.io/faq/#what-happens-to-linkerds-proxies-if-the-control-plane-is-down

we're unable to reproduce it, it'd be very helpful if someone can guide here.