Linkerd fails to proxy outgoing gRPC traffic to upstream when application is redployed

[sumit-joshi-mt]

Hi,
Linkerd fails to proxy outgoing grpc traffic if upstream/backend application to which it is talking is down for sometime and when application is up(scaled down to 0 replicas and again scaled back to desired replicas) it does not recovers, looks like application process hangs.
When I remove linkerd and do the same this application throws error "connection refused" when upstream application is down and again starts working after application is up but in case of linkerd application process just hangs does not throw even connection refused error

• linkerd check looks fine.

• linkerd proxy version: edge-20.10.6

• kubernetes version: v1.17.9-eks-4c6976

linkerd-proxy logs

[     0.001439s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[     0.001461s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[     0.001466s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[     0.001470s]  INFO ThreadId(01) linkerd2_proxy: Tap interface on 0.0.0.0:4190
[     0.001473s]  INFO ThreadId(01) linkerd2_proxy: Local identity is default.load-test.serviceaccount.identity.linkerd.cluster.local
[     0.001479s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.001482s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.001871s]  INFO ThreadId(01) outbound: linkerd2_app: listen.addr=127.0.0.1:4140 ingress_mode=false
[     0.001927s]  INFO ThreadId(01) inbound: linkerd2_app: listen.addr=0.0.0.0:4143
[     0.501048s]  INFO ThreadId(02) daemon:identity: linkerd2_app: Certified identity: default.load-test.serviceaccount.identity.linkerd.cluster.local
[     9.732819s]  WARN ThreadId(01) dst: linkerd2_reconnect::service: Service failed error=channel closed
[   117.751303s]  INFO ThreadId(01) outbound: linkerd2_proxy_api_resolve::resolve: No endpoints
[   120.789087s]  WARN ThreadId(01) outbound:accept{peer.addr=10.11.101.238:51752 target.addr=172.20.29.171:80}: linkerd2_app_core::errors: Failed to proxy request: Service in fail-fast
[   120.789138s]  WARN ThreadId(01) outbound:accept{peer.addr=10.11.101.238:51752 target.addr=172.20.29.171:80}: linkerd2_app_core::errors: Failed to proxy request: Service in fail-fast
[   120.789151s]  WARN ThreadId(01) outbound:accept{peer.addr=10.11.101.238:51752 target.addr=172.20.29.171:80}: linkerd2_app_core::errors: Failed to proxy request: Service in fail-fast
[   120.789163s]  WARN ThreadId(01) outbound:accept{peer.addr=10.11.101.238:51752 target.addr=172.20.29.171:80}: linkerd2_app_core::errors: Failed to proxy request: Service in fail-fast
[   120.789170s]  WARN ThreadId(01) outbound:accept{peer.addr=10.11.101.238:51752 target.addr=172.20.29.171:80}: linkerd2_app_core::errors: Failed to proxy request: Service in fail-fast
[   120.789181s]  WARN ThreadId(01) outbound:accept{peer.addr=10.11.101.238:51752 target.addr=172.20.29.171:80}: linkerd2_app_core::errors: Failed to proxy request: Service in fail-fast
[   120.789188s]  WARN ThreadId(01) outbound:accept{peer.addr=10.11.101.238:51752 target.addr=172.20.29.171:80}: linkerd2_app_core::errors: Failed to proxy request: Service in fail-fast
[   120.789200s]  WARN ThreadId(01) outbound:accept{peer.addr=10.11.101.238:51752 target.addr=172.20.29.171:80}: linkerd2_app_core::errors: Failed to proxy request: Service in fail-fast
[   120.789207s]  WARN ThreadId(01) outbound:accept{peer.addr=10.11.101.238:51752 target.addr=172.20.29.171:80}: linkerd2_app_core::errors: Failed to proxy request: Service in fail-fast
[   120.789217s]  WARN ThreadId(01) outbound:accept{peer.addr=10.11.101.238:51752 target.addr=172.20.29.171:80}: linkerd2_app_core::errors: Failed to proxy request: Service in fail-fast
[   174.548564s]  INFO ThreadId(01) outbound: linkerd2_proxy_api_resolve::resolve: No endpoints
[   174.557666s]  INFO ThreadId(01) outbound: linkerd2_proxy_api_resolve::resolve: No endpoints
[   174.684941s]  INFO ThreadId(01) outbound: linkerd2_proxy_api_resolve::resolve: No endpoints
[   174.694179s]  INFO ThreadId(01) outbound: linkerd2_proxy_api_resolve::resolve: No endpoints
[   184.201251s]  INFO ThreadId(01) outbound: linkerd2_proxy_api_resolve::resolve: No endpoints
➜  linkerd2-edge-20.10.6                                                                           

Request flow is --
application --> linkerd-proxy --> k8s services --> backend gRPC pods
(backend/upstream does not uses linkerd in some cases; linkerd is enabled at client side only)

has any one seen this issue? any pointers around this would be helpful.
Thank you.

[grampelberg]

@zaharidichev this sounds kinda like some of the HA stuff you fixed awhile ago. Any ideas?

[zaharidichev]

Hi there, a few questions:

• Are you running Linkerd in HA mode ?
• You mention that when you remove Linkerd, the app still errors/hangs. Why is that?
• Do you have more exact steps to repro all that, perhaps with an example application?

[sumit-joshi-mt]

Hi, @zaharidichev

1. Yes
2. No. Sorry I meant If I remove linkerd and do the same thing application does not hang it throws connection refused error as backend application is down and recovers when application is up again, we can see success responses. Applications hangs only when linkerd is enabled.
3. Its simple grpc go application which pings backend application.
   For ex: Lets say two applications A and B, only A has linkerd enabled. App A calls App B status route (we are trying to load test App B here). When I scale down App B to 0 replicas and scale again to desired replica application hangs. I case of rollout restart we do not see this issue, it happens only when B is completely down or undeployed and deployed again.

[sumit-joshi-mt]

Hi, we did some debugging. applications does not hangs it waits for response as it does not know connection is broken.
Lookslike this is related to keepalive flags. So in case of abrupt termination, connection breaks
Linkerd checks keepalive after 10 sec , if connection is in-active state. It will request another keep-alive at the interval of 75 sec for 9 times (default values). If connection is broken then, it will not receive ack and after waiting for 9 retries linkerd closed connection to the server .
Linkerd is taking around 12 mins to detect connection is broken, is there any way to override this flags? so we can change this flag, 12 mins is significant time to detect connection is closed.

[olix0r]

We've recently enabled HTTP/2 PING keepalives to help detect this sort of issue. I'd recommend testing the most recent edge release to take advantage of these changes. We'd be eager to know whether this resolves the issue or if other changes are needed.