Exported services return 502 in a multicluster setup (Linkerd 2.11.1)

[RicardoNalesAmato]

What is the issue?

A multi-cluster setup created by following the guide found here has issues when services are exported. All services can't be accessed because a 502 error appears.

How can it be reproduced?

It can be reproduced on a newly created linkerd 2.11.1 cluster and following this document

Logs, error output, etc

The linkerd-proxy used by the gateway logs:

linkerd-proxy [   840.027889s]  INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated

Every time it receives a request from a meshed pod in the target cluster

output of linkerd check

Linkerd core checks
===================

kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ control plane pods are ready

linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
‼ issuer cert is valid for at least 60 days
    issuer certificate will expire on 2022-02-12T20:09:47Z
    see https://linkerd.io/2.11/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints
√ issuer cert is issued by the trust anchor

linkerd-webhooks-and-apisvc-tls
-------------------------------
√ proxy-injector webhook has valid cert
√ proxy-injector cert is valid for at least 60 days
√ sp-validator webhook has valid cert
√ sp-validator cert is valid for at least 60 days
√ policy-validator webhook has valid cert
√ policy-validator cert is valid for at least 60 days

linkerd-version
---------------
√ can determine the latest version
√ cli is up-to-date

control-plane-version
---------------------
√ can retrieve the control plane version
√ control plane is up-to-date
√ control plane and cli versions match

linkerd-control-plane-proxy
---------------------------
√ control plane proxies are healthy
√ control plane proxies are up-to-date
√ control plane proxies and cli versions match

Environment

• Kubernetes version: 1.20
• Cluster Environment: EKS
• Host OS: Amazon Linux
• Linkerd Version: 2.11.1

Possible solution

It seems like the client isn't trying to establish TLS to the gateway

Additional context

multicluster check:

linkerd --context=sandbox multicluster check
linkerd-multicluster
--------------------
√ Link CRD exists
√ Link resources are valid
	* sandbox-eks
√ remote cluster access credentials are valid
	* sandbox-eks
√ clusters share trust anchors
	* sandbox-eks
√ service mirror controller has required permissions
	* sandbox-eks
√ service mirror controllers are running
	* sandbox-eks
√ all gateway mirrors are healthy
	* sandbox-eks
√ all mirror services have endpoints
√ all mirror services are part of a Link
√ multicluster extension proxies are healthy
√ multicluster extension proxies are up-to-date
√ multicluster extension proxies and cli versions match

Status check results are √

Would you like to work on fixing this bug?

maybe

[adleong]

The error message direct connections must be mutually authenticated suggests that either the client (i.e. the pod in the source cluster which is attempting to connect to an export service in another cluster) is not meshed, or that the two clusters have not been set up with a shared trust root.

[RicardoNalesAmato]

Hi @adleong, the request is made from a meshed pod in the target cluster. These are the logs from its linkerd-proxy for a request to flagr-eks (eks being the source cluster):

dnstools-ricardo linkerd-proxy [   212.754697s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}: linkerd_cache: Caching new service target=Accept { orig_dst: OrigDstAddr(10.32.150.47:80), protocol: () }
dnstools-ricardo linkerd-proxy [   212.754801s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile: linkerd_app_outbound::discover: Allowing profile lookup
dnstools-ricardo linkerd-proxy [   212.754845s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}: linkerd_stack::failfast: TCP Server service has become unavailable
dnstools-ricardo linkerd-proxy [   212.754868s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: tower::balance::p2c::service: updating from discover
dnstools-ricardo linkerd-proxy [   212.754894s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: tower::buffer::worker: service.ready=true processing request
dnstools-ricardo linkerd-proxy [   212.754971s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:49318}:server{orig_dst=10.32.150.47:80}:profile:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.100.181.246:8086}:h2:Connection{peer=Client}: h2::codec::framed_write: send frame=Headers { stream_id: StreamId(11), flags: (0x4: END_HEADERS) }
dnstools-ricardo linkerd-proxy [   212.755011s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:49318}:server{orig_dst=10.32.150.47:80}:profile:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.100.181.246:8086}:h2:Connection{peer=Client}: h2::codec::framed_write: send frame=Data { stream_id: StreamId(11) }
dnstools-ricardo linkerd-proxy [   212.755021s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:49318}:server{orig_dst=10.32.150.47:80}:profile:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.100.181.246:8086}:h2:Connection{peer=Client}: h2::codec::framed_write: send frame=Data { stream_id: StreamId(11), flags: (0x1: END_STREAM) }
dnstools-ricardo linkerd-proxy [   212.756502s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:49318}:server{orig_dst=10.32.150.47:80}:profile:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.100.181.246:8086}:h2:Connection{peer=Client}: h2::codec::framed_read: received frame=Headers { stream_id: StreamId(11), flags: (0x4: END_HEADERS) }
dnstools-ricardo linkerd-proxy [   212.756550s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:49318}:server{orig_dst=10.32.150.47:80}:profile:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.100.181.246:8086}:h2:Connection{peer=Client}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(11) }
dnstools-ricardo linkerd-proxy [   212.756561s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:49318}:server{orig_dst=10.32.150.47:80}:profile:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.100.181.246:8086}:h2:Connection{peer=Client}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(11) }
dnstools-ricardo linkerd-proxy [   212.756568s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:49318}:server{orig_dst=10.32.150.47:80}:profile:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.100.181.246:8086}:h2:Connection{peer=Client}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(11) }
dnstools-ricardo linkerd-proxy [   212.756659s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}: tower::buffer::worker: service.ready=true processing request
dnstools-ricardo linkerd-proxy [   212.756748s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile: linkerd_detect: DetectResult protocol=Some(Http1) elapsed=10.7µs
dnstools-ricardo linkerd-proxy [   212.756763s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}: linkerd_proxy_http::server: Creating HTTP service version=Http1
dnstools-ricardo linkerd-proxy [   212.756778s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}: linkerd_cache: Caching new service target=Logical { protocol: Http1, profile: .., logical_addr: LogicalAddr(flagr-eks.default.svc.cluster.local:80) }
dnstools-ricardo linkerd-proxy [   212.756831s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}: linkerd_proxy_http::server: Handling as HTTP version=Http1
dnstools-ricardo linkerd-proxy [   212.756867s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}: hyper::proto::h1::io: parsed 3 headers
dnstools-ricardo linkerd-proxy [   212.756873s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}: hyper::proto::h1::conn: incoming body is empty
dnstools-ricardo linkerd-proxy [   212.756913s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}: linkerd_service_profiles::http::route_request: Updating HTTP routes routes=0
dnstools-ricardo linkerd-proxy [   212.756922s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}: tower::buffer::worker: service.ready=true processing request
dnstools-ricardo linkerd-proxy [   212.756936s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}: linkerd_service_profiles::split: Updating targets=[Target { addr: flagr-eks.default.svc.cluster.local:80, weight: 10000 }]
dnstools-ricardo linkerd-proxy [   212.756951s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}: linkerd_proxy_api_resolve::resolve: Resolving dst=flagr-eks.default.svc.cluster.local:80 context={"ns":"default", "nodeName":"ip-172-22-15-171.eu-west-1.compute.internal"}
dnstools-ricardo linkerd-proxy
dnstools-ricardo linkerd-proxy [   212.756980s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}: linkerd_stack::failfast: HTTP Logical service has become unavailable
dnstools-ricardo linkerd-proxy [   212.757018s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: tower::balance::p2c::service: updating from discover
dnstools-ricardo linkerd-proxy [   212.757050s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: tower::buffer::worker: service.ready=true processing request
dnstools-ricardo linkerd-proxy [   212.757125s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:49318}:server{orig_dst=10.32.150.47:80}:profile:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.100.116.205:8086}:h2:Connection{peer=Client}: h2::codec::framed_write: send frame=Headers { stream_id: StreamId(11), flags: (0x4: END_HEADERS) }
dnstools-ricardo linkerd-proxy [   212.757167s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:49318}:server{orig_dst=10.32.150.47:80}:profile:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.100.116.205:8086}:h2:Connection{peer=Client}: h2::codec::framed_write: send frame=Data { stream_id: StreamId(11) }
dnstools-ricardo linkerd-proxy [   212.757176s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:49318}:server{orig_dst=10.32.150.47:80}:profile:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.100.116.205:8086}:h2:Connection{peer=Client}: h2::codec::framed_write: send frame=Data { stream_id: StreamId(11), flags: (0x1: END_STREAM) }
dnstools-ricardo linkerd-proxy [   212.758948s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:49318}:server{orig_dst=10.32.150.47:80}:profile:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.100.116.205:8086}:h2:Connection{peer=Client}: h2::codec::framed_read: received frame=Headers { stream_id: StreamId(11), flags: (0x4: END_HEADERS) }
dnstools-ricardo linkerd-proxy [   212.759017s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}: tower::balance::p2c::service: updating from discover
dnstools-ricardo linkerd-proxy [   212.759030s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}: linkerd_stack::failfast: HTTP Balancer service has become unavailable
dnstools-ricardo linkerd-proxy [   212.759046s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}: tower::balance::p2c::service: updating from discover
dnstools-ricardo linkerd-proxy [   212.759149s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:49318}:server{orig_dst=10.32.150.47:80}:profile:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.100.116.205:8086}:h2:Connection{peer=Client}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(11) }
dnstools-ricardo linkerd-proxy [   212.759189s]  WARN ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}: linkerd_proxy_api_resolve::pb: Ignoring invalid identity: flagr.default.svc.cluster.local:80
dnstools-ricardo linkerd-proxy [   212.759200s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}: linkerd_proxy_api_resolve::resolve: Add endpoints=1
dnstools-ricardo linkerd-proxy [   212.759238s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}: tower::balance::p2c::service: updating from discover
dnstools-ricardo linkerd-proxy [   212.759272s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}:endpoint{server.addr=172.22.16.152:4143}: linkerd_reconnect: Disconnected backoff=false
dnstools-ricardo linkerd-proxy [   212.759280s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}:endpoint{server.addr=172.22.16.152:4143}: linkerd_reconnect: Creating service backoff=false
dnstools-ricardo linkerd-proxy [   212.759288s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}:endpoint{server.addr=172.22.16.152:4143}: linkerd_proxy_http::client: Building HTTP client settings=OrigProtoUpgrade
dnstools-ricardo linkerd-proxy [   212.759301s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}:endpoint{server.addr=172.22.16.152:4143}: linkerd_tls::client: Peer does not support TLS reason=not_provided_by_service_discovery
dnstools-ricardo linkerd-proxy [   212.759309s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}:endpoint{server.addr=172.22.16.152:4143}: linkerd_proxy_transport::connect: Connecting server.addr=172.22.16.152:4143
dnstools-ricardo linkerd-proxy [   212.759500s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}: tower::balance::p2c::service: updating from discover
dnstools-ricardo linkerd-proxy [   212.760126s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}:endpoint{server.addr=172.22.16.152:4143}:h2: linkerd_proxy_transport::connect: Connected local.addr=10.100.96.246:38910 keepalive=Some(10s)
dnstools-ricardo linkerd-proxy [   212.760148s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}:endpoint{server.addr=172.22.16.152:4143}:h2: linkerd_transport_metrics::client: client connection open
dnstools-ricardo linkerd-proxy [   212.760161s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}:endpoint{server.addr=172.22.16.152:4143}:h2: h2::client: binding client connection
dnstools-ricardo linkerd-proxy [   212.760210s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}:endpoint{server.addr=172.22.16.152:4143}:h2: h2::client: client connection bound
dnstools-ricardo linkerd-proxy [   212.760218s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}:endpoint{server.addr=172.22.16.152:4143}:h2: h2::codec::framed_write: send frame=Settings { flags: (0x0), enable_push: 0, initial_window_size: 65535, max_frame_size: 16384 }
dnstools-ricardo linkerd-proxy [   212.760247s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}:endpoint{server.addr=172.22.16.152:4143}:h2:Connection{peer=Client}: h2::codec::framed_write: send frame=WindowUpdate { stream_id: StreamId(0), size_increment: 983041 }
dnstools-ricardo linkerd-proxy [   212.760299s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}: tower::balance::p2c::service: updating from discover
dnstools-ricardo linkerd-proxy [   212.760311s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}:endpoint{server.addr=172.22.16.152:4143}: linkerd_reconnect: Connected
dnstools-ricardo linkerd-proxy [   212.760326s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}: tower::balance::p2c::service: updating from discover
dnstools-ricardo linkerd-proxy [   212.760345s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}: tower::buffer::worker: service.ready=true processing request
dnstools-ricardo linkerd-proxy [   212.760357s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}:endpoint{server.addr=172.22.16.152:4143}: linkerd_proxy_http::override_authority: Stripped header header=host value="flagr-eks"
dnstools-ricardo linkerd-proxy [   212.760365s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}:endpoint{server.addr=172.22.16.152:4143}: linkerd_proxy_http::override_authority: Stripped header header=l5d-dst-canonical value="flagr-eks.default.svc.cluster.local:80"
dnstools-ricardo linkerd-proxy [   212.760375s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}:endpoint{server.addr=172.22.16.152:4143}: linkerd_proxy_http::override_authority: Overriding authority=linkerd-gateway.linkerd-multicluster.serviceaccount.identity.linkerd.cluster.local
dnstools-ricardo linkerd-proxy [   212.760389s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}:endpoint{server.addr=172.22.16.152:4143}:orig-proto-upgrade: linkerd_proxy_http::client: method=HEAD uri=http://linkerd-gateway.linkerd-multicluster.serviceaccount.identity.linkerd.cluster.local/ version=HTTP/1.1
dnstools-ricardo linkerd-proxy [   212.760397s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}:endpoint{server.addr=172.22.16.152:4143}:orig-proto-upgrade: linkerd_proxy_http::client: headers={"accept": "*/*", "user-agent": "curl/7.60.0"}
dnstools-ricardo linkerd-proxy [   212.760406s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}:endpoint{server.addr=172.22.16.152:4143}:orig-proto-upgrade: linkerd_proxy_http::orig_proto: Upgrading request version=HTTP/1.1 absolute_form=false
dnstools-ricardo linkerd-proxy [   212.760461s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}:endpoint{server.addr=172.22.16.152:4143}:h2:Connection{peer=Client}: h2::codec::framed_write: send frame=Headers { stream_id: StreamId(1), flags: (0x5: END_HEADERS | END_STREAM) }
dnstools-ricardo linkerd-proxy [   212.762178s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}:endpoint{server.addr=172.22.16.152:4143}:h2: hyper::proto::h2::client: client response error: broken pipe
dnstools-ricardo linkerd-proxy [   212.762225s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}:endpoint{server.addr=172.22.16.152:4143}:h2: linkerd_proxy_http::h2: failed error=connection error: broken pipe
dnstools-ricardo linkerd-proxy [   212.762262s]  INFO ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}:endpoint{server.addr=172.22.16.152:4143}:rescue{client.addr=10.100.96.246:58650}: linkerd_app_core::errors::respond: Request failed error=connection error: broken pipe
dnstools-ricardo linkerd-proxy [   212.762274s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}:logical{dst=flagr-eks.default.svc.cluster.local:80}:concrete{addr=flagr-eks.default.svc.cluster.local:80}:endpoint{server.addr=172.22.16.152:4143}: linkerd_app_core::errors::respond: Handling error on HTTP connection status=502 Bad Gateway version=HTTP/1.1 close=true
dnstools-ricardo linkerd-proxy [   212.762305s]  INFO ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}: linkerd_app_outbound::http::proxy_connection_close: Received unmeshed response with l5d-proxy-connection set
dnstools-ricardo linkerd-proxy [   212.762369s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}: hyper::proto::h1::io: flushed 133 bytes
dnstools-ricardo linkerd-proxy [   212.762414s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}:server{orig_dst=10.32.150.47:80}:profile:http{v=1.x}: linkerd_proxy_http::server: The client is shutting down the connection res=Ok(())
dnstools-ricardo linkerd-proxy [   212.762445s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:58650}: linkerd_app_core::serve: Connection closed
dnstools-ricardo linkerd-proxy [   212.762478s] DEBUG ThreadId(01) evict{target=Accept { orig_dst: OrigDstAddr(10.32.150.47:80), protocol: () }}: linkerd_cache: Awaiting idleness
dnstools-ricardo linkerd-proxy [   212.762500s] DEBUG ThreadId(01) evict{target=Logical { protocol: Http1, profile: .., logical_addr: LogicalAddr(flagr-eks.default.svc.cluster.local:80) }}: linkerd_cache: Awaiting idleness

The gateway in the source cluster only logs:

[    38.426210s] DEBUG ThreadId(01) inbound:accept{client.addr=172.22.13.199:25007}:server{port=4143}:direct: linkerd_tls::server: Reading bytes from TCP stream buf.capacity=8192
[    38.426246s]  INFO ThreadId(01) inbound:accept{client.addr=172.22.13.199:25007}: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated

for that given request and rejects it.

Regarding the trust root, we are definitely using the same ancho. I checked that again right now just to make sure that could be ruled out. Here is the linkerd mc check:

linkerd-multicluster
--------------------
√ Link CRD exists
√ Link resources are valid
	* eks
√ remote cluster access credentials are valid
	* eks
√ clusters share trust anchors
	* eks
√ service mirror controller has required permissions
	* eks
√ service mirror controllers are running
	* eks
√ all gateway mirrors are healthy
	* eks
√ all mirror services have endpoints
√ all mirror services are part of a Link
√ multicluster extension proxies are healthy
√ multicluster extension proxies are up-to-date
√ multicluster extension proxies and cli versions match

Status check results are √

[adleong]

One line that stands out to me in your logs is linkerd_proxy_api_resolve::pb: Ignoring invalid identity: flagr.default.svc.cluster.local:80

flagr.default.svc.cluster.local:80 is, indeed, not the correct identity string. Is it possible that you have the mirror.linkerd.io/gateway-identity set incorrectly on your gateway service? or the GatewayIdentity field is set incorrectly in your Link resource?

[RicardoNalesAmato]

Yes, that also looks off. The value I have set on both the Service and the Link is: gatewayIdentity: linkerd-gateway.linkerd-multicluster.serviceaccount.identity.linkerd.cluster.local and the service being called is flagr-eks via curl. Would anything else come to mind?

[RicardoNalesAmato]

Not sure if this would be helpful, but flagr.default.svc.cluster.local:80 Is the identity of the service flagr in the source cluster.

[RicardoNalesAmato]

This is the configuration for my Link:

apiVersion: multicluster.linkerd.io/v1alpha1
kind: Link
metadata:
    name: eks
    namespace: linkerd-multicluster
spec:
    clusterCredentialsSecret: cluster-credentials-eks
    gatewayAddress: <redacted>.elb.amazonaws.com
    gatewayIdentity: linkerd-gateway.linkerd-multicluster.serviceaccount.identity.linkerd.cluster.local
    gatewayPort: "4143"
    probeSpec:
      path: /ready
      period: 3
      port: "4191"
    selector:
      matchExpressions:
      - key: mirror.linkerd.io/exported
        operator: Exists
    targetClusterDomain: cluster.local
    targetClusterLinkerdNamespace: linkerd
    targetClusterName: eks

The service annotation mirror.linkerd.io/gateway-identity has the same value as gatewayIdentity in the Link. The configurations used are exactly the ones mentioned in the tutorial using the linkerd binary for all operations.

This is another request, this time to another service leading to the same result:

dnstools-ricardo linkerd-proxy [ 28552.829135s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}: linkerd_cache: Caching new service target=Accept { orig_dst: OrigDstAddr(10.32.244.218:9402), protocol: () }
dnstools-ricardo linkerd-proxy [ 28552.829235s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile: linkerd_app_outbound::discover: Allowing profile lookup
dnstools-ricardo linkerd-proxy [ 28552.829273s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}: linkerd_stack::failfast: TCP Server service has become unavailable
dnstools-ricardo linkerd-proxy [ 28552.829289s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: tower::balance::p2c::service: updating from discover
dnstools-ricardo linkerd-proxy [ 28552.829314s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: tower::buffer::worker: service.ready=true processing request
dnstools-ricardo linkerd-proxy [ 28552.829404s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:49318}:server{orig_dst=10.32.150.47:80}:profile:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.100.116.205:8086}:h2:Connection{peer=Client}: h2::codec::framed_write: send frame=Headers { stream_id: StreamId(49), flags: (0x4: END_HEADERS) }
dnstools-ricardo linkerd-proxy [ 28552.829449s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:49318}:server{orig_dst=10.32.150.47:80}:profile:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.100.116.205:8086}:h2:Connection{peer=Client}: h2::codec::framed_write: send frame=Data { stream_id: StreamId(49) }
dnstools-ricardo linkerd-proxy [ 28552.829458s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:49318}:server{orig_dst=10.32.150.47:80}:profile:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.100.116.205:8086}:h2:Connection{peer=Client}: h2::codec::framed_write: send frame=Data { stream_id: StreamId(49), flags: (0x1: END_STREAM) }
dnstools-ricardo linkerd-proxy [ 28552.833652s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:49318}:server{orig_dst=10.32.150.47:80}:profile:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.100.116.205:8086}:h2:Connection{peer=Client}: h2::codec::framed_read: received frame=Headers { stream_id: StreamId(49), flags: (0x4: END_HEADERS) }
dnstools-ricardo linkerd-proxy [ 28552.833700s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:49318}:server{orig_dst=10.32.150.47:80}:profile:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.100.116.205:8086}:h2:Connection{peer=Client}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(49) }
dnstools-ricardo linkerd-proxy [ 28552.833710s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:49318}:server{orig_dst=10.32.150.47:80}:profile:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.100.116.205:8086}:h2:Connection{peer=Client}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(49) }
dnstools-ricardo linkerd-proxy [ 28552.833715s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:49318}:server{orig_dst=10.32.150.47:80}:profile:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.100.116.205:8086}:h2:Connection{peer=Client}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(49) }
dnstools-ricardo linkerd-proxy [ 28552.833822s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}: tower::buffer::worker: service.ready=true processing request
dnstools-ricardo linkerd-proxy [ 28552.833858s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile: linkerd_detect: DetectResult protocol=Some(Http1) elapsed=11.87µs
dnstools-ricardo linkerd-proxy [ 28552.833873s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}: linkerd_proxy_http::server: Creating HTTP service version=Http1
dnstools-ricardo linkerd-proxy [ 28552.833885s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}: linkerd_cache: Caching new service target=Logical { protocol: Http1, profile: .., logical_addr: LogicalAddr(cert-manager-default-eks.kube-system.svc.cluster.local:9402) }
dnstools-ricardo linkerd-proxy [ 28552.833917s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}: linkerd_proxy_http::server: Handling as HTTP version=Http1
dnstools-ricardo linkerd-proxy [ 28552.833959s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}: hyper::proto::h1::io: parsed 3 headers
dnstools-ricardo linkerd-proxy [ 28552.833966s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}: hyper::proto::h1::conn: incoming body is empty
dnstools-ricardo linkerd-proxy [ 28552.834010s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}: linkerd_service_profiles::http::route_request: Updating HTTP routes routes=0
dnstools-ricardo linkerd-proxy [ 28552.834039s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}: tower::buffer::worker: service.ready=true processing request
dnstools-ricardo linkerd-proxy [ 28552.834056s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}: linkerd_service_profiles::split: Updating targets=[Target { addr: cert-manager-default-eks.kube-system.svc.cluster.local:9402, weight: 10000 }]
dnstools-ricardo linkerd-proxy [ 28552.834070s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:concrete{addr=cert-manager-default-eks.kube-system.svc.cluster.local:9402}: linkerd_proxy_api_resolve::resolve: Resolving dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402 context={"ns":"default", "nodeName":"ip-172-22-15-171.eu-west-1.compute.internal"}
dnstools-ricardo linkerd-proxy
dnstools-ricardo linkerd-proxy [ 28552.834137s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}: linkerd_stack::failfast: HTTP Logical service has become unavailable
dnstools-ricardo linkerd-proxy [ 28552.834176s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:concrete{addr=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: tower::balance::p2c::service: updating from discover
dnstools-ricardo linkerd-proxy [ 28552.834217s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:concrete{addr=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: tower::buffer::worker: service.ready=true processing request
dnstools-ricardo linkerd-proxy [ 28552.834280s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:49318}:server{orig_dst=10.32.150.47:80}:profile:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.100.181.246:8086}:h2:Connection{peer=Client}: h2::codec::framed_write: send frame=Headers { stream_id: StreamId(47), flags: (0x4: END_HEADERS) }
dnstools-ricardo linkerd-proxy [ 28552.834315s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:49318}:server{orig_dst=10.32.150.47:80}:profile:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.100.181.246:8086}:h2:Connection{peer=Client}: h2::codec::framed_write: send frame=Data { stream_id: StreamId(47) }
dnstools-ricardo linkerd-proxy [ 28552.834322s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:49318}:server{orig_dst=10.32.150.47:80}:profile:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.100.181.246:8086}:h2:Connection{peer=Client}: h2::codec::framed_write: send frame=Data { stream_id: StreamId(47), flags: (0x1: END_STREAM) }
dnstools-ricardo linkerd-proxy [ 28552.835942s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:49318}:server{orig_dst=10.32.150.47:80}:profile:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.100.181.246:8086}:h2:Connection{peer=Client}: h2::codec::framed_read: received frame=Headers { stream_id: StreamId(47), flags: (0x4: END_HEADERS) }
dnstools-ricardo linkerd-proxy [ 28552.835982s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:49318}:server{orig_dst=10.32.150.47:80}:profile:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.100.181.246:8086}:h2:Connection{peer=Client}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(47) }
dnstools-ricardo linkerd-proxy [ 28552.836060s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:concrete{addr=cert-manager-default-eks.kube-system.svc.cluster.local:9402}: tower::balance::p2c::service: updating from discover
dnstools-ricardo linkerd-proxy [ 28552.836075s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:concrete{addr=cert-manager-default-eks.kube-system.svc.cluster.local:9402}: linkerd_stack::failfast: HTTP Balancer service has become unavailable
dnstools-ricardo linkerd-proxy [ 28552.836109s]  WARN ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:concrete{addr=cert-manager-default-eks.kube-system.svc.cluster.local:9402}: linkerd_proxy_api_resolve::pb: Ignoring invalid identity: cert-manager-default.kube-system.svc.cluster.local:9402
dnstools-ricardo linkerd-proxy [ 28552.836119s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:concrete{addr=cert-manager-default-eks.kube-system.svc.cluster.local:9402}: linkerd_proxy_api_resolve::resolve: Add endpoints=1
dnstools-ricardo linkerd-proxy [ 28552.836159s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:concrete{addr=cert-manager-default-eks.kube-system.svc.cluster.local:9402}: tower::balance::p2c::service: updating from discover
dnstools-ricardo linkerd-proxy [ 28552.836178s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:concrete{addr=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:endpoint{server.addr=172.22.13.38:4143}: linkerd_reconnect: Disconnected backoff=false
dnstools-ricardo linkerd-proxy [ 28552.836185s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:concrete{addr=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:endpoint{server.addr=172.22.13.38:4143}: linkerd_reconnect: Creating service backoff=false
dnstools-ricardo linkerd-proxy [ 28552.836193s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:concrete{addr=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:endpoint{server.addr=172.22.13.38:4143}: linkerd_proxy_http::client: Building HTTP client settings=OrigProtoUpgrade
dnstools-ricardo linkerd-proxy [ 28552.836209s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:concrete{addr=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:endpoint{server.addr=172.22.13.38:4143}: linkerd_tls::client: Peer does not support TLS reason=not_provided_by_service_discovery
dnstools-ricardo linkerd-proxy [ 28552.836218s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:concrete{addr=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:endpoint{server.addr=172.22.13.38:4143}: linkerd_proxy_transport::connect: Connecting server.addr=172.22.13.38:4143
dnstools-ricardo linkerd-proxy [ 28552.836420s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:concrete{addr=cert-manager-default-eks.kube-system.svc.cluster.local:9402}: tower::balance::p2c::service: updating from discover
dnstools-ricardo linkerd-proxy [ 28552.836749s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:concrete{addr=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:endpoint{server.addr=172.22.13.38:4143}:h2: linkerd_proxy_transport::connect: Connected local.addr=10.100.96.246:36800 keepalive=Some(10s)
dnstools-ricardo linkerd-proxy [ 28552.836773s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:concrete{addr=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:endpoint{server.addr=172.22.13.38:4143}:h2: linkerd_transport_metrics::client: client connection open
dnstools-ricardo linkerd-proxy [ 28552.836784s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:concrete{addr=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:endpoint{server.addr=172.22.13.38:4143}:h2: h2::client: binding client connection
dnstools-ricardo linkerd-proxy [ 28552.836829s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:concrete{addr=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:endpoint{server.addr=172.22.13.38:4143}:h2: h2::client: client connection bound
dnstools-ricardo linkerd-proxy [ 28552.836855s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:concrete{addr=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:endpoint{server.addr=172.22.13.38:4143}:h2: h2::codec::framed_write: send frame=Settings { flags: (0x0), enable_push: 0, initial_window_size: 65535, max_frame_size: 16384 }
dnstools-ricardo linkerd-proxy [ 28552.836892s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:concrete{addr=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:endpoint{server.addr=172.22.13.38:4143}:h2:Connection{peer=Client}: h2::codec::framed_write: send frame=WindowUpdate { stream_id: StreamId(0), size_increment: 983041 }
dnstools-ricardo linkerd-proxy [ 28552.836947s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:concrete{addr=cert-manager-default-eks.kube-system.svc.cluster.local:9402}: tower::balance::p2c::service: updating from discover
dnstools-ricardo linkerd-proxy [ 28552.836979s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:concrete{addr=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:endpoint{server.addr=172.22.13.38:4143}: linkerd_reconnect: Connected
dnstools-ricardo linkerd-proxy [ 28552.836997s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:concrete{addr=cert-manager-default-eks.kube-system.svc.cluster.local:9402}: tower::balance::p2c::service: updating from discover
dnstools-ricardo linkerd-proxy [ 28552.837016s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}: tower::buffer::worker: service.ready=true processing request
dnstools-ricardo linkerd-proxy [ 28552.837029s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:concrete{addr=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:endpoint{server.addr=172.22.13.38:4143}: linkerd_proxy_http::override_authority: Stripped header header=host value="cert-manager-default-eks.kube-system:9402"
dnstools-ricardo linkerd-proxy [ 28552.837037s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:concrete{addr=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:endpoint{server.addr=172.22.13.38:4143}: linkerd_proxy_http::override_authority: Stripped header header=l5d-dst-canonical value="cert-manager-default-eks.kube-system.svc.cluster.local:9402"
dnstools-ricardo linkerd-proxy [ 28552.837043s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:concrete{addr=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:endpoint{server.addr=172.22.13.38:4143}: linkerd_proxy_http::override_authority: Overriding authority=linkerd-gateway.linkerd-multicluster.serviceaccount.identity.linkerd.cluster.local
dnstools-ricardo linkerd-proxy [ 28552.837058s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:concrete{addr=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:endpoint{server.addr=172.22.13.38:4143}:orig-proto-upgrade: linkerd_proxy_http::client: method=HEAD uri=http://linkerd-gateway.linkerd-multicluster.serviceaccount.identity.linkerd.cluster.local/ version=HTTP/1.1
dnstools-ricardo linkerd-proxy [ 28552.837068s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:concrete{addr=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:endpoint{server.addr=172.22.13.38:4143}:orig-proto-upgrade: linkerd_proxy_http::client: headers={"accept": "*/*", "user-agent": "curl/7.60.0"}
dnstools-ricardo linkerd-proxy [ 28552.837080s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:concrete{addr=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:endpoint{server.addr=172.22.13.38:4143}:orig-proto-upgrade: linkerd_proxy_http::orig_proto: Upgrading request version=HTTP/1.1 absolute_form=false
dnstools-ricardo linkerd-proxy [ 28552.837144s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:concrete{addr=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:endpoint{server.addr=172.22.13.38:4143}:h2:Connection{peer=Client}: h2::codec::framed_write: send frame=Headers { stream_id: StreamId(1), flags: (0x5: END_HEADERS | END_STREAM) }
dnstools-ricardo linkerd-proxy [ 28552.838261s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:concrete{addr=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:endpoint{server.addr=172.22.13.38:4143}:h2: hyper::proto::h2::client: client response error: broken pipe
dnstools-ricardo linkerd-proxy [ 28552.838309s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:concrete{addr=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:endpoint{server.addr=172.22.13.38:4143}:h2: linkerd_proxy_http::h2: failed error=connection error: broken pipe
dnstools-ricardo linkerd-proxy [ 28552.838337s]  INFO ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:concrete{addr=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:endpoint{server.addr=172.22.13.38:4143}:rescue{client.addr=10.100.96.246:57648}: linkerd_app_core::errors::respond: Request failed error=connection error: broken pipe
dnstools-ricardo linkerd-proxy [ 28552.838368s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}:logical{dst=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:concrete{addr=cert-manager-default-eks.kube-system.svc.cluster.local:9402}:endpoint{server.addr=172.22.13.38:4143}: linkerd_app_core::errors::respond: Handling error on HTTP connection status=502 Bad Gateway version=HTTP/1.1 close=true
dnstools-ricardo linkerd-proxy [ 28552.838397s]  INFO ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}: linkerd_app_outbound::http::proxy_connection_close: Received unmeshed response with l5d-proxy-connection set
dnstools-ricardo linkerd-proxy [ 28552.838581s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}: hyper::proto::h1::io: flushed 133 bytes
dnstools-ricardo linkerd-proxy [ 28552.838662s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}:server{orig_dst=10.32.244.218:9402}:profile:http{v=1.x}: linkerd_proxy_http::server: The client is shutting down the connection res=Ok(())
dnstools-ricardo linkerd-proxy [ 28552.838723s] DEBUG ThreadId(01) outbound:accept{client.addr=10.100.96.246:57648}: linkerd_app_core::serve: Connection closed
dnstools-ricardo linkerd-proxy [ 28552.838748s] DEBUG ThreadId(01) evict{target=Accept { orig_dst: OrigDstAddr(10.32.244.218:9402), protocol: () }}: linkerd_cache: Awaiting idleness
dnstools-ricardo linkerd-proxy [ 28552.838757s] DEBUG ThreadId(01) evict{target=Logical { protocol: Http1, profile: .., logical_addr: LogicalAddr(cert-manager-default-eks.kube-system.svc.cluster.local:9402) }}: linkerd_cache: Awaiting idleness

[adleong]

Hmmmm I'm having a hard time figuring out how this is possible. Based on the logs, it seems like the destination service is returning an invalid identity (cert-manager-default.kube-system.svc.cluster.local:9402) but it's unclear to me what would cause it do this. This value should always be populated from the mirror.linkerd.io/remote-gateway-identity annotation on the mirror endpoint.

Furthermore, I can't reproduce this behavior by following the multicluster guide. If you can provide more detailed step-by-step instructions for reproducing this behavior (or even better, some automated scripts which demonstrate the issue) I will be able to dig in further. Otherwise, I'm not sure how to help.

[RicardoNalesAmato]

Thanks for looking into this! I will make sure to do that so you can reproduce it. In the meantime these are the logs from the destination service when a request to flagr-eks is made:

linkerd-destination-59f78878c6-h7df4 destination time="2022-01-20T00:30:19Z" level=debug msg="Get flagr-eks.default.svc.cluster.local:80" addr=":8086" component=server remote="10.100.60.2:54722"
linkerd-destination-59f78878c6-rzqdp destination time="2022-01-20T00:30:19Z" level=debug msg="Establishing watch on service default/flagr-eks" addr=":8086" component=traffic-split-watcher
linkerd-destination-59f78878c6-rzqdp destination time="2022-01-20T00:30:19Z" level=debug msg="Starting watch on service default/flagr-eks" addr=":8086" component=opaque-ports-watcher
linkerd-destination-59f78878c6-h7df4 destination time="2022-01-20T00:30:19Z" level=debug msg="Establishing watch on endpoint [default/flagr-eks:80]" addr=":8086" component=endpoints-watcher
linkerd-destination-59f78878c6-rzqdp destination time="2022-01-20T00:30:19Z" level=debug msg="Sending profile update: fully_qualified_name:\"flagr-eks.default.svc.cluster.local\"  retry_budget:{retry_ratio:0.2  min_retries_per_second:10  ttl:{seconds:10}}  dst_overrides:{authority:\"flagr-eks.default.svc.cluster.local.:80\"  weight:10000}" addr=":8086" component=profile-translator remote="10.100.124.179:59818"
linkerd-destination-59f78878c6-rzqdp destination time="2022-01-20T00:30:19Z" level=debug msg="Establishing watch on profile default/flagr-eks.default.svc.cluster.local" addr=":8086" component=profile-watcher
linkerd-destination-59f78878c6-h7df4 destination time="2022-01-20T00:30:19Z" level=debug msg="Sending destination add: add:{addrs:{addr:{ip:{ipv4:2887127192}  port:4143}  weight:10000  tls_identity:{dns_like_identity:{name:\"flagr.default.svc.cluster.local:80\"}}  protocol_hint:{h2:{}}  authority_override:{authority_override:\"linkerd-gateway.linkerd-multicluster.serviceaccount.identity.linkerd.cluster.local\"}}  metric_labels:{key:\"namespace\"  value:\"default\"}  metric_labels:{key:\"service\"  value:\"flagr-eks\"}  metric_labels:{key:\"target_cluster\"  value:\"eks\"}  metric_labels:{key:\"target_service\"  value:\"flagr\"}  metric_labels:{key:\"target_service_namespace\"  value:\"default\"}}" addr=":8086" component=endpoint-translator remote="10.100.60.2:54722" service="flagr-eks.default.svc.cluster.local:80"
linkerd-destination-59f78878c6-rzqdp destination time="2022-01-20T00:30:19Z" level=debug msg="Sending profile update: fully_qualified_name:\"flagr-eks.default.svc.cluster.local\"  retry_budget:{retry_ratio:0.2  min_retries_per_second:10  ttl:{seconds:10}}  dst_overrides:{authority:\"flagr-eks.default.svc.cluster.local.:80\"  weight:10000}" addr=":8086" component=profile-translator remote="10.100.124.179:59818"
linkerd-destination-59f78878c6-rzqdp destination time="2022-01-20T00:30:19Z" level=debug msg="Establishing watch on profile default/flagr-eks.default.svc.cluster.local" addr=":8086" component=profile-watcher
linkerd-destination-59f78878c6-rzqdp destination time="2022-01-20T00:30:19Z" level=debug msg="Sending profile update: fully_qualified_name:\"flagr-eks.default.svc.cluster.local\"  retry_budget:{retry_ratio:0.2  min_retries_per_second:10  ttl:{seconds:10}}  dst_overrides:{authority:\"flagr-eks.default.svc.cluster.local.:80\"  weight:10000}" addr=":8086" component=profile-translator remote="10.100.124.179:59818"
linkerd-destination-59f78878c6-rzqdp destination time="2022-01-20T00:30:20Z" level=debug msg="Updating EndpointSlice for default/flagr-eks" addr=":8086" component=service-publisher ns=default svc=flagr-eks
linkerd-destination-59f78878c6-h7df4 destination time="2022-01-20T00:30:20Z" level=debug msg="Updating EndpointSlice for default/flagr-eks" addr=":8086" component=service-publisher ns=default svc=flagr-eks
linkerd-destination-59f78878c6-h7df4 destination time="2022-01-20T00:30:20Z" level=debug msg="Sending destination remove: remove:{addrs:{ip:{ipv4:2887127192}  port:4143}}" addr=":8086" component=endpoint-translator remote="10.100.60.2:54722" service="flagr-eks.default.svc.cluster.local:80"
linkerd-destination-59f78878c6-h7df4 destination time="2022-01-20T00:30:20Z" level=debug msg="Sending destination add: add:{addrs:{addr:{ip:{ipv4:2887128646}  port:4143}  weight:10000  tls_identity:{dns_like_identity:{name:\"flagr.default.svc.cluster.local:80\"}}  protocol_hint:{h2:{}}  authority_override:{authority_override:\"linkerd-gateway.linkerd-multicluster.serviceaccount.identity.linkerd.cluster.local\"}}  metric_labels:{key:\"namespace\"  value:\"default\"}  metric_labels:{key:\"service\"  value:\"flagr-eks\"}  metric_labels:{key:\"target_cluster\"  value:\"eks\"}  metric_labels:{key:\"target_service\"  value:\"flagr\"}  metric_labels:{key:\"target_service_namespace\"  value:\"default\"}}" addr=":8086" component=endpoint-translator remote="10.100.60.2:54722" service="flagr-eks.default.svc.cluster.local:80"
^[linkerd-destination-59f78878c6-h7df4 destination time="2022-01-20T00:30:24Z" level=debug msg="Get flagr-eks.default.svc.cluster.local:80 cancelled" addr=":8086" component=server remote="10.100.60.2:54722"
linkerd-destination-59f78878c6-h7df4 destination time="2022-01-20T00:30:24Z" level=debug msg="Stopping watch on endpoint [default/flagr-eks:80]" addr=":8086" component=endpoints-watcher
linkerd-destination-59f78878c6-rzqdp destination time="2022-01-20T00:30:24Z" level=debug msg="Stopping watch on profile default/flagr-eks.default.svc.cluster.local" addr=":8086" component=profile-watcher
linkerd-destination-59f78878c6-rzqdp destination time="2022-01-20T00:30:24Z" level=debug msg="Stopping watch on profile default/flagr-eks.default.svc.cluster.local" addr=":8086" component=profile-watcher
linkerd-destination-59f78878c6-rzqdp destination time="2022-01-20T00:30:24Z" level=debug msg="Stopping watch on service default/flagr-eks" addr=":8086" component=opaque-ports-watcher
linkerd-destination-59f78878c6-rzqdp destination time="2022-01-20T00:30:24Z" level=debug msg="Stopping watch on profile default/flagr-eks" addr=":8086" component=traffic-split-watcher

[RicardoNalesAmato]

@adleong I found what the issue is after much debugging: We had the enableEndpointSlices flag enabled which led to the error mentioned in this discussion. If you install linkerd2 via helm with that flag you should be able to reproduce it without a problem (I used K8s 1.20.X). Disabling that completely fixed the issue.

[adleong]

Great find! I'm glad this is working now. It would be super helpful if you could provide the steps to configure the cluster in this way so that we can write tests for this scenario and prevent other users from hitting this. If you're able to open an issue describing the steps, that would be much appreciated!

[RicardoNalesAmato]

@adleong Thanks a lot for all the help!