Empty reply from mirrored Service in multicluster setup

[shotgunner]

Hi. I followed the setup on official multicluster docs. I have two GKE clusters and deployed simple http server in east cluster and two clients one in west and another in east with this yaml files:

east-deployment.yml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello
  template:
    metadata:
      labels:
        app: hello
    spec:
      containers:
      - name: hello
        image: hashicorp/http-echo
        args:
        - "-text=Hello, World"
        ports:
        - containerPort: 5678
---
apiVersion: v1
kind: Service
metadata:
  name: hello
spec:
  ports:
  - port: 5678
    targetPort: 5678
  selector:
    app: hello

west-deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: client
spec:
  replicas: 1
  selector:
    matchLabels:
      app: client
  template:
    metadata:
      labels:
        app: client
    spec:
      containers:
      - name: client
        image: appropriate/curl
        command: ["/bin/sh"]
        args:
        - "-c"
        - |
          POD_NAME=$(hostname)
          NODE_NAME=$(kubectl get pod $POD_NAME -o jsonpath='{.spec.nodeName}')
          while true; do
            echo "Instance: $POD_NAME, Node: $NODE_NAME"
            curl -s hello:5678
            sleep 5
          done
        ports:
        - containerPort: 80

here is summary of pods and services in both clusters:

east

$ kubectl --context=east get svc
NAME         TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE
hello        ClusterIP   10.6.104.14   <none>        5678/TCP   45m
kubernetes   ClusterIP   10.6.0.1      <none>        443/TCP    23h

$ kubectl --context=east get pods
NAME                      READY   STATUS    RESTARTS   AGE
client-59fd7bc956-lbv4t   1/1     Running   0          45m
hello-5988d85c9b-ck885    1/1     Running   0          45m

west

$ kubectl --context=west get svc
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
kubernetes   ClusterIP   10.9.0.1       <none>        443/TCP    23h

$ kubectl --context=west get pods
NAME                      READY   STATUS    RESTARTS      AGE
client-59fd7bc956-8lkd9   1/1     Running   0             38m

My linkerd version is:

$ linkerd version
Client version: edge-24.8.2
Server version: edge-24.8.2

here is multicluster check for west:

$ linkerd --context=west multicluster check
linkerd-multicluster
--------------------
√ Link CRD exists
√ Link resources are valid
        * east
√ remote cluster access credentials are valid
        * east
√ clusters share trust anchors
        * east
√ service mirror controller has required permissions
        * east
√ service mirror controllers are running
        * east
√ probe services able to communicate with all gateway mirrors
        * east
√ all mirror services have endpoints
√ all mirror services are part of a Link
√ multicluster extension proxies are healthy
√ multicluster extension proxies are up-to-date
√ multicluster extension proxies and cli versions match

Status check results are √

and here the gateways:

$ linkerd --context=west multicluster gateways
CLUSTER  ALIVE    NUM_SVC      LATENCY  
east     True           1          4ms  

What is the Issue ?

After exporting east hello server to west with this command:

$ kubectl --context=east label svc hello mirror.linkerd.io/exported=true 

I can see mirrored service in west now:

$ kubectl --context=west get svc
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
hello-east   ClusterIP   10.9.248.104   <none>        5678/TCP   19m   # This one added and didn't exist before
kubernetes   ClusterIP   10.9.0.1       <none>        443/TCP    23h

$ kubectl --context=east get svc
NAME         TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE
hello        ClusterIP   10.6.104.14   <none>        5678/TCP   26m
kubernetes   ClusterIP   10.6.0.1      <none>        443/TCP    23h

I am able to curl that server in east client that is obvious:

$ kubectl --context=east exec --stdin --tty client-59fd7bc956-lbv4t -- /bin/sh 
kubectl --context=east exec --stdin --tty client-59fd7bc956-lbv4t -- /bin/sh 
/ # curl hello:5678
Hello, World

/ # curl hello-east:5678
curl: (6) Could not resolve host: hello-east

/ # nslookup hello
nslookup: can't resolve '(null)': Name does not resolve

Name:      hello
Address 1: 10.6.104.14 hello.default.svc.cluster.local

/ # nslookup hello-east
nslookup: can't resolve '(null)': Name does not resolve

nslookup: can't resolve 'hello-east': Name does not resolve

but when I try to do the same curl on west client I get curl: (52) Empty reply from server instead of get the response :Hello, World:

$ kubectl --context=west exec --stdin --tty client-59fd7bc956-8lkd9 -- /bin/sh
/ # curl hello:5678
curl: (6) Could not resolve host: hello

/ # curl hello-east:5678
curl: (52) Empty reply from server

/ # nslookup hello
nslookup: can't resolve '(null)': Name does not resolve

nslookup: can't resolve 'hello': Name does not resolve

/ # nslookup hello-east
nslookup: can't resolve '(null)': Name does not resolve

Name:      hello-east
Address 1: 10.9.248.104 hello-east.default.svc.cluster.local

Question

What I've missed here? It seems packet is dropped somewhere when mirrored service trying to connect to the real one. if yes why and how can I resolve this issue ?

[shotgunner]

It seems I forgot to inject linkerd to client. after inject it was works like a charm !!!
I just see notices this post after a week search about it: #7790

thank you guys