Using linkerd-destination for service discovery

[rejmond]

I am looking into service discovery process and observe a behavior that doesn't correlates with the documentation.

According to the documentation Linkerd should use its destination component from the control plane for service discovery and ask K8S DNS if control plane is down. However I see DNS requests at the very beginning of client pod lifecycle.

I use the emojivoto demo app from the documenation as an example and trace vote-bot to web pods requests.

Pods and services IP from the demo app and the control plane namespace:

# kubectl get pod -n linkerd -o wide
NAME                                      READY   STATUS    RESTARTS      AGE   IP           NODE           NOMINATED NODE   READINESS GATES  
linkerd-destination-5d979897dd-vsws9      4/4     Running   0             20m   10.244.1.4   kind-worker2   <none>           <none>  
linkerd-identity-85cdbddfd4-fhxmg         2/2     Running   0             20m   10.244.1.2   kind-worker2   <none>           <none>  
linkerd-proxy-injector-66fd47c7f6-zqv29   2/2     Running   1 (18m ago)   20m   10.244.1.3   kind-worker2   <none>           <none>

# kubectl get svc -n linkerd -o wide
NAME                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE   SELECTOR  
linkerd-dst                 ClusterIP   10.96.235.249   <none>        8086/TCP   20m   linkerd.io/control-plane-component=destination  
linkerd-dst-headless        ClusterIP   None            <none>        8086/TCP   20m   linkerd.io/control-plane-component=destination  
linkerd-identity            ClusterIP   10.96.46.79     <none>        8080/TCP   20m   linkerd.io/control-plane-component=identity  
linkerd-identity-headless   ClusterIP   None            <none>        8080/TCP   20m   linkerd.io/control-plane-component=identity  
linkerd-policy              ClusterIP   None            <none>        8090/TCP   20m   linkerd.io/control-plane-component=destination  
linkerd-policy-validator    ClusterIP   10.96.240.99    <none>        443/TCP    20m   linkerd.io/control-plane-component=destination  
linkerd-proxy-injector      ClusterIP   10.96.212.100   <none>        443/TCP    20m   linkerd.io/control-plane-component=proxy-injector  
linkerd-sp-validator        ClusterIP   10.96.7.228     <none>        443/TCP    20m   linkerd.io/control-plane-component=destination

# kubectl get svc -n emojivoto -o wide
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE     SELECTOR  
emoji-svc    ClusterIP   10.96.200.42   <none>        8080/TCP,8801/TCP   2m41s   app=emoji-svc
voting-svc   ClusterIP   10.96.226.84   <none>        8080/TCP,8801/TCP   2m41s   app=voting-svc
web-svc      ClusterIP   10.96.134.67   <none>        80/TCP              2m41s   app=web-svc

# kubectl get pods -n emojivoto -o wide
NAME                        READY   STATUS    RESTARTS   AGE     IP           NODE           NOMINATED NODE   READINESS GATES  
emoji-58c4547bf4-gltgl      3/3     Running   0          2m49s   10.244.2.3   kind-worker    <none>           <none>  
vote-bot-644d49d4d7-vgqkc   3/3     Running   0          2m49s   10.244.2.4   kind-worker    <none>           <none>  
voting-c785f488f-jc76z      3/3     Running   0          2m49s   10.244.2.5   kind-worker    <none>           <none>  
web-796567f465-xvkkf        3/3     Running   0          2m49s   10.244.1.5   kind-worker2   <none>           <none>

# kubectl get svc -n kube-system -o wide
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE   SELECTOR  
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   23m   k8s-app=kube-dns

Logs from the injected linkerd-debug container:

  224 7.275323178   10.244.2.4 → 10.96.0.10   DNS 107 Standard query 0x3e49 AAAA web-svc.emojivoto.emojivoto.svc.cluster.local
  225 7.275323172   10.244.2.4 → 10.96.0.10   DNS 107 Standard query 0xd2f0 A web-svc.emojivoto.emojivoto.svc.cluster.local
  226 7.276029946   10.96.0.10 → 10.244.2.4   DNS 200 Standard query response 0xd2f0 No such name A web-svc.emojivoto.emojivoto.svc.cluster.local SOA ns.dns.cluster.local
  227 7.276161687   10.96.0.10 → 10.244.2.4   DNS 200 Standard query response 0x3e49 No such name AAAA web-svc.emojivoto.emojivoto.svc.cluster.local SOA ns.dns.cluster.local
  228 7.276401243   10.244.2.4 → 10.96.0.10   DNS 97 Standard query 0xb89a AAAA web-svc.emojivoto.svc.cluster.local
  229 7.276464943   10.244.2.4 → 10.96.0.10   DNS 97 Standard query 0x9e18 A web-svc.emojivoto.svc.cluster.local
  230 7.276898359   10.96.0.10 → 10.244.2.4   DNS 148 Standard query response 0x9e18 A web-svc.emojivoto.svc.cluster.local A 10.96.134.67
  231 7.276957218   10.96.0.10 → 10.244.2.4   DNS 190 Standard query response 0xb89a AAAA web-svc.emojivoto.svc.cluster.local SOA ns.dns.cluster.local

Full log file: debug.logs.txt

I see multiple requests to kube-dns service IP for resolving web-svc service IP. There are a few requests to linkerd-destination pod IP, but they happens when service IP is already resolved. Also I don't see /Get gRPC request to linkerd-destination at the VIZ dashboard or using linkerd tap command.

How can I check that Linkerd proxies are really connected to the control plane?

Thank you!

[mateiidavid]

Hey @rejmond, the main container will resolve the DNS target before the proxy intercepts any packet. For example, consider two containers (each in a separate pod): foo and bar. If foo tries to connect to bar through a clusterIP services (e.g bar-svc:8080), then whatever http client implementation foo uses will resolve the host to the clusterIP before a connection is attempted. Once the packets are sent out, netfilter will re-write their destination using DNAT to go through the proxy, which will do a look-up on the target and resolve endpoints. It does so using the destination service.

The proxy in its current form resolves DNS only when communicating with the control plane. In order for the proxy to start up, it must first acquire its identity from the control plane's identity service -- it's safe to assume that if the proxy starts up, it has managed to connect to the control plane.

There are a few requests to linkerd-destination pod IP, but they happens when service IP is already resolved

Because the main (app) container will resolve it first. If you want to see responses from the control plane, you can turn the log level up on the proxies and look for profile responses, or for the first few log lines when the proxy connects to the identity service.