Linkerd Routes - Why does it not work in this case?

[kitkars]

Check this simple yaml. It receives traffic from ingress (ingress namespace) -> deployment (default namespace). You could try this yaml file. (Take a look at the ingress section and you might want to update site name)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - name: nginx-port
          containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  labels:
    app: nginx
spec:
  type: ClusterIP
  selector:
    app: nginx
  ports:
    - port: 80
      targetPort: nginx-port
      protocol: TCP
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port;
      grpc_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port;
spec:
  rules:
  - host: mysite.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx-service
          servicePort: 80

Tap & Top work fine.
But Routes section does not show any info at all. Is this expected.? My linkerd version is 2.9.0

[adleong]

In order to get per-route metrics in Linkerd, you need to create a service profile: https://linkerd.io/2/tasks/getting-per-route-metrics/

[kitkars]

@adleong

Hi, I am not talking about per-route metrics. I am talking about DEFAULT metrics. I do not think I need a service profile. I have attached a simple yaml file as well to reproduce yourself.

[adleong]

Ah, I see what you mean. I'm assuming that your nginx deployment receives TLS traffic from the outside world. This means that the Linkerd proxy is not able to decrypt this traffic and therefore it proxies it transparently and so we don't have any HTTP metrics for traffic coming into nginx. You should still have metrics from traffic outbound from nginx to services in your cluster.

[kitkars]

@adleong

outside world --> ingress --> first pod

Can I assume that the very first pod receives a request from the outside world via ingress will not show any metrics?
Because I do not use TLS. Even without nginx - I also tried with simple spring boot API which receives traffic from ingress does not show any metric.

It basically makes me think that any pod irrespective of TLS enabled or not which receives traffic from ingress will NOT show any metric in the routes section. But as you say outbound metrics are shown for the pod.

[adleong]

If your nginx ingress is correctly setting the l5d-dst-override header then you should see route metrics for that first pod that the ingress sends to. This makes me suspect that maybe something is going wrong with your ingress setting this header.

There are a few ways to confirm this: if you run linkerd metrics targeting your first pod, you should see a metric called route_response_total with a label dst=nginx-service.default.svc.cluster.local:80 which increments for each request.

You can also run linkerd tap -o json targeting your first pod. In each request you should see a "headers" which contains this header:

      {
        "name": "l5d-dst-canonical",
        "valueStr": "nginx-service.default.svc.cluster.local:80"
      }

If those are absent then I recommend double checking your ingress configuration. Is your ingress injected in ingress mode?

[kitkars]

@adleong
I have already attached the yaml file. Can you please check that if you do not mind? Am i missing anything there?
Can you please try once and confirm if you are not able to replicate?

If you are trying to replicate, Please DO NOT USE kubectl port-forward. instead use ingress as I had mentioned.

[cpretzer]

@kitkars when you are sending the traffic to the ingress, what tool are you using to do that?

I was able to reproduce the behavior using curl and an ingress with no host setting.

Once I added the host value to the ingress and specified the host header in my curl command, the routes metrics began to display.

In the yaml that you sent; the ``hostkey is set tomysite.example.com`:

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port;
      grpc_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port;
spec:
  rules:
  - host: mysite.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx-service
          servicePort: 80

Can you test whether the metrics show up when you send traffic with the appropriate host header? For curl, that would look like:

curl -h "host: mysite.example.com" http://<ingress-ip-address/

[adleong]

I attempted to reproduce this by installing Linkerd edge-20.12.3 and the Nginx ingress controller and injecting it in ingress mode. Then I piped your yamls above through linkerd inject and applied them.

I port-forwarded from my computer to svc/ingress-nginx-controller and sent some test requests like this:

curl localhost:8080 -H "Host: mysite.example.com"

These were correctly routed to the nginx-service. After sending the test requests, I saw route data in both the CLI and dashboard:

linkerd routes deploy
ROUTE             SERVICE   SUCCESS      RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99
[DEFAULT]   nginx-service   100.00%   0.3rps           1ms           1ms           1ms

[kitkars]

@adleong @cpretzer

This is how I set up. (Please I had explicitly mentioned earlier in my comment that it works fine when you use port-forward. Please ingore that option. This is more of ingress specific issue. It does not really bother me much. But if you think it is an issue, please fix it).

I have updated /etc/host like this - the ip is k8s cluster ip for ingress.

10.11.12.13 mysite.example.com

Now try to send to request to the pod via ingress via browser.

I see the Host header is appended by the browser itself when it sends a request. But it is HTTP not HTTPS. does it matter?

[adleong]

@kitkars the one thing I don't see in your yaml above is the deployment for your ingress controller. Make sure that your ingress controller is injected in "ingress" mode: https://linkerd.io/2/tasks/using-ingress/#proxy-ingress-mode

[kitkars]

@adleong @cpretzer I was travelling. could not get back to you guys immediately. Thanks for helping.

[cpretzer]

@kitkars thanks for following up on this and letting us know that injecting the proxy in ingress mode answered the question