Linkerd External Traffic Split with Nginx ingress returns 503

[arunalakmal]

Hi

I was working on a Linkerd implementation and I wanted to use external traffic split with Nginx ingress controller.

My Nginx controller was deployed as a DeamonSet and, Linkerd proxy mode was injected as below
kubectl get ds nginx-ingress-controller-ingress-nginx-controller -n nginx -o yaml | linkerd inject --ingress - | kubectl apply -f -

Linkerd dashboard is showing the deployed pods as meshed as below

I have also deployed these two emojivoto services and web-apex service along with the traffic-split (500m each) as described here. I can confirm that the traffic split is perfectly working within the cluster and I wanted to implement this blue green/Canery implementation with the external traffic and traffic-split. I have created an ingress resource as below to expose the web-apex service

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web-apex-ingress
  namespace: emojivoto
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port;
      grpc_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port;
spec:
  rules:
  - host: apex.dumb.cloud
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: web-apex
            port:
              number: 80

Please note that domain name is changed

If I expose these services individually with two different Ingresses those can be accessed through the web browser even without the annotation in the ingress resource.

Here is the annotations of the Nginx controller which have the details of Linkerd versions, etc :

Here is the output of services and ingress resource

Here is the output of traffic Split configuration:

But when I access it through the browser or curl with the url it is returning 503 service unavailable message from the Nginx controller. It seems that Nginx controller is not aware about the mesh and it can't find the endpoints which doesn't have for the web-apex service.

I went through the previous discussions and those didn't help me to fix this. I'm pretty sure this is something I misconfigured and I look for your support and advice to get this completed with the ingress traffic.

[arunalakmal]

I have raised this question in the Linkerd Slack channel and got this fixed with the wonderful support from the community. Seems Nginx doesn't like the service which doesn't have an endpoint. My configuration was correct and asked to change the service pointed in the traffic split to a service with an endpoint and it fixed the issue.

In a nutshell, my traffic split was configured with web-svc and web-svc-2 services. I have changed the traffic split spec.service to the same web-svc and it worked

Here is the traffic split configuration after the update.

apiVersion: split.smi-spec.io/v1alpha1
kind: TrafficSplit
metadata:
  name: web-svc-ts
  namespace: emojivoto
spec:
  # The root service that clients use to connect to the destination application.
  service: web-svc
  # Services inside the namespace with their own selectors, endpoints and configuration.
  backends:
  - service: web-svc
    # Identical to resources, 1 = 1000m
    weight: 500m
  - service: web-svc-2
    weight: 500m

Again thanks for the wonderful support 🙏🏽

[cpretzer]

thanks for the update @arunalakmal