linkerd + externalTrafficPolicy: Local + skip-inbound-ports = wrong X-Real-Ip?

[sebadob]

Hey together,

I have a small problem I am facing and I cannot make it work, I am a bit out of ideas atm.

The situation:

• I have a small K3s cluster
• Traefik 2.5 for Ingress with Ingressroutes and CRDs
• Metallb in L2 mode
• Calico as CNI
• and of course linkerd

I am running 2 instances of Traefik with externalTrafficPolicy: Local to have at least automatic failover, because with Local policy there is no load balancing.
I need the X-Real-Ip in some situations in the backend.

When linkerd is not injected into the Traefik pods, everythings works fine and as expected, but of course I lose all my beloved linkerd features.

The problem:
As soon as I inject linkerd, the X-Real-Ip in the Traefik access logs changes to the Pods cluster IP and therefore of course everything else that comes afterwards.

What I tried out so far:

• just linkerd.io/inject: enabled as well as linkerd.io/inject: ingress + config.linkerd.io/skip-outbound-ports: 443,(because Traefik talks to the API server over 443)
• config.linkerd.io/skip-inbound-ports: 80, 443, 8000, 8443
• Changing the log output to json instead of CLF and validate, that the X-Real-IP Header has really changed
• adding the linkerd-debug proxy sidecar - interesting here is, that it shows my correct real IP, when I access the API, but as soon as the traffic gets forwarded to Traefik inside the same pod, it only sees the pods Cluster IP again
• When I change to externalTrafficPolicy: Cluster, it behaves like expected - Load Balancing between my 2 Traefik Pods and I only see the Cluster IP, even in the debug sidecar
• When I change to externalTrafficPolicy: Local though, it behaves like described above - Load Balancing between my 2 Traefik Pods is not working anymore (like expected), I see the real IP in the debug sidecar but not afterwards anymore.

There is one thing left, that might cause a problem.
Traefik is running internally on ports 8000 and 8443, but it's exposing 80 and 443. I am not sure if this is a problem. I really don't want to run Traefik inside the container as root and I want / need 80 and 443 from the outside.

I hope that someone has maybe an idea how I can solve this.