Linkerd still crushes

[kannam12]

Hi all,
I'm trying to install linkerd.

• It seemed to be all ok, after I run linkerd check --pre, but it took really long for linkerd destination and proxy-injector to stand up (it got stuck on PodInitalizing for ca. 20 minutes.
• Then after 7x linkerd heartbeat it managed to become Running.
• I installed linkerd viz, all pods are also Running , but linkerd viz check fails at viz extension self-check.
• After ca. 30 minutes dasboard is up. But it shows a lot of internal errors...

Any hints why it took so much time and why it still isn't working?

Linkerd version is: 2.11
Kubectl version is: 1.22
OS: Ubuntu 20.04

kubectl -n linkerd-viz logs deploy/metrics-api metrics-api returns:

time="2021-11-11T20:35:16Z" level=info msg="running version dev-undefined"
time="2021-11-11T20:35:16Z" level=info msg="prometheusClient: &api.httpClient{endpoint:(*url.URL)(0xc0000cb050), client:http.Client{Transport:(*http.Transport)(0x2ffbac0), CheckRedirect:(func(*http.Request, []*http.Request) error)(nil), Jar:http.CookieJar(nil), Timeout:0}}"
time="2021-11-11T20:35:16Z" level=info msg="Using cluster domain: cluster.local"
time="2021-11-11T20:35:16Z" level=info msg="waiting for caches to sync"
time="2021-11-11T20:35:17Z" level=info msg="caches synced"
time="2021-11-11T20:35:17Z" level=info msg="starting admin server on :9995"
time="2021-11-11T20:35:17Z" level=info msg="starting HTTP server on :8085"
time="2021-11-11T20:43:21Z" level=error msg="Query(max(process_start_time_seconds{}) by (pod, namespace)) failed with: Post \"http://prometheus.linkerd-viz.svc.cluster.local:9090/api/v1/query\": context canceled"
time="2021-11-11T20:43:49Z" level=error msg="Query(max(process_start_time_seconds{}) by (pod, namespace)) failed with: Post \"http://prometheus.linkerd-viz.svc.cluster.local:9090/api/v1/query\": read tcp 10.10.149.140:53862->10.110.49.177:9090: read: connection reset by peer"
time="2021-11-11T20:43:57Z" level=error msg="Query(max(process_start_time_seconds{}) by (pod, namespace)) failed with: Post \"http://prometheus.linkerd-viz.svc.cluster.local:9090/api/v1/query\": read tcp 10.10.149.140:53932->10.110.49.177:9090: read: connection reset by peer"

Errors on dashboard:

500 Internal Server Error
Post "http://metrics-api.linkerd-viz.svc.cluster.local:8085/api/v1/StatSummary": read tcp 10.10.38.226:41000->10.102.15.119:8085: read: connection reset by peer
http://localhost:50750/api/tps-reports?resource_type=cronjob&namespace=default&tcp_stats=true&window=1m

500 Internal Server Error
Post "http://metrics-api.linkerd-viz.svc.cluster.local:8085/api/v1/ListPods": read tcp 10.10.38.226:46310->10.102.15.119:8085: read: connection reset by peer
http://localhost:50750/api/pods?namespace=linkerd

[adleong]

I wonder if your cluster is suffering from resource scarcity. I would recommend looking into the Kubernetes events and/or logs to determine why pods get stuck in PodInitalizing for so long.