"linkerd viz edges po -A" returns "No edges found."

[stevehuangc7s]

I have installed the linkerd and linkerd-viz with helm charts in my EKS cluster at 1.18, where the helm repo is https://helm.linkerd.io/stable and chart version is 2.11.1 for both charts. Then I followed steps from https://linkerd.io/2.11/getting-started/#step-4-install-the-demo-app to install the demo app. When trying to see if the mTLS is properly enabled for the demo app, I got No edges found. in response to command linkerd viz edges po -A.

To investigate the issue, I have tried the linkerd check, linkerd viz check and linkerd check --proxy. All check returned one issue - ‼ prometheus is installed and configured correctly. And linkerd check --proxy returned one more issue:

‼ data plane proxy metrics are present in Prometheus
    Data plane metrics not found for emojivoto/vote-bot-64dd87cb87-qwl67, emojivoto/voting-f7f875999-hk69p, emojivoto/web-5798f7d79f-czq48, emojivoto/emoji-5f947b5dbd-m6shh.

I am not sure if it's relevant to the linkerd viz edges problems but I used an existing prometheus setup from within the EKS cluster in monitoring namespace for the linkerd-viz installation. Please see below the chart values for more information. Could you please suggest what I can do to fix the problem and to validate if mTLS is enabled?

Please see below my values for both charts:

# values for linkerd
  valuesFrom:
    - kind: Secret
      name: linkerd-certs
      valuesKey: ca.crt
      targetPath: identityTrustAnchorsPEM
    - kind: Secret
      name: linkerd-certs
      valuesKey: issuer.crt
      targetPath: identity.issuer.tls.crtPEM
    - kind: Secret
      name: linkerd-certs
      valuesKey: issuer.key
      targetPath: identity.issuer.tls.keyPEM

  # Default values
  # https://artifacthub.io/packages/helm/linkerd2/linkerd2
  values:
    installNamespace: false
    enablePodAntiAffinity: true

    # proxy configuration
    proxy:
      resources:
        cpu:
          request: 100m
        memory:
          limit: 250Mi
          request: 20Mi

    # controller configuration
    controllerReplicas: 3
    controllerResources:
      cpu:
        limit: ""
        request: 100m
      memory:
        limit: 250Mi
        request: 50Mi
    destinationResources:
      cpu:
        limit: ""
        request: 100m
      memory:
        limit: 250Mi
        request: 50Mi

    # identity configuration
    identityResources:
      cpu:
        limit: ""
        request: 100m
      memory:
        limit: 250Mi
        request: 10Mi

    # heartbeat configuration
    heartbeatResources:
      cpu:
        limit: ""
        request: 100m
      memory:
        limit: 250Mi
        request: 50Mi

    # proxy injector configuration
    proxyInjectorResources:
      cpu:
        limit: ""
        request: 100m
      memory:
        limit: 250Mi
        request: 50Mi

    webhookFailurePolicy: Fail

    # service profile validator configuration
    spValidatorResources:
      cpu:
        limit: ""
        request: 100m
      memory:
        limit: 250Mi
        request: 50Mi

# values for linkerd-viz
  values:
    linkerdNamespace: linkerd
    installNamespace: false
    namespace: linkerd-viz
    prometheusUrl: "http://prometheus-stack-prometheus.monitoring.svc.cluster.local:9090"
    grafanaUrl: "http://prometheus-stack-grafana.monitoring.svc.cluster.local:80"
    grafana:
      enabled: false
    prometheus:
      enabled: false

[olix0r]

Could you please suggest what I can do to fix the problem

Have you configured your prometheus instance with an appropriate scrape config? For example:

linkerd2/viz/charts/linkerd-viz/templates/prometheus.yaml

Lines 18 to 159 in 98fe879

	prometheus.yml: |-
	global:
	{{- if .Values.prometheus.globalConfig -}}
	{{- toYaml .Values.prometheus.globalConfig | trim | nindent 6 }}
	{{- end}}
	rule_files:
	- /etc/prometheus/*_rules.yml
	- /etc/prometheus/*_rules.yaml
	scrape_configs:
	- job_name: 'prometheus'
	static_configs:
	- targets: ['localhost:9090']
	# Required for: https://grafana.com/grafana/dashboards/315
	- job_name: 'kubernetes-nodes-cadvisor'
	scheme: https
	tls_config:
	ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
	insecure_skip_verify: true
	bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
	kubernetes_sd_configs:
	- role: node
	relabel_configs:
	- action: labelmap
	regex: __meta_kubernetes_node_label_(.+)
	- target_label: __address__
	replacement: kubernetes.default.svc:443
	- source_labels: [__meta_kubernetes_node_name]
	regex: (.+)
	target_label: __metrics_path__
	replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
	metric_relabel_configs:
	- source_labels: [__name__]
	regex: '(container|machine)_(cpu|memory|network|fs)_(.+)'
	action: keep
	- source_labels: [__name__]
	regex: 'container_memory_failures_total' # unneeded large metric
	action: drop
	- job_name: 'linkerd-controller'
	kubernetes_sd_configs:
	- role: pod
	namespaces:
	names:
	- '{{.Values.linkerdNamespace}}'
	- '{{.Release.Namespace}}'
	relabel_configs:
	- source_labels:
	- __meta_kubernetes_pod_container_port_name
	action: keep
	regex: admin-http
	- source_labels: [__meta_kubernetes_pod_container_name]
	action: replace
	target_label: component
	- job_name: 'linkerd-service-mirror'
	kubernetes_sd_configs:
	- role: pod
	relabel_configs:
	- source_labels:
	- __meta_kubernetes_pod_label_component
	- __meta_kubernetes_pod_container_port_name
	action: keep
	regex: linkerd-service-mirror;admin-http$
	- source_labels: [__meta_kubernetes_pod_container_name]
	action: replace
	target_label: component
	- job_name: 'linkerd-proxy'
	kubernetes_sd_configs:
	- role: pod
	relabel_configs:
	- source_labels:
	- __meta_kubernetes_pod_container_name
	- __meta_kubernetes_pod_container_port_name
	- __meta_kubernetes_pod_label_linkerd_io_control_plane_ns
	action: keep
	regex: ^{{default .Values.proxyContainerName "linkerd-proxy" .Values.proxyContainerName}};linkerd-admin;{{.Values.linkerdNamespace}}$
	- source_labels: [__meta_kubernetes_namespace]
	action: replace
	target_label: namespace
	- source_labels: [__meta_kubernetes_pod_name]
	action: replace
	target_label: pod
	# special case k8s' "job" label, to not interfere with prometheus' "job"
	# label
	# __meta_kubernetes_pod_label_linkerd_io_proxy_job=foo =>
	# k8s_job=foo
	- source_labels: [__meta_kubernetes_pod_label_linkerd_io_proxy_job]
	action: replace
	target_label: k8s_job
	# drop __meta_kubernetes_pod_label_linkerd_io_proxy_job
	- action: labeldrop
	regex: __meta_kubernetes_pod_label_linkerd_io_proxy_job
	# __meta_kubernetes_pod_label_linkerd_io_proxy_deployment=foo =>
	# deployment=foo
	- action: labelmap
	regex: __meta_kubernetes_pod_label_linkerd_io_proxy_(.+)
	# drop all labels that we just made copies of in the previous labelmap
	- action: labeldrop
	regex: __meta_kubernetes_pod_label_linkerd_io_proxy_(.+)
	# __meta_kubernetes_pod_label_linkerd_io_foo=bar =>
	# foo=bar
	- action: labelmap
	regex: __meta_kubernetes_pod_label_linkerd_io_(.+)
	# Copy all pod labels to tmp labels
	- action: labelmap
	regex: __meta_kubernetes_pod_label_(.+)
	replacement: __tmp_pod_label_$1
	# Take `linkerd_io_` prefixed labels and copy them without the prefix
	- action: labelmap
	regex: __tmp_pod_label_linkerd_io_(.+)
	replacement: __tmp_pod_label_$1
	# Drop the `linkerd_io_` originals
	- action: labeldrop
	regex: __tmp_pod_label_linkerd_io_(.+)
	# Copy tmp labels into real labels
	- action: labelmap
	regex: __tmp_pod_label_(.+)
	{{- if .Values.prometheus.scrapeConfigs }}
	{{- toYaml .Values.prometheus.scrapeConfigs | trim | nindent 4 }}
	{{- end }}
	{{- if (or .Values.prometheus.alertmanagers .Values.prometheus.alertRelabelConfigs) }}
	alerting:
	alert_relabel_configs:
	{{- if .Values.prometheus.alertRelabelConfigs }}
	{{- toYaml .Values.prometheus.alertRelabelConfigs | trim | nindent 6 }}
	{{- end }}
	alertmanagers:
	{{- if .Values.prometheus.alertmanagers }}
	{{- toYaml .Values.prometheus.alertmanagers | trim | nindent 6 }}
	{{- end }}
	{{- end }}
	{{- if .Values.prometheus.remoteWrite }}
	remote_write:
	{{- toYaml .Values.prometheus.remoteWrite | trim | nindent 4 }}
	{{- end }}

. If you're using an external prometheus, it needs to be configured to collect the proxy's metrics with the configured labels.

and to validate if mTLS is enabled?

You can always spot-check a pod's metrics without prometheus by using linkerd diagnostics proxy-metrics ... to check that the relevant metrics include a tls="true" label.

[stevehuangc7s]

Hey @olix0r, thanks for the input.

You can always spot-check a pod's metrics without prometheus by using linkerd diagnostics proxy-metrics ... to check that the relevant metrics include a tls="true" label.

I checked the demo app pod's metrics and found there's tls="true" and tls="no_identity" in the output. Can I think the mTLS is already enabled for the demo pod as long as there's tls="true" in the output?

...
# TYPE request_total counter
request_total{direction="inbound",authority="emoji-svc.emojivoto.svc.cluster.local:8080",target_addr="10.0.3.100:8080",target_ip="10.0.3.100",target_port="8080",tls="true",client_id="web.emojivoto.servi
ceaccount.identity.linkerd.cluster.local",srv_name="default:all-unauthenticated",saz_name="default:all-unauthenticated"} 173346
request_total{direction="inbound",target_addr="0.0.0.0:4191",target_ip="0.0.0.0",target_port="4191",tls="no_identity",no_tls_reason="no_tls_from_remote",srv_name="default:all-unauthenticated",saz_name="
default:all-unauthenticated"} 17472
...

Have you configured your prometheus instance with an appropriate scrape config?

No I haven't configured the scape config on the in-cluster prometheus. Is it a dependency to the linkerd viz edges ... commands?

Btw, given that I am using in-cluster grafana and prometheus, are my grafanaUrl and prometheusUrl configured properly?

    prometheusUrl: "http://prometheus-stack-prometheus.monitoring.svc.cluster.local:9090"
    grafanaUrl: "http://prometheus-stack-grafana.monitoring.svc.cluster.local:80"

[stevehuangc7s]

The issue has been resolved after the prometheus scape config has been added to the existing prometheus instance. Thanks for helping out!