Missing TCP stats on database pods

[nevetS]

Bug Report

What is the issue?

I have applied linkerd inject on a mariadb pod. I have verified that the proxy container is active.

I only see prometheus traffic going to those pods. Other pods that access the database do not generate metrics that I see in promethus graphs, within the dashboard, or in the linkerd cli.

I do see promethous traffic going to mariadb in the dashboard and grafana.

How can it be reproduced?

Create a deployment for mariadb. Inject that deployment. Create a deployment for a web service that accesses that database. Inject that deployment. Verify proxy injection. Watch while nothing happens.

Logs, error output, etc

kgp mariadb-deployment-6548c7df8c-9gvc6
NAME                                  READY   STATUS    RESTARTS   AGE
mariadb-deployment-6548c7df8c-9gvc6   2/2     Running   0          2d3h

linkerd stat deployments -n default
NAME                                    MESHED   SUCCESS      RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99   TCP_CONN
mariadb-deployment                         1/1         -        -             -             -             -          -

linkerd check output

 linkerd check
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ controller pod is running
√ can initialize the client
√ can query the control plane API

linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ control plane PodSecurityPolicies exist

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust anchor

linkerd-webhooks-and-apisvc-tls
-------------------------------
√ tap API server has valid cert
√ tap API server cert is valid for at least 60 days
√ proxy-injector webhook has valid cert
√ proxy-injector cert is valid for at least 60 days
√ sp-validator webhook has valid cert
√ sp-validator cert is valid for at least 60 days

linkerd-api
-----------
√ control plane pods are ready
√ control plane self-check
√ [kubernetes] control plane can talk to Kubernetes
√ [prometheus] control plane can talk to Prometheus
√ tap api service is running

linkerd-version
---------------
√ can determine the latest version
√ cli is up-to-date

control-plane-version
---------------------
√ control plane is up-to-date
√ control plane and cli versions match

linkerd-prometheus
------------------
√ prometheus add-on service account exists
√ prometheus add-on config map exists
√ prometheus pod is running

linkerd-grafana
---------------
√ grafana add-on service account exists
√ grafana add-on config map exists
√ grafana pod is running

Status check results are √

Environment

• Kubernetes Version: 1.16
• Cluster Environment: EKS
• Host OS: linux
• Linkerd version: 2.9

Possible solution

Additional context

My searching through issues and docs seems to indicate that I should be seeing metrics associated with TCP traffic.

[olix0r]

@nevetS If mariadb uses a mysql-like protocol, the server sends data before the client -- the proxy does not currently support this class of protocol, though we plan to add this support in stable-2.10 (planned for ~January). In fact, if mariadb uses the default mysql port of 3306, we currently avoid proxying these connections altogether:
https://github.com/linkerd/linkerd2/blob/main/charts/linkerd2/values.yaml#L98-L99

If my assumptions about mariadb are incorrect, you can check the raw proxy metrics via:

linkerd metrics -n $NS po/$pod | grep ^tcp_open_total