ERROR: failed to verify issuer credentials for 'identity.linkerd.cluster.local' with trust anchors: x509: certificate has expired or is not yet valid

[bhavyaravilla]

We have deployed Linkerd using the helm charts.
For certificates, we are using cert-manager for auto-renewal of certificates. The certificate itself would be valid for 48h but we have set the renewal for 24h. Randomly, in about a month or two there is an warning that comes up saying

{"log":"time=\"2023-12-13T13:40:50Z\" level=warning msg=\"Skipping issuer update as certs could not be read from disk: failed to verify issuer credentials for 'identity.linkerd.cluster.local' with trust anchors: x509: certificate has expired or is not yet valid: current time 2023-12-13T13:40:50Z is after 2023-12-13T13:40:08Z - Current Time : 2023-12-13 13:40:50.279788145 +0000 UTC m=+5023800.297985164 - Invalid before 2023-12-13 13:40:13 +0000 UTC - Invalid After 2023-12-15 13:40:13 +0000 UTC\"","logtag":"F"}

And after this none of the linkerd proxy certificate were renewed. All of them failed with the below error

{"log":"time=\"2023-12-13T13:51:06Z\" level=error msg=\"could not process CSR because of CA cert validation failure: x509: certificate has expired or is not yet valid: current time 2023-12-13T13:51:06Z is after 2023-12-13T13:40:08Z - Current Time : 2023-12-13 13:51:06.01223291 +0000 UTC m=+5024416.030429929 - Invalid before 2023-12-12 13:40:13 +0000 UTC - Invalid After 2023-12-14 13:40:13 +0000 UTC - CSR Identity : audit-service.serviceaccount.identity.linkerd.cluster.local\"","logtag":"F"}

And on the proxy containers

{"log":"[5081495.311735s] ERROR ThreadId(02) identity: linkerd_proxy_identity_client::certify: Failed to obtain identity error=status: Unknown, message: \"x509: certificate has expired or is not yet valid: current time 2023-12-13T13:51:06Z is after 2023-12-13T13:40:08Z - Current Time : 2023-12-13 13:51:06.01223291 +0000 UTC m=+5024416.030429929 - Invalid before 2023-12-12 13:40:13 +0000 UTC - Invalid After 2023-12-14 13:40:13 +0000 UTC\", details: [], metadata: MetadataMap { headers: {\"content-type\": \"application/grpc\", \"date\": \"Wed, 13 Dec 2023 13:51:06 GMT\"} }","logtag":"F"}

The only thing that we can do to resolve this is to restart the control plane. But, meanwhile all our applications are down. This happens only on Prod.
Is there any solution for this please.

[wmorgan]

@bhavyaravilla you will need to help us debug this a bit. One thing that sticks out to me is that the core error message "current time 2023-12-13T13:40:50Z is after 2023-12-13T13:40:08Z" -- the :08 timestamp there falls outside of the bounds reported later in the message. When you see this happening, can you provide some details about the certificates (e.g. the validity dates as expressed by step certificate inspect) so that we can understand what certificates are actually in play?

[bhavyaravilla]

Thanks @wmorgan. After troubleshooting on our end, we realize the problem with the root certs (trust anchor certificate) being rotated every 60 days by the cert manager. Since the certs were loaded as volumes the identity pods had to be restarted for the changes to be effective.
We were unable to check this as the issue occurred a month after these certificates were rotated (previous certs being valid for 90 days).
We are currently considering either manually restarting the identity pods every 75 days or using a certificate that's valid longer. But we would need a proper solution. Any suggestions here would be very helpful.

[bhavyaravilla]

@wmorgan Is there a way to add automatic update of the trust anchor certificates after they are rotated by cert manager?

[wmorgan]

Not really. If nothing else, the trust anchor for proxies is immutable during runtime, so they need to be restarted to pick up the new trust bundle. In Linkerd Enterprise we have an operator that does this restarting but there isn't really an open source equivalent.

[siarener]

Hello, I'm looking into this, too (I replied on the Forum post https://linkerd.buoyant.io/t/error-failed-to-verify-issuer-credentials-for-identity-linkerd-cluster-local-with-trust-anchors-x509-certificate-has-expired-or-is-not-yet-valid/348). Would the Stakater Reloader be a viable open source solution? It is mentioned here: https://cert-manager.io/docs/tutorials/getting-started-with-trust-manager/#rollout-ca-bundle-changes.

[siarener]

I tried it out, the Skatater Reloader works, and the linkerd pods restart as expected, and my setup continues to work - I had two trust anchor changes in the meantime. Thank you for identifying the problem @bhavyaravilla, @wmorgan !

[bhavyaravilla]

@wmorgan Thank you. Would there be anything in the future to add this feature?
Also, what would be the recommended max duration that can be set for the trust anchor certificates?