expose issuer certificate TTL as a prometheus metric

[n-oden]

Problem: There is currently no simple way to monitor the expiration time of the issuer certificate in use by linkerd; a surprising omission considering that issuer cert expiration will almost certainly cause visible cluster issues.

Solution:

• When a new issuer certificate is loaded, log its NotAfter time in unix epoch format, along with the current process wall clock time. The two timestamps are passed in via the logrus Fields pattern, allowing operators to easily pull these numbers from pod logs.
• Register a prometheus gauge function metric to expose the TTL for monitoring

Fixes: #11215

[n-oden]

cc: @whickman :)

[n-oden]

(IMO it would be somewhat preferable to expose this as a prometheus metric, but to put it mildly I found the internal metrics story here opaque. If someone wanted to hold my hand a bit, I'd happily add it. But in the meantime, the log line should suffice for the basic case.)

[alpeb]

Thanks @n-oden , this looks good to me 👍
For adding the metric, you can leverage the promauto package, like in this example in the Destination controller:
https://github.com/linkerd/linkerd2/blob/edge-25.1.2/controller/api/destination/watcher/cluster_store.go#L102-L105
You could add a new field in the Service struct tracking the cert expiry (that would get updated in loadCredentials) and wire the metric in the NewService function.

[n-oden]

@alpeb thank you very much for the pointer! I've taken a stab at it. I may have over-thought the initialization logic; let me know. :)

[n-oden]

@alpeb for some reason trying to use promauto failed in the test suite because it attempted to register the gauge function twice; I've replaced it with a more traditional use of prometheus.Register().

The failing tests seem to be a github actions issue; I cannot reproduce them locally:

=== Skipped
=== SKIP: viz/cmd TestRequestTapByResourceFromAPI/Should_return_error_if_stream_returned_error (0.00s)
    --- SKIP: TestRequestTapByResourceFromAPI/Should_return_error_if_stream_returned_error (0.00s)

DONE 1063 tests, 1 skipped in 94.401s

[alpeb]

Thanks, yeah, better to be more explicit than relying on magical APIs 😉

Can you however issue a warning log instead of panic'ing? It's not critical if the metric cannot be registered, so better to get an informative error rather than failing the whole controller.

Another thing I believe we don't have metrics for is the trust root certs. If you're up for it, you could include that here or in a separate PR. The trust root might be a bundle of certs, so we could track the ttl for the cert that is the closest to expiry...

N.B.: Tests are green, failures were just flakiness.

[n-oden]

I've added a metric for the trust anchor TTL but beware that I couldn't figure out any plausible way to do this that didn't involve the deprecated Subjects() method -- looking at the CertPool documentation I'm not sure how one would go about doing this without touching that method.

[n-oden]

Reading through golang/go#46287 it looks like the intent is for tls.CertPool to be entirely opaque and the answer is "use the system verifier", which... I'm not sure how that is even supposed to apply to a case where you're essentially building your own CA locally? Anyway I tagged it //nolint for now but obviously this will break badly if/when the deprecated method is finally removed so your call whether to leave this in or not.

[alpeb]

We require the full certs, not just their subjects. Testing with a plain vanilla linkerd instance shows that can't be parsed:

time="2025-02-06T14:30:07Z" level=error msg="could not parse trust anchor certificate: x509: malformed tbs certificate"
time="2025-02-06T14:30:07Z" level=warning msg="Could not parse any trust anchor certs; cannot get TTL"

It seems CertPool doesn't expose a public API to retrieve its certs 🤷‍♂️
We could have this service receive the raw certs instead of CertPool, but this was just a nice-to-have and not worth the refactoring. So I think it's fine to leave this out and just add the issuer cert stuff... Thanks anyways for digging into this 🙂

[n-oden]

Oh well, it was worth a shot. :)

[n-oden]

Hey @alpeb, hope you had a good weekend. I think this is basically good to go, if you concur?