invalid bearer token, Token has been invalidated

[justadevopsguy]

Hello! We are currently investigating a surge of 503 errors in one of our components last March 15. Upon checking on our logs. We saw a lot of this errors in the linkerd-proxy of our component

ERROR ThreadId(02) daemon:identity: linkerd_proxy_identity::certify: Failed to certify identity: status: InvalidArgument, message: "[invalid bearer token, Token has been invalidated, unknown]",

typ: Alert,
    version: TLSv1_3,
    payload: Alert(
        AlertMessagePayload {
            level: Fatal,
            description: BadCertificate,
        },

Note: This only happen on one of our component over 398 deployments

Some info:
we are running AWS EKS with version: v1.19.15
Linkerd: stable-2.10.2

Ask:
Any idea what this might have caused?

[mateiidavid]

@justadevopsguy hey, this is a bit odd indeed. Do you see this even after the workload is restarted?

My first hunch would be to check your ServiceAccount token. SAs are tied to identity in Linkerd's operational model. Whenever a proxy spins up, it first acquires a leaf certificate by talking to the identity service. It will send a Certificate Signing Request which will include the SA name as the cert SAN and the SA token. The identity service will validate the token and if everything matches, it issues the certificate.

Currently, this is the only type of token we use. It's possible perhaps the token was malformed or the token expired? Especially if it's only one workload that suffers from it. I'd try to either rotate or re-create the SA for that workload and give it another try, it might end up fixing the issue.

If that fails, we can have a look at the identity service's logs to see if there are more warnings or errors that can help us pinpoint the problem.

[justadevopsguy]

hi @mateiidavid . thanks for the reply!

Do you see this even after the workload is restarted?

Restart works. Based on our CI/CD the dev triggered a restart deployment that day and the errors was gone.

One observation that I saw on identity pod, is that there's no logs like this on March 14 (UTC on our log server)

msg="certifying <pod-name>.<namespace>.serviceaccount.identity.linkerd.cluster.local until

You can see on the screenshot that the next log was March 15, the day & time it was restarted

It's possible perhaps the token was malformed or the token expired?

I tried to replicate the issue by modifying the SA token of my test service, I can replicate only this error

[     0.025551s] ERROR ThreadId(02) daemon:identity: linkerd_proxy_identity::certify: Failed to certify identity: status: InvalidArgument, message: "[invalid bearer token, Token has been invalidated, unknown]", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 30 Mar 2022 00:10:20 GMT"} }

But can't replicate the BadCertificate even though I used the old token. Using old token will result to Sending fatal alert AccessDenied

[justadevopsguy]

btw just to add @mateiidavid this happened to us twice already. Previous occurrence was Jan 21, 2022. It would be great to know if there's a possible resolution aside from "restarting" the deployment.

[mateiidavid]

@justadevopsguy I'm very sympathetic to certificate issues :) It's hard for us to provide a concrete resolution though, unless we know what the exact problem is. In such a huge distributed system, with varying configuration pieces, issues that relate to certificates or SA tokens are not isolated to Linkerd -- a bunch of different parts of k8s can trigger errors.

If you want to, we can turn this into an issue if you give us some repro steps. We can investigate further to see if it's a bug on our side. It is worth pointing out that you're also running 2.10.2, we have made quite a few changes since then.

A couple of other questions:

• do you have any policies in place to refresh your SA tokens?
• you mentioned this only happened to one workload. Is this the same workload that had issues in January?

[justadevopsguy]

@mateiidavid, thanks for the insights and I agree with you have said

If you want to, we can turn this into an issue if you give us some repro steps.

Unfortunately, We don't really have a step on how to reproduce :( , the one that I tried is just to update the SA token secret directly. However, I don't think it's the same case.

do you have any policies in place to refresh your SA tokens?

none.

you mentioned this only happened to one workload. Is this the same workload that had issues in January?

Yep, same workload.

Appreciate the help @mateiidavid 🙇