Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add documentation for loopback certificates in kube-apiserver #49518

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add documentation for loopback certificates in kube-apiserver #49518

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion api-ref-generator
Open in desktop
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW this doesn't look right

Submodule api-ref-generator updated 187 files
16 changes: 16 additions & 0 deletions content/en/docs/setup/best-practices/certificates.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,22 @@ Kubernetes requires PKI for the following operations:
for each kubelet (every {{< glossary_tooltip text="node" term_id="node" >}} runs a kubelet)
* Optional server certificate for the [front-proxy](/docs/tasks/extend-kubernetes/configure-aggregation-layer/)

#### Loopback Certificate in kube-apiserver
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(nit - fixing case and setting an explicit fragment identifier)

Suggested change
#### Loopback Certificate in kube-apiserver
#### Loopback certificate within kube-apiserver {#kube-apiserver-loopback-certificate)


The loopback certificate is used by the kube-apiserver to establish privileged loopback connections.
Copy link
Contributor

@sftim sftim Feb 4, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The loopback certificate is used by the kube-apiserver to establish privileged loopback connections.
The kube-apiserver issues a special, internal-use certificate that it uses to establish privileged _loopback_ connections.

This certificate is stored in memory and has a validity period of one year.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
This certificate is stored in memory and has a validity period of one year.
This certificate and its private key are stored in memory; the certificate has a validity period of one year.

However, it does not renew automatically. If the certificate expires, the kube-apiserver will cease to function.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would suggest to give an example of the error that happens in this case.


The kube-apiserver is supported by the Kubernetes community for a [standard period](/releases/patch-releases/#support-period) of 12 months,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

q: what about the additional 2 months of "maintenance support"?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current code sets the validity period of the loopback certificate to 1 year. If we add documentation explaining support for an additional two months, it would create inconsistencies in the documentation. Alternatively, if/when we eventually set the certificate validity period to 14 months in the future, we could then add documentation for the extra two months at that time.

which aligns with the validity duration of its loopback certificate.
Copy link
Member

@sbueringer sbueringer Feb 6, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get the point that this explains why this usually shouldn't happen. But honestly, I think kube-apiserver not working anymore should not be the way we enforce our support policy. If people want to keep it running for 2 years or longer. It should just work. (I"m not saying it should still be supported by the community, but the kube-apiserver should also not just stop working because the support "expired")

Everything else will lead to needless production outages

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The issues you mentioned have already been discussed in kubernetes/kubernetes#86552. As of now, we have no plans to extend the validity period of the loopback certificate or enable automatic rotation of the certificate.

/cc @liggitt

Copy link
Member

@sbueringer sbueringer Feb 6, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh wow, wasn't aware of that. Thx for sharing the issue!

In any case, thx for the PR to document the issue to alert users!

Copy link
Contributor

@sftim sftim Feb 6, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we don't automate reissue, I think we should instead formally recommend a restart every 6 months (six, not twelve). Or more often, of course, and I've proposed it should be rolling, ie one at a time.

To maintain compatibility with community support timelines, it's recommended to ensure your kube-apiserver is upgraded within this support window.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about we say this?

Suggested change
To maintain compatibility with community support timelines, it's recommended to ensure your kube-apiserver is upgraded within this support window.
To maintain compatibility with community support timelines, the Kubernetes project recommends that you:
- upgrade to the most recent patch releases whereever feasible
- perform a rolling restart of API servers within your production cluster at least once every 6 months
- upgrade your cluster to a newer Kubernetes release before its standard support period elapses

Can I get a tech review from SIG Cluster Lifecycle on whether we do / don't want to add the rolling restart that I'm personally advocating for here? I think it's a good idea but it's not for SIG Docs to call.

The advice about patch releases isn't new; I'm just reinforcing it here.

You might feel that recommending a rolling restart allows people to find a way not to upgrade. That's true, in a way, but I think it's not much mental effort to figure that out anyway. An LLM could do it. So, we may as well just tell them.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can I get a tech review from SIG Cluster Lifecycle on whether we do / don't want to add the rolling restart that I'm personally advocating for here? I think it's a good idea but it's not for SIG Docs to call.

personally, i have not heard of such a recommendation before. cc @justinsb
sig api machnery should have the final say as apiserver component owners.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It'd be a new recommendation; we don't have any existing good practice about the loopback certificate.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ideally, the apiserver should rotate any memory certificates instead of requiring restarts by the user.
@HirazawaUi do you have links to the k/k source code?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the code path where GenerateSelfSignedCertKey creates a self-signed certificate for the loopback client (with a validity period of one year), the generated loopbackClientConfig is applied to the kube-apiserver through the ApplyTo function. This process only executes once during startup, so we currently do not rotate this certificate.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the code you linked existed for 7 years and likely a lot of users don't upgrade once a year.
so, if apiserver is managed as a static pod or by systemd, those would supposedly just restart it once it errors, so maybe nobody noticed until now.

in any case, sounds like a bug / missing feature. you should log a k/k ticket and assign /sig api-machinery, and/or even PR k/k with a fix if you'd like. client-go did have a cert manager of sorts that supports rotation.

given this has existed for so long, might still be suitable to be added to the best-practices/certificates page, but better to be in the troubleshooting docs, imo.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so, if apiserver is managed as a static pod or by systemd, those would supposedly just restart it once it errors, so maybe nobody noticed until now.

If an issue occurs, the apiserver will stop working but will not restart. Previously, users have reported this issue in the k/k repo, but we think the current behavior is expected, so we have not fixed it. Ref: kubernetes/kubernetes#86552

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the API server pass a readiness check after that expiry? If it does, we could consider that a bug.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so, if apiserver is managed as a static pod or by systemd, those would supposedly just restart it once it errors, so maybe nobody noticed until now.

If an issue occurs, the apiserver will stop working but will not restart. Previously, users have reported this issue in the k/k repo, but we think the current behavior is expected, so we have not fixed it. Ref: kubernetes/kubernetes#86552

that's more of a bug, than a missing feature in that case.

The loopback certificate will be automatically renewed during the kube-apiserver upgrade process.

{{< note >}}
Restarting the kube-apiserver is another way to renew the loopback certificate.
{{< /note >}}


### Client certificates

* Client certificates for each kubelet, used to authenticate to the API server as a client of
Expand Down