Add a good practices page for mutating webhook design

[shannonxtreme]

Add a good practices page for designing and running mutating admission webhooks

Doc plan: https://docs.google.com/document/d/1pg80Qn3Uz2SupCHjuQ_CRhnh9m3q9a3KU_PGV3ecVFE/edit?pli=1&tab=t.0

Overview of changes

• Create a new page based on the discussion in the doc plan

• Move existing best practices from Dynamic Admission Control page

• Deleted any existing practices that were significantly expanded on in the new page

• Added guidance to compare types of admission control mechanisms and choose the best one

  I used an HTML table to do lists and to organize the info more clearly.

Outstanding questions and omitted sections

The following good practices didn't have enough information, so I omitted them from this iteration:

• Webhooks vs. CI static yaml modifications and templating like kustomize? When to use what.
• Scope what events the Pod should react to
• Make sure to never double-mutate (check if object already has all modifications made)
• If you're on older Kubernetes versions, they don't warn you when you provide an unknown or malformed field - be extra careful
• Group endpoints into one or more binaries
• Keep your mutating webhook up to date - make sure you update often as webhook can adopt new k8s API changes with every release. (TODO: is there any tooling that can help detect that it can be updated?)
• Use of webhook abstraction tools such as Kyverno that should handle a lot of the best practices for you already

Co-authors/contributors

• @jpbetz
• @SergeyKanzhelev
• @cici37
• Max Smythe
• @AverageMarcus
• @sftim

Closes: #47465

/sig docs
/language en

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign dipesh-rawat for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• content/en/docs/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[shannonxtreme]

One question for reviewers: Some of the good practices that I moved seem relevant to validating webhooks as well. Should I keep the old content on the original page to avoid a loss of info?

[netlify]

✅ Pull request preview available for checking

Built without sensitive environment variables

Name	Link
🔨 Latest commit	58a1ab9
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-io-main-staging/deploys/67b61061d2e6b5000848a0a1
😎 Deploy Preview	https://deploy-preview-49626--kubernetes-io-main-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[sftim]

Thanks for this PR. I recommend a copy editing round (see inline feedback), as well as a fact check.

Although this is a move of existing content, I recommend getting the styling right where we can.

It's not obligatory to fix any of the moved stuff. The net new text should follow our style guide where possible though.

[sftim]

We used to support configuring these statically by passing a plugin configuration file to the kube-apiserver.

At technical review time or earlier, let's validate the the old method is completely omitted. Otherwise, this text isn't 100% accurate.

[shannonxtreme]

Is this the bit that's listed here: https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#authenticate-apiservers? Or is that different?

[sftim]

Outstanding questions and omitted sections

Once this merges, you are welcome - but not forced! - to file an issue about the remaining gaps.

[sftim]

I might put this in the Cluster Administration section, and have https://kubernetes.io/docs/concepts/security/ also link there.

[shannonxtreme]

I'll do this before submission so that it doesn't muddle up the reviews

[sftim]

I expect to LGTM this next review.

The quality bar is: does merging this make our docs overall better—provided that we didn't introduce anything that obviously misleads.

[guettli]

@shannonxtreme my best practice is to avoid mutating webhooks, because they confuse gitops tools. It's likely that ArgoCD will think that a resource is out of sync, and revert the changes made by the mutating webhooks.

We even avoid that gitops tool change the desired state stored in git. It's called the Rendered Manifests Pattern: https://akuity.io/blog/the-rendered-manifests-pattern

What about adding the RMP to the instruction section?

Additional note: some parts of the doc could link to the API conventions. The API conventions doc is the "single source of truth" about how to design or interact with Kubernetes APIs.

[SergeyKanzhelev]

@shannonxtreme my best practice is to avoid mutating webhooks, because they confuse gitops tools. It's likely that ArgoCD will think that a resource is out of sync, and revert the changes made by the mutating webhooks.

We even avoid that gitops tool change the desired state stored in git. It's called the Rendered Manifests Pattern: https://akuity.io/blog/the-rendered-manifests-pattern

What about adding the RMP to the instruction section?

Additional note: some parts of the doc could link to the API conventions. The API conventions doc is the "single source of truth" about how to design or interact with Kubernetes APIs.

It will be a great addition to the article. Maybe we can start with merging this iteration. The article already stating some practices how to avoid using webhook, adding more is definitely beneficial.

[sftim]

You might find #46798 useful too @shannonxtreme; feel free to use elements of that PR (with appropriate credit!)

[SergeyKanzhelev]

Also this one: https://github.com/kubernetes/website/pull/46798/files#r1946911968

It would be nice to merge this PR so we can iterate

[shannonxtreme]

@SergeyKanzhelev I decided not to add the content from kubernetes/enhancements#5068 (comment) in this version of the doc as it would tie up the reviews a bit more. Let's merge this MVP and build on it!

[shannonxtreme]

@sftim @SergeyKanzhelev this is ready for another review. There are some unresolved comments in this PR that should either be a follow-up or need more info. I've listed them here:

1. Rendered Manifest Patterns comment: follow-up
2. Validation-only examples of idempotency comment: add now or follow up?
3. Service load-balancing - no info in Service docs about it even though it's in the webhook best practices even today comment: remove the paragraph or keep?
4. Static webhook configuration using file - is it omitted in the new page? comment: tech reviewer should confirm whether this is good to resolve

[k8s-ci-robot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.