Provider keeps updating and creating objects

[recht]

What happened?

We recently upgraded from 0.9 to 0.16, and a while after that we modified an existing XR to add more Object resources. When applied, provider-kubernetes kept creating the objects over and over again with different names. At this point we also noticed that the generation field of all previous Objects had been incremented to around 14000, and a watch on one of the resources showed that it kept being updated every minute (no actual changes, just a write that triggered a new generation/resourceVersion).

We triaged the problem to be in 0.14.0 - any release before that only creates objects once and do not keep updating existing objects, any version after has that behavior.

Debug logging only showed Reconciling/Observing/Up to date!, so there's some indication that it's because it's either doing an update no matter what, or maybe it's updating the status?

This screenshot shows number of reconciles, and it's quite visible where we downgraded to 0.13.0.

How can we reproduce it?

Reproducing outside our environment is unclear, but the generation/resourceVersion of the Object k8s resources kept increasing every minute once the provider is at least v0.14.0.

What environment did it happen in?

Crossplane version: v0.17.3, provider-kubernetes v0.14.0-v0.16.0

[daniel-palmer-gu]

@recht How are you creating the Objects?

I had this issue for a while and, it was preventing me from upgrading to a newer version. I had been using the go-templating function to create Objects and realized recently that I had still been using kubernetes.crossplane.io/v1alpha1 apiVersion for Objects. After updating that to kubernetes.crossplane.io/v1alpha2, my problem went away.

Looking at the CRD, it has a conversion webhook and only v1alpha2 is a stored version. According to Kubernetes docs, when a resource is being created that isn't in the stored versions, it will attempt to use the conversion webhook. As a result, anytime a v1alpha1 Object is created, it is converted to a v1alpha2 Object in the background.

My guess is that a composition you have is still trying to use the v1alpha1 apiVersion, and Crossplane is getting confused since it doesn't realize Kubernetes is changing the apiVersion after the fact.

What is uncertain is why this is cropping up all of a sudden. v1alpha1 has been deprecated for a while as far as I can tell, so I wonder if some logic in how Crossplane generates/reconciles resources has changed.

[bobh66]

@recht this might be a result of the problem described in crossplane/crossplane#6260 which is fixed by crossplane/crossplane#6261 in upcoming releases of 1.19 and 1.18

[ryanseipp]

I'm seeing this with:

• Crossplane 1.20
• Provider Kubernetes 0.18
• Composition w/ custom Go function that creates kubernetes.crossplane.io/v1alpha2 Objects

I enabled debug logging, and reconciliation seems to be absolutely thrashing itself, hammering the API server:

2025-07-26T19:39:14.734Z    DEBUG    provider-kubernetes    Reconciling ...
2025-07-26T19:39:14.737Z    DEBUG    provider-kubernetes    Observing ...
2025-07-26T19:39:14.775Z    DEBUG    provider-kubernetes    External resource is up to date ...

This is happening for 230 such objects, hitting ~20MB/s on average traffic to the kube-apiserver

[bobh66]

Can you attach a copy of the composition that is creating the objects? It sounds like the composition is defining something in the object which conflicts with what kubernetes wants it to be.

[ryanseipp]

Can you attach a copy of the composition that is creating the objects? It sounds like the composition is defining something in the object which conflicts with what kubernetes wants it to be.

Unfortunately I can't attach a copy, but I was able to find the issue. In that custom Go function, I was passing typed objects from k8s.io/api to provider-kubernetes, which included empty structs for status fields in some cases. I refactored to use unstructured.Unstructured types with typed spec fields and this seems to have resolved the issue for me.

So TL;DR: don't tell provider-kubernetes to manage status fields 🤦