Provider keeps updating and creating objects

[recht]

What happened?

We recently upgraded from 0.9 to 0.16, and a while after that we modified an existing XR to add more Object resources. When applied, provider-kubernetes kept creating the objects over and over again with different names. At this point we also noticed that the generation field of all previous Objects had been incremented to around 14000, and a watch on one of the resources showed that it kept being updated every minute (no actual changes, just a write that triggered a new generation/resourceVersion).

We triaged the problem to be in 0.14.0 - any release before that only creates objects once and do not keep updating existing objects, any version after has that behavior.

Debug logging only showed Reconciling/Observing/Up to date!, so there's some indication that it's because it's either doing an update no matter what, or maybe it's updating the status?

This screenshot shows number of reconciles, and it's quite visible where we downgraded to 0.13.0.

How can we reproduce it?

Reproducing outside our environment is unclear, but the generation/resourceVersion of the Object k8s resources kept increasing every minute once the provider is at least v0.14.0.

What environment did it happen in?

Crossplane version: v0.17.3, provider-kubernetes v0.14.0-v0.16.0

[daniel-palmer-gu]

@recht How are you creating the Objects?

I had this issue for a while and, it was preventing me from upgrading to a newer version. I had been using the go-templating function to create Objects and realized recently that I had still been using kubernetes.crossplane.io/v1alpha1 apiVersion for Objects. After updating that to kubernetes.crossplane.io/v1alpha2, my problem went away.

Looking at the CRD, it has a conversion webhook and only v1alpha2 is a stored version. According to Kubernetes docs, when a resource is being created that isn't in the stored versions, it will attempt to use the conversion webhook. As a result, anytime a v1alpha1 Object is created, it is converted to a v1alpha2 Object in the background.

My guess is that a composition you have is still trying to use the v1alpha1 apiVersion, and Crossplane is getting confused since it doesn't realize Kubernetes is changing the apiVersion after the fact.

What is uncertain is why this is cropping up all of a sudden. v1alpha1 has been deprecated for a while as far as I can tell, so I wonder if some logic in how Crossplane generates/reconciles resources has changed.

[bobh66]

@recht this might be a result of the problem described in crossplane/crossplane#6260 which is fixed by crossplane/crossplane#6261 in upcoming releases of 1.19 and 1.18