Provider keeps updating and creating objects

[recht]

What happened?

We recently upgraded from 0.9 to 0.16, and a while after that we modified an existing XR to add more Object resources. When applied, provider-kubernetes kept creating the objects over and over again with different names. At this point we also noticed that the generation field of all previous Objects had been incremented to around 14000, and a watch on one of the resources showed that it kept being updated every minute (no actual changes, just a write that triggered a new generation/resourceVersion).

We triaged the problem to be in 0.14.0 - any release before that only creates objects once and do not keep updating existing objects, any version after has that behavior.

Debug logging only showed Reconciling/Observing/Up to date!, so there's some indication that it's because it's either doing an update no matter what, or maybe it's updating the status?

This screenshot shows number of reconciles, and it's quite visible where we downgraded to 0.13.0.

How can we reproduce it?

Reproducing outside our environment is unclear, but the generation/resourceVersion of the Object k8s resources kept increasing every minute once the provider is at least v0.14.0.

What environment did it happen in?

Crossplane version: v0.17.3, provider-kubernetes v0.14.0-v0.16.0