Orphan secondary resources are left when using annotation based owner reference #2259
Comments
|
Hi @AleksanderBrzozowski , thx for the issue,
Actually, this should be the case. Will take a look. |
|
This is the code part: |
|
Yeah, I saw this code part. But it seems that in some cases it doesn't work correctly. Have you tried running the test from my sample repo? 🙂 |
|
reproduce it, seems that the cleanup is not called when it fails (at least in my case), while the finalizer is added. If you log the resource in the reconciler the finalizer is always there. |
|
Notes: it theoretically can happen that the ConfigMap is created but on cleanup is not in the cache yet, so it will not be deleted (this is a limitation and issue with this method - when resources are in different NS), but dependent resource mitigate / eliminate this issue, so in this : https://javaoperatorsdk.io/docs/dependent-resources#caching-and-event-handling-in-kubernetesdependentresource So event this should not be the case. |
|
Not sure that I follow 😛 So I guess that you were able to reproduce the issue by running a test? Are you saying that we can somehow fix this issue by modifying the code? The |
No, not saying that :) still investigating, will probably take a while |
|
No need to rush. We don't have this problem in production, we only discovered it when running integration tests, and we were able to solve it 🙂 |
|
interesting is that I created a reproducer now in pure Java and not using Quarkus and not able to reproduce there: |
|
The problem seems to be that there is a |
|
The problem seems to be with either k8s API or informer: As can be seen here we receive a delete event for a resource that still has a finalizer. |
|
Full log of a failing test case: |
|
Why the test case fails:
|
Just to make sure, are there any labels or other filters that the informer is based off of? If that's the case you will see a delete event when any of that no longer matches, not just when the resource no longer exists. |
|
No there are no labels involved. client.resource(resource).create()
await().untilAsserted{
val res = client.resources(MyCustomResource::class.java)
.inNamespace("default")
.withName("test")
.get();
assertEquals(res.metadata.finalizers.size,1)
}
Thread.sleep(5)
// delete the primary, and wait until it is deleted
client.resources(MyCustomResource::class.java)
.inNamespace("default")
.withName("test")
.withTimeout(5, TimeUnit.SECONDS)
.delete() |
|
Btw AFAIK |
|
asked also for clarification on k8s slack: https://kubernetes.slack.com/archives/C0EG7JC6T/p1709561696968339 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Bug Report
What did you do?
We have a primary resource, let's call it
MyCustomResource. The reconciler for this resource creates and maintains aConfigMapbased on theMyCustomResource. TheConfigMapis created in a different namespace thanMyCustomResource, thus we cannot use theownerReferencemechanism to ensure that the resource is deleted when the primary resource is deleted. To do so, we use JOSDK annotations (primary-nameandprimary-namespace).The issue is that sometimes when the primary resource is deleted very quickly - right after it was created, the secondary resource is not created:
So it seems that the secondary resource is created, even though the primary resource has no finalizer added.
This issue was reproduced with a sample repo. There is a repeated test that fails after several test loops - it is non deterministic.
What did you expect to see?
There are no Orphan secondary resources left, when the Primary is deleted.
What did you see instead? Under which circumstances?
There is an Orphan secondary resource left, that will be never deleted.
Environment
Kubernetes cluster type:
rancher desktop, version: 1.25
$ Mention java-operator-sdk version from pom.xml file4.8.0$ java -versionJava Temurin 17.0.6$ kubectl versionkubectl version doesn't seem to be related, under the hood JOSDK is using fabric8 kubernetes client with version
6.10.0Possible Solution
Make sure that the finalizer is applied first before creating secondary resources
The text was updated successfully, but these errors were encountered: