Ryan and I were discussing this previously and I wanted to share an idea that might help you move away from the current cleanup.sh script and instead leverage Kubernetes native garbage-collection features. Right now, your script checks for orphaned sessions and deletes them based on naming conventions. While it works, it’s not the most efficient or reliable in the long run, especially if something goes wrong with the cron job or session naming. We believe you can improve your workflow by incorporating ownerReferences in the resources you generate.

The key idea is to let Kubernetes do the cleanup automatically: once a parent Job is deleted, any child objects that reference that Job via ownerReferences are removed as well. You could implement logic similar to what you do with GPU scheduling in SessionJobBuilder#mergeAffinity. In that method, you're already parsing a YAML string, manipulating its data (for GPU affinity), and returning updated YAML. You can follow the same pattern to merge the UID from the starting Job into the session and ingress resources, which are generated around PostAction lines 648–674.

Specifically, once you create the Job, you’d grab its UID (likely via the Kubernetes Java client), and then inject that UID as an owner reference in each child resource. If you add something like a mergeOwnerReference method—mirroring how mergeAffinity works—it will parse the existing YAML, insert the owner reference details, and spit out an updated YAML to create the session and ingress. This should let you completely rely on Kubernetes for garbage collection and remove the need for a manual cleanup script.

Happy to discuss further if it is of interest / you'd likely clarification on anything above 😄

@sbathgate @rptaylor - Thanks so much for drawing our attention to this child objects feature--it sounds perfect for the job cleanup.sh is currently doing! I'll be turning your comments into a story on our side, so closing this issue off for now.