[BUG] v2: run absent from UI but its TaskAction CR reconciles forever, recreating the task pod #7926

Activity

1. runllm commented on Aug 27, 2026

   

   runllmBot

   on Aug 27, 2026 – with RunLLM

   More actions

   It could be possible that this issue arises from a desynchronization between the control plane (Run Service), the TaskAction controller, and the backend worker, especially when the run is no longer present in the control plane but TaskAction CRs persist and are continually reconciled. According to the Flyte v2 architecture, when a run is terminal or deleted, all in-progress descendant actions (TaskAction CRs) should be automatically aborted and garbage-collected, and the controller should not keep recreating pods. However, if the informer or event stream between the control plane and the controller is interrupted or the run is missing from the control plane, the TaskAction controller may not receive the signal to finalize or abort the CR, leading to indefinite reconciliation and pod recreation. This is hinted at in the life of a run and abort/cancel actions documentation.

   Though I'm not entirely certain, it might be that the controller should detect when the root run is missing or terminal in the control plane and then mark all associated TaskAction CRs as terminal (e.g., ABORTED or FAILED), triggering garbage collection and stopping pod recreation. The current behavior you describe—where the TaskAction CR cycles phases and the pod is recreated—suggests this detection or reconciliation is not happening as expected, possibly due to a dropped informer watch or a missed event. There is no explicit mention in the data sources of a built-in mechanism for the controller to forcibly finalize orphaned TaskActions when the run is missing, but this would be a reasonable enhancement or bugfix for the controller logic.

   Would you like a step-by-step breakdown of how Flyte's run and action lifecycle is supposed to work, or more detail on possible controller-side mitigations?

   Sources:

   • life of a run
   • abort/cancel actions

   ---

   This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.

   Mention @Herald to ask follow-up questions. See runllm.com for more!
2. ChickenBenny commented on Oct 13, 2026

   

   ChickenBenny

   on Oct 13, 2026

   Contributor

   More actions

   Hi, I'd like to work on this. I'll start by trying to reproduce it locally and will share what I find here.
3. ChickenBenny commented on Oct 13, 2026

   

   ChickenBenny

   on Oct 13, 2026

   Contributor

   More actions

   I dug into this and opened #8172 as a first fix.

   What the PR does: when a driver pod gets killed and restarts, it can try to re-create child actions that already exist. The task path in Enqueue returned INTERNAL for that instead of ALREADY_EXISTS, so the SDK kept retrying for ~17 min and then marked the child as failed. The PR makes it a no-op, same as the condition path already does.

   One thing I noticed while reproducing: this doesn't happen on every restart. On my local devbox I killed the driver, and the pod came back in the same attempt as expected, but the new driver got the existing children from the watch snapshot and never re-enqueued them, so the run just finished fine. It looks like you only hit the already exists error if the watch also fails around that time, which would match the Informer watch retrying / unexpected EOF lines in your logs. I couldn't reproduce that part locally yet, so I'll keep looking into it.

   Not covered yet, and I'll follow up separately:

   • the pod being recreated forever after SIGKILL (the system retry counter keeps resetting)
   • why the run disappeared from the UI while the CR was still running

   @earlybard if you still have it, the CR's full status and the a0 row from the actions table would really help with the last one.
4. ChickenBenny commented on Oct 14, 2026

   

   ChickenBenny

   on Oct 14, 2026

   Contributor

   More actions

   Follow-up on the pod being recreated forever. I reproduced it locally and found the cause. I'd like to agree on the approach before opening a PR, since it changes how system failures are handled.

   (Correction to my last comment: #8172 now returns ALREADY_EXISTS to the SDK instead of a no-op.)

   Repro

   A task that sleeps for an hour, no children. I deleted its pod 5 times with kubectl delete pod --force, waiting for Running before each kill:

   initial: attempts=1 systemFailures=0 systemRetries=0 phase=Running
   kill #1: attempts=1 systemFailures=0 systemRetries=1 phase=Running
   kill #2: attempts=1 systemFailures=0 systemRetries=2 phase=Running
   kill #3: attempts=1 systemFailures=0 systemRetries=3 phase=Running
   kill #4: attempts=1 systemFailures=0 systemRetries=4 phase=Running   <- expected PermanentFailure here (maxSystemFailures=3)
   kill #5: attempts=1 systemFailures=0 systemRetries=5 phase=Running
   

   Every kill was recorded as a system error (UnexpectedObjectDeletion), but the pod was recreated each time and the task never failed, even after maxSystemFailures (3).

   Cause

   A pod deletion (plugin_manager.go#L278) or SIGKILL (pod_helper.go#L1657) is a system-retryable failure, so the controller relaunches the pod in place (taskaction_controller.go#L882-L884): SystemFailures goes 0 → 1 and the plugin state is cleared. On the next reconcile, the plugin launches a new pod and returns Queued. That is not a system error, so #L897 resets SystemFailures to 0.

   SystemRetries keeps growing but nothing checks it, and Attempts doesn't change, so nothing bounds the loop.

   Problem 1: the counter resets before any progress

   The #8070 test (#L1522) says "the progress in between puts the consecutive failure count back to zero". But the reset happens on the relaunch's own Queued, so even a pod killed before it ever runs is relaunched forever.

   Fix: only reset SystemFailures once the attempt reaches Running or later.

   Side effect: transient errors from Plugin.Handle before Running would also start to accumulate. We could apply the new rule everywhere (A), or only after a relaunch (B).

   Problem 2: no bound when the pod makes progress between kills

   That is my repro and the original report, so Problem 1 alone doesn't fix it. #8070 seems to allow repeated non-consecutive failures on purpose (e.g. Kueue preemptions), so this is a policy question.

   Possible fix: a separate maxSystemRetries on total in-place relaunches, configurable, possibly not counting preemptions.

   Questions

   1. Do you want a total bound on in-place relaunches (Problem 2)? If so, what default, and should preemptions count?
   2. For Problem 1, is Running the right definition of progress, and A or B?
   3. One PR for both, or Problem 1 first?

   cc @AdilFayyaz and @Sovietaced. Would appreciate your thoughts.
5. ChickenBenny commented on Oct 24, 2026

   

   ChickenBenny

   on Oct 24, 2026

   Contributor

   More actions

   Hi @earlybard, I opened #8237 for the pod being recreated forever.

   For the last part (the run disappearing from the UI while the TaskAction CR was still running), I can't reproduce it locally yet. A few quick questions would help narrow it down, even without logs:

   1. When you say the run disappeared, was it missing from the runs list, or did opening the run's URL directly also fail? If it opened, what status did it show?
   2. Was the run (or any action in it) aborted from the UI/CLI before it disappeared?
   3. If the cluster is still around, the output of:
      kubectl get taskaction <name> -n <ns> -o jsonpath='{.metadata.labels}{"\n"}{.status.conditions}'
6. earlybard commented on Oct 29, 2026

   

   earlybard

   on Oct 29, 2026

   ContributorAuthor

   More actions

   Thanks for digging into this. The TaskAction CRs are gone (I deleted them by run label to unblock the node drain), so I can't give you (3). But the control plane still has the run and our Loki still has that day's logs, and together they answer most of it.

   1. It didn't disappear. That was my mistake in the report. The run still opens by URL / flyte get run, and it shows ACTION_PHASE_ABORTED. It had dropped out of the running view, which is where I was looking.

   2. Nobody aborted it, as far as I can tell. a0 finished normally, and the pod that ran for 17 days was a re-run of a0, started right after a backend restart. Timeline in UTC, 2026-08-10 unless noted, from flyte get run and Loki:

   time	source	event
   04:07:32	a0 container	Flyte runtime started for action a0 (schedule trigger s2_every_3h, SDK 2.5.17, backend v2.0.42)
   04:42:04	a0 container	Flyte runtime completed for action a0
   04:42:06.284	run record	attempt 1 → SUCCEEDED
   04:42:06.330	run record	run → ABORTED, abortInfo: {}
   05:17:56, 05:47:56	backend	Reconciler error … taskactions.flyte.org "<run>-<action>" not found for ~90 of the run's CRs, including <run>-a0
   05:47:57	run record (cluster event)	aborted
   06:03:46	backend	backend pod shuts down for a deploy
   06:03:58	k8s events	<run>-a0-0 created again, on a new node
   06:04:06	a0 container	Flyte runtime started for action a0: the task body runs again from the start. WatchForUpdates/Enqueue get connection refused while the new backend comes up, then Informer cache sync timed out
   06:30:15	a0 container	started a third time
   → 08-26		still running until I deleted the CRs

   What looks wrong:

   • The run went ABORTED with an empty abortInfo 2 s after a0 completed on its own, with no abort from us.
   • After the restart, a0's CR came back even though it had already been deleted (not found at 05:47), and the task re-ran for a run that was already terminal.
   • The re-run asked for its children again under the same deterministic IDs (e.g. 4324w5siuqbm5tvgxi79re0in ran in the first pass), which is where the already exists loop in my report came from. The informer-watch errors were the re-run losing the backend during the restart, not the cause.

   So the trigger may be "restart the backend shortly after a run finishes" rather than a dropped watch. I haven't tried to reproduce it that way yet.

   Happy to share the full get run output or more log excerpts if it helps.