Repository navigation
fix(klaud): verify older and aggregate baselines, free abandoned work / fix(klaud): 验证旧版与聚合基线并释放放弃的工作 - #4137
Conversation
…urvive install flakes - Producers older than the generator's --no-evals flag rejected it, so their baselines could never be verified and nine multi-node families were deferred every wave; retry without the flag and keep only throughput rows. - select now records why each candidate's baseline was deferred in selection.json and the step summary instead of discarding the error. - Both Klaud workflows retry the Claude CLI install once when the native binary is missing. - An attempt whose agent step never ran is retried next wave instead of cooling down for 24 hours. - 早于生成器 --no-evals 参数的 producer 会拒绝该参数，导致其基线无法验证， 九个多节点配置族每一轮都被推迟；现在会不带该参数重试，并只保留吞吐量行。 - select 现在会在 selection.json 和步骤摘要中记录每个候选基线被推迟的原因， 而不是丢弃错误。 - 两个 Klaud 工作流在缺少原生二进制时会重试一次 Claude CLI 安装。 - Agent 步骤从未运行的尝试会在下一轮重试，而不是冷却 24 小时。
…them The app mirrors an aggregate engine into both role columns, so a one-engine aggregate row publishes decode_num_workers equal to prefill's, while the matrix gives decode zero workers. Older result writers also published a one-worker aggregate as a single-node row. Either way no public row matched its producer entry, deferring nine agentic agg families with "No verified public baseline points". 应用会把聚合引擎镜像到 prefill 和 decode 两列，因此聚合行发布的 decode_num_workers 与 prefill 相同，而矩阵中 decode 为零个 worker；较旧的结果 写入器还会把单 worker 聚合发布为单节点行。两种情况下公开行都无法匹配其 producer 条目，导致九个 agentic agg 配置族因"No verified public baseline points"被推迟。
adibarra
changed the title
Sessions that hit the 6h job limit left draft PRs, klaud/auto branches and family claims behind, and recovery could not reconcile them, so their families stayed blocked indefinitely. Before selection, recovery now treats a candidate as abandoned once no planner session or final sweep is active: it cancels the candidate's leftover smoke runs, closes a draft PR that only Klaud committed to, deletes leftover branches of PRs Klaud itself closed and releases claims held by finished parents. Handoff, maintainer-touched and ready PRs are left alone. Recovery also bounds its targeted-run listing to the parent's lifetime, which exceeded GitHub's 1000-result cap for older parents. 达到 6 小时作业上限的会话会遗留草稿 PR、klaud/auto 分支和配置族声明，且恢复 流程无法收尾，导致这些配置族被无限期阻塞。现在恢复流程会在选择前，将没有活动 规划会话或最终 sweep 的候选视为已放弃：取消其残留的冒烟运行，关闭只有 Klaud 提交过的草稿 PR，删除 Klaud 自己关闭的 PR 遗留的分支，并释放已结束父运行持有 的声明。交接、维护者改动过或已就绪的 PR 保持不变。恢复流程还会将定向运行列表 限定在父运行的存续期内，此前较旧父运行的列表超过了 GitHub 1000 条结果上限。
adibarra
changed the title
…k sweep Each recovery pass leaves a klaud/recovery-<parent>-<candidate> ref behind and none was ever deleted (568 today). When no other planner is still reconciling, the sweep now deletes leases whose parent run finished and whose candidate is not active; a later pass recreates any it still needs. 每次恢复都会留下 klaud/recovery-<父运行>-<候选> 引用，且从未删除（目前有 568 个）。 当没有其他规划运行仍在收尾时，清扫现在会删除父运行已结束且候选不再活动的租约； 之后的恢复如仍需要会重新创建。
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| for pull in pulls: | ||
| match = re.fullmatch(rf"klaud/auto-({CANDIDATE_ID})", pull["head"]["ref"]) | ||
| if not match or match[1] in live: | ||
| continue | ||
| if ( | ||
| pull["draft"] | ||
| and pull["user"]["login"] == BOT | ||
| and (pull["head"]["repo"] or {}).get("full_name") == repository | ||
| and not any(label["name"] == "klaud-handoff" for label in pull["labels"]) | ||
| and bot_committed( | ||
| github.items(repository, f"pulls/{pull['number']}/commits?per_page=100") | ||
| ) | ||
| ): | ||
| abandoned.append(pull) |
There was a problem hiding this comment.
🔴 Klaud draft PRs a maintainer has re-swept, or whose recovery hit a transient error, now get closed and their branch deleted in the same recovery pass. The base left them open. sweep_abandoned only checks author, draft, labels and commit authorship. It ignores that reconcile() failed for this candidate moments earlier, for example with "New maintainer runs require explicit handoff". Fix: have recover() pass sweep_abandoned the ids whose reconcile raised anything but PendingCleanup, and skip their PR close, branch delete and claim release. Orphaned-smoke sessions still get freed.
Why this was flagged
Trigger: a session hits the 6h job limit and leaves a draft Klaud PR with a sweep label. A maintainer then starts a run-sweep.yml run on it with no commit. When that run ends, the next recover() calls reconcile(). Session.runs() raises VerificationError("New maintainer runs require explicit handoff") at lifecycle.py:136-140. The except at lifecycle.py:567-574 only adds the family to blocked, not the id to live. sweep_abandoned (lifecycle.py:654-667) then finds a draft, Klaud-authored PR with only github-actions[bot] commits and no klaud-handoff label. It comments "abandoned", closes the PR and deletes the branch (lifecycle.py:691-706), and releases the family claim (lifecycle.py:676-681). On the base branch the PR, branch and claim stayed intact and the family stayed blocked until a maintainer added a handoff label. The owned_activity() liveness check only covers run-sweep runs that are still active, so it does not prevent any of this.
Verification: Session.runs() raises VerificationError("New maintainer runs require explicit handoff") at lifecycle.py:136-140. The except at lifecycle.py:567-574 only does blocked.add(candidate.family). Unlike the lease and non-terminal branches (lines 557 and 563), it never adds candidate.id to live. The PR gets the "abandoned" comment, is PATCHed to closed, and its branch is DELETEd.
| for attempt in 1 2; do | ||
| rm -rf "$RUNNER_TEMP/claude-code" | ||
| npm install --prefix "$RUNNER_TEMP/claude-code" --no-audit --no-fund @anthropic-ai/claude-code@2.1.282 && | ||
| [ "$("$claude_cli" --version 2>&1)" = "2.1.282 (Claude Code)" ] && break |
There was a problem hiding this comment.
🟡 (optional) If claude --version ever writes to stderr, both Klaud workflows now fail the install step twice and no candidate starts. The base ignored stderr there. The new check merges stderr into the compared string with 2>&1, so any warning breaks the exact match against "2.1.282 (Claude Code)". Fix: compare only stdout, which the base already did, and send stderr to the log, e.g. [ "$("$claude_cli" --version)" = "2.1.282 (Claude Code)" ]. Apply this at both sites (klaud-candidate.yml:85, klaud-plan.yml:142).
Why this was flagged
This finding only applies if the pinned 2.1.282 CLI, its npm wrapper or Node prints anything to stderr on a successful --version, such as a deprecation, ExperimentalWarning or updater notice. The new loop at klaud-candidate.yml:85 and klaud-plan.yml:142 compares $("$claude_cli" --version 2>&1) with the exact string "2.1.282 (Claude Code)". Any stderr text makes the comparison fail, so both attempts fail and [ "$attempt" = 2 ] && exit 1 aborts the step. The base used test "$("$claude_cli" --version)" = ... and compared stdout only, so a working install with a stderr warning passed there. After this change, the claude step is skipped for every candidate, and the planner's overlap-review step is skipped too. The retry does not help because it repeats the same comparison.
Verification: In both klaud-candidate.yml:85 and klaud-plan.yml:142 the check is now [ "$("$claude_cli" --version 2>&1)" = "2.1.282 (Claude Code)" ] && break, which folds stderr into the string being compared. The base line captured only stdout. With any stderr text, both attempts fail, [ "$attempt" = 2 ] && exit 1 aborts the install step, and no candidate or planner session starts.
1 participant
Fixes found while monitoring the first planner run after #4118 (run 38802283618). That run approved 26 candidates and launched only 4: 22 were deferred at the baseline step, and
selection.jsondidn't say why. One of the 4 launched candidates then failed before the agent started.--no-evalsflag exit withunrecognized arguments: --no-evals, so their published baselines can never be verifieddsr1-fp8-b200-dynamo-trt,dsr1-fp8-h200-dynamo-trt,dsr1-fp4-gb200-dynamo-trt, …), producers from Marchproducer_matrixretries without the flag when the producer rejects it and keeps only throughput rows.InferenceX-apppackages/db/src/etl/topology.ts), so it publishesdecode_num_workersequal to prefill's while the matrix has 0.public_pointnow mirrors prefill for aggregates. Older result writers also published a one-worker aggregate as a single-node row (the 2026-09-18dsv4-*-dynamo-sglang-agentic-aggproducer did);_matches_public_pointaccepts that encoding only for one-worker aggregates.selection.jsongainsbaseline-deferral-reasons, and the step summary counts each reason. Verification errors keep their fixed public message, subprocess errors record only their type, and other errors record their type plus up to 300 characters.e920fad…in that run; 1 of the last 15 runsaction-outcome: skippedis retried next wave, like a capacity deferral. Attempts where the agent ran keep their cooldown.Abandoned work
Sessions that hit the 6h
ubuntu-latestjob limit (14 of the last 55) left draft PRs,klaud/auto-*branches and family claims behind. Recovery couldn't clean them up, because its targeted-run listing for older parents exceeded GitHub's 1000-result cap ("Incomplete GitHub listing"). The families stayed blocked: 40 of 42 claim refs belonged to finished parents.Recovery now sweeps before selection. A candidate is abandoned once no planner session (
Run candidate <id>job) and no final sweep (run-sweep.ymlon its branch) is active. For an abandoned candidate it:e2e-tests.ymlstarted by Klaud; nothing reads them once the session is gone);klaud-handofflabel, then deletes the branch;klaud/recovery-*lease refs whose parent run finished, when no other planner is still reconciling. These refs stop two recovery passes from cleaning up the same session at once; none were ever deleted, so 568 had piled up.Families blocked by recovery stay blocked only while their candidate is still active. Recovery's targeted-run listing is now limited to the parent's lifetime. A listing error skips the sweep for that wave instead of failing the planner.
Dry run against the live repo, with no writes: 11 abandoned drafts closed (#4081, #4082, #4091, #4092, #4093, #4095, #4096, #4098, #4120, #4122, #4124); 9 orphaned smoke runs cancelled, some of them queued since 04:35 UTC; 43
klaud/auto-*branches deleted, the stale claims released and 402 recovery leases deleted; recovery-blocked families down to 3 active ones. #4115 is kept because its final sweep is still running, and ready PRs are left for review.Replaying all 22 deferred candidates locally with their real producers: 17 now verify (previously 0). The remaining 5 are expected or not Klaud bugs:
glm5.2-fp4-b200-sglang-agentic-mtp,dsv4-fp4-b200-vllm-agentic-mtpTOTAL_CPU_DRAM_GBrecipe mismatch after #4125 (intentional)dsr1-fp8-gb200-dynamo-sglangkimik3-fp4-h200-vllm-agentic-balancedworkflow-info, so their producer can't be proven中文
本 PR 修复了在 #4118 之后首次规划运行（run 38802283618）中发现的问题：26 个候选通过评审，却只启动了 4 个，其余 22 个在基线步骤被推迟，
selection.json也没有记录原因；启动的 4 个候选中又有 1 个在 Agent 启动前失败。旧 producer 拒绝
--no-evals： 早于该参数的 producer 会以unrecognized arguments: --no-evals退出，其已发布基线永远无法验证；9 个多节点 DSR1 Dynamo 配置族受影响（producer 来自 3 月）。producer_matrix现在会在 producer 拒绝该参数时不带参数重试，并只保留吞吐量行。聚合多节点行无法匹配 producer 条目： 9 个 agentic agg 配置族报 "No verified public baseline points"。应用会把聚合引擎镜像到两个角色列（
InferenceX-app的packages/db/src/etl/topology.ts），因此发布的decode_num_workers与 prefill 相同，而矩阵中为 0。public_point现在对聚合配置镜像 prefill。较旧的结果写入器还会把单 worker 聚合发布为单节点行（2026-09-18 的dsv4-*-dynamo-sglang-agentic-aggproducer 即如此）；_matches_public_point仅对单 worker 聚合接受这种编码。基线推迟丢弃错误：
selection.json新增baseline-deferral-reasons，步骤摘要按原因计数。验证错误保留其固定的公开消息，子进程错误只记录类型，其他错误记录类型和最多 300 个字符。Claude CLI 安装偶发失败： 安装有时缺少原生二进制（"claude native binary not installed"），该运行中的候选
e920fad…即如此，最近 15 次运行中出现 1 次。两个 Klaud 工作流现在会在失败前重装一次。Agent 未启动的尝试被冷却 24 小时： diagnostics 记录
action-outcome: skipped的尝试现在会在下一轮重试，与容量推迟相同；Agent 已运行的尝试保持原有冷却。放弃的工作： 达到 6 小时
ubuntu-latest作业上限的会话（最近 55 次中有 14 次）会遗留草稿 PR、klaud/auto-*分支和配置族声明；恢复流程对较旧父运行的定向运行列表超过 GitHub 1000 条上限（"Incomplete GitHub listing"），无法收尾，配置族一直被阻塞（42 个声明中有 40 个属于已结束的父运行）。现在恢复流程会在选择前清扫：当没有规划会话（Run candidate <id>作业）和最终 sweep（其分支上的run-sweep.yml）在运行时，候选视为已放弃；取消其残留的冒烟运行，关闭只有 Klaud 提交且没有klaud-handoff标签的草稿 PR 并删除分支，删除 Klaud 自己关闭的 PR 遗留的分支（维护者关闭的保留），并释放已结束父运行的声明。当没有其他规划运行仍在收尾时，还会删除父运行已结束的klaud/recovery-*租约引用（此前从未删除，已累积 568 个）。恢复阻塞的配置族只在候选仍活动时保持阻塞。定向运行列表限定在父运行的存续期内；列表错误只会跳过本轮清扫。对线上仓库的只读演练：关闭 11 个放弃的草稿 PR，取消 9 个孤立的冒烟运行，删除 43 个klaud/auto-*分支并释放陈旧声明，恢复阻塞的配置族降至 3 个。用真实 producer 在本地重放全部 22 个被推迟的候选：现在有 17 个通过验证（此前为 0）。其余 5 个属于预期情况或并非 Klaud 缺陷：两个是 #4125 之后预期的
TOTAL_CPU_DRAM_GB配方不匹配；dsr1-fp8-gb200-dynamo-sglang有 3 个当前配置族不再生成的冻结基线点；kimik3-fp4-h200-vllm-agentic-balanced有 25 行来自workflow-info中缺失的 run 30781313910，无法证明其 producer。AI model disclosure
Prepared with
anthropic/claude-opus-5-5, which investigated the planner run, made the changes and tests, and drafted this description. No delegated agents were used.