Repository navigation
fix(klaud): unblock AgentX smoke and stop re-picking blocked candidates / fix(klaud): 修复 AgentX 冒烟测试并停止重复选择受阻候选 - #4075
Conversation
…ed candidates - Smoke eval rows no longer carry eval-all-concs, which the multi-node AgentX dispatch schema rejects; 65 live families failed get-jobs before any GPU work. - The cooldown now spans every base and skips candidates whose last attempt was not a capacity deferral, instead of only reordering same-SHA attempts. - The overlap index drops non-Klaud PRs idle for 7 days, and the review treats cross-cutting PRs as overlap only when they touch the family's image or flags. - Planner waves run every 2 hours instead of every 6. - 冒烟评测行不再携带 eval-all-concs，多节点 AgentX 调度 schema 会拒绝该字段； 此前 65 个现行配置族在任何 GPU 任务开始前就在 get-jobs 失败。 - 冷却期现在覆盖所有基准提交，并跳过最近一次尝试不是容量推迟的候选， 而不是只对同一 SHA 的尝试重新排序。 - 重叠索引会剔除 7 天无活动的非 Klaud PR；评审仅在跨领域 PR 修改该配置族的 镜像或引擎参数时才视为重叠。 - 规划任务改为每 2 小时运行一次，而不是每 6 小时。
- 删除新增的注释。
|
Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package. 中文大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要，改动通常也很小。请让你的 agent 清理 PR，使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| if row.get("prefill") is not None: | ||
| row["conc"] = [row["eval-conc"]] | ||
| row["eval-all-concs"] = False | ||
| row.pop("eval-all-concs", None) |
There was a problem hiding this comment.
📊 Line Count Report
- Total Lines: 1906
- Base Lines: 1906
- Change: ➡️ +0 lines
There was a problem hiding this comment.
Beyond the inline finding, I also checked whether recorded_outcome()'s sequential gh api downloads in recent_candidate_outcomes could stall a planning wave — each call has a 60s subprocess timeout, but under normal network conditions the total wait stays well under the 2h cron cadence, so this reads as a performance characteristic rather than a bug.
Extended reasoning...
The change reworks Klaud Cold's candidate-selection cooldown (inferencex-e2e/infx/klaud/main.py: recent_candidate_outcomes, recorded_outcome, order_by_cooldown, active_pulls), a matrix-generation fix (smoke_entries popping eval-all-concs), and workflow cadence/prompt tuning (.github/workflows/klaud-plan.yml); it touches no auth/crypto surface but recorded_outcome() does parse downloaded GitHub Actions artifacts via subprocess gh calls bounded by VerificationError checks. The already-confirmed inline finding (recorded_outcome swallowing transient download/parse errors into outcome=None, which silently drops a candidate from both the fresh and retry lists for up to cooldown_hours) is a plausible real regression worth a human look. I additionally traced the sequential per-candidate download loop for a wave-stalling concern and ruled it out as bounded/non-blocking under normal conditions given the 60s per-call timeout and 2h cadence.
| def recorded_outcome(repository: str, artifact: dict) -> str | None: | ||
| with tempfile.TemporaryDirectory() as scratch: | ||
| try: | ||
| download_json(repository, artifact, Path(scratch)) | ||
| outcome = json.loads((Path(scratch) / "candidate-diagnostics.json").read_text())[ | ||
| "candidate-outcome" | ||
| ] | ||
| return CandidateOutcome.model_validate(outcome).outcome | ||
| except ( | ||
| OSError, | ||
| ValueError, | ||
| KeyError, | ||
| TypeError, | ||
| subprocess.SubprocessError, | ||
| ): | ||
| return None | ||
|
|
||
|
|
||
| def order_by_cooldown(candidates: list[dict], recent: dict[str, str | None]) -> list[dict]: | ||
| fresh = [candidate for candidate in candidates if candidate["id"] not in recent] | ||
| deferred = [ | ||
| candidate for candidate in candidates if recent.get(candidate["id"], "") in RETRY_NEXT_WAVE | ||
| ] | ||
| return fresh + deferred |
There was a problem hiding this comment.
🔴 A transient failure downloading or parsing a candidate's outcome artifact now silently blocks that candidate from the next ~cooldown_hours of waves. recorded_outcome() (L202-217) returns None on any OSError/ValueError/KeyError/TypeError/SubprocessError from download_json or from reading candidate-diagnostics.json, and order_by_cooldown() (L220-225) treats id->None the same as a real 'blocked' outcome: it's excluded from both fresh and deferred, so the candidate disappears from review until the artifact ages out of the cooldown window. Fix: distinguish 'outcome unknown/fetch failed' from a real blocking outcome so transient failures don't exclude a candidate for the full cooldown window; e.g. keep such ids out of recent instead of mapping them to None.
Why this was flagged
Trigger: a gh api zip download for one candidate's artifact times out, hits a rate limit, or the zip/json is malformed (any OSError/ValueError/SubprocessError), which recorded_outcome() at main.py:210-217 catches and turns into outcome=None instead of propagating or dropping the id. order_by_cooldown() (L220-225) requires recent.get(id) in RETRY_NEXT_WAVE to retry; None fails that test just like a genuine 'readiness-blocked' outcome, so the candidate is dropped from both fresh and deferred lists for the whole cooldown_hours window (24h in the workflow). On base branch, the only failure path was the outer github_items call in recent_candidate_ids, which failed open (returned empty set, candidates unaffected); this diff adds a per-artifact network/parse path with no equivalent fail-open behavior, so legitimate candidates can be starved of review for up to 24h by an unrelated GitHub API hiccup, with waves now running every 2h (klaud-plan.yml cron) increasing total download volume and therefore the odds of hitting this.
Verification: nit. recorded_outcome() (main.py:202-217) catches OSError/ValueError/KeyError/TypeError/subprocess.SubprocessError and returns None; download_json() (github.py:54-82) raises these on gh api timeout, non-zero exit, or malformed zip. The id is still stored in recent (L198) with value None. order_by_cooldown (L220-225) then excludes it from both fresh and deferred, dropping it like a genuine blocked outcome.
1 participant
Klaud Cold has been opening image-bump PRs that close without landing. Its planner runs are green. The p90 oldest AgentX refresh is at 53 days. Four causes, all in the selection path:
smoke_entriesbegan stampingeval-all-concs: falseonto multi-node AgentX eval rows, andMultiNodeAgenticMatrixEntryrejects that field.get-jobsdied withextra_forbidden. On current main, 65 live families fail the dispatch schema in smoke mode; with this change, none do. Affected Klaud PRs include [Klaud Cold] Update Qwen3.5 FP8 B300 Dynamo-SGLang AgentX disagg image to v0.5.21-cu130 / [Klaud Cold] 将 Qwen3.5 FP8 B300 Dynamo-SGLang AgentX 分离式镜像更新至 v0.5.21-cu130 #3969, [Klaud Cold] Update Qwen3.5 FP8 B300 Dynamo-SGLang AgentX disagg image to SGLang v0.5.21-cu130 / [Klaud Cold] 将 Qwen3.5 FP8 B300 Dynamo-SGLang AgentX 分离式镜像更新至 SGLang v0.5.21-cu130 #4068, [Klaud Cold] Update Qwen3.5 FP8 GB300 Dynamo-SGLang PP AgentX image to nightly-dev-cu13-20261014 / [Klaud Cold] 将 Qwen3.5 FP8 GB300 Dynamo-SGLang PP AgentX 镜像更新至 nightly-dev-cu13-20261014 #4061, [Klaud Cold] Update GB300 MiniMax-M3 Dynamo vLLM disagg AgentX image to v0.31.0 / [Klaud Cold] 将 GB300 MiniMax-M3 Dynamo vLLM 分离式 AgentX 镜像更新至 v0.31.0 #4057, [Klaud Cold] Update GB300 MiniMax-M3 Dynamo vLLM AgentX image to v0.31.0 / [Klaud Cold] 将 GB300 MiniMax-M3 Dynamo vLLM AgentX 镜像更新至 v0.31.0 #4049 and [Klaud Cold] Update dsv4-fp4-b200-dynamo-sglang-agentic-disagg SGLang image to v0.5.21 / [Klaud Cold] 将 dsv4-fp4-b200-dynamo-sglang-agentic-disagg 的 SGLang 镜像更新至 v0.5.21 #3957.mainSHA, and main moves faster than the 6h cadence, so it almost never applied. Even when it did, it only reordered. DSR1 B200 (missing weights on b200-nscale) andqwen3.5-fp8-b300-dynamo-sglang-agentic-disaggwere retried 4 to 10 times. The cooldown now reads each recent attempt's verified outcome across all bases. A capacity deferral still retries next wave; anything else waits 24h.duplicatebecause of open PRs with no activity for weeks: 8[Klaud Cold]PRs from Sept 18 to 28 ([Klaud Cold] Update qwen3.5-fp4-b300-sglang (+mtp) SGLang image to nightly-dev-cu13-20260925-8ca82118 #3441, [Klaud Cold] Update qwen3.5-fp4-b300-sglang-agentic-mtp SGLang image to nightly-dev-cu13-20260925-8ca82118 #3454, [Klaud Cold] Update GLM-5.2 B300 SGLang AgentX (FP4 + FP8) image to nightly-dev-cu13-20260918-20518d85 / 将 GLM-5.2 B300 SGLang AgentX（FP4 + FP8）镜像更新至 nightly-dev-cu13-20260918-20518d85 #3276, [Klaud Cold] Update GLM-5.2 B200 SGLang AgentX (FP4 + FP8) image to nightly-dev-cu13-20260918-20518d85 / 将 GLM-5.2 B200 SGLang AgentX（FP4 + FP8）镜像更新至 nightly-dev-cu13-20260918-20518d85 #3275, [Klaud Cold] Update qwen3.5-fp8-b200-sglang (+mtp) SGLang image to nightly-dev-cu13-20260925-8ca82118 #3438, [Klaud Cold] Update qwen3.5-fp8-h200-sglang (+mtp) to SGLang nightly-dev-cu13-20260928 / [Klaud Cold] 将 qwen3.5-fp8-h200-sglang（含 MTP）更新至 SGLang nightly-dev-cu13-20260928 #3535, [Klaud Cold] Add B300 DeepSeek V4.1 Flash SGLang DSpark AgentX / 新增 B300 DeepSeek V4.1 Flash SGLang DSpark AgentX 配方 #3242, [Klaud Cold] Update qwen3.5-fp8-gb300-dynamo-sglang-mtp SGLang image to nightly-dev-cu13-20260918-20518d85 / 将 qwen3.5-fp8-gb300-dynamo-sglang-mtp 的 SGLang 镜像更新至 nightly-dev-cu13-20260918-20518d85 #3255). The planner now drops non-Klaud PRs idle for 7 days from the overlap index. Klaud's own candidate PRs always count. Today that removes 110 of the 204 open PRs from the index.The planner also runs every 2h instead of every 6h. In today's 18:11 UTC run, 15 of the 22 candidates that cleared review were dropped as capacity-deferred, so more frequent waves catch short idle windows. The cap stays at 5 per wave, and the existing capacity gate and family claims still bound load. Review cost is about $2 to $6 per wave.
Not fixed here: DSR1 weights (
/scratch/models/DeepSeek-R1-0528,DeepSeek-R1-0528-NVFP4-v2) are missing on b200-nscale, so those bumps stay readiness-blocked until the weights are restored. They now retry once a day instead of every wave.Risk: an idle human PR no longer blocks a family, so a Klaud refresh can land first and force that PR to rebase.
中文
Klaud Cold 的规划任务运行正常，但它开出的镜像升级 PR 大多未合并就被关闭；AgentX 最旧刷新的 p90 已达 53 天。原因有四个，都在候选选择路径上：
smoke_entries开始给多节点 AgentX 评测行写入eval-all-concs: false，而MultiNodeAgenticMatrixEntry不允许该字段，导致get-jobs以extra_forbidden失败。当前 main 上有 65 个现行配置族的冒烟矩阵无法通过调度 schema；本 PR 之后为 0。受影响的 Klaud PR 包括 [Klaud Cold] Update Qwen3.5 FP8 B300 Dynamo-SGLang AgentX disagg image to v0.5.21-cu130 / [Klaud Cold] 将 Qwen3.5 FP8 B300 Dynamo-SGLang AgentX 分离式镜像更新至 v0.5.21-cu130 #3969、[Klaud Cold] Update Qwen3.5 FP8 B300 Dynamo-SGLang AgentX disagg image to SGLang v0.5.21-cu130 / [Klaud Cold] 将 Qwen3.5 FP8 B300 Dynamo-SGLang AgentX 分离式镜像更新至 SGLang v0.5.21-cu130 #4068、[Klaud Cold] Update Qwen3.5 FP8 GB300 Dynamo-SGLang PP AgentX image to nightly-dev-cu13-20261014 / [Klaud Cold] 将 Qwen3.5 FP8 GB300 Dynamo-SGLang PP AgentX 镜像更新至 nightly-dev-cu13-20261014 #4061、[Klaud Cold] Update GB300 MiniMax-M3 Dynamo vLLM disagg AgentX image to v0.31.0 / [Klaud Cold] 将 GB300 MiniMax-M3 Dynamo vLLM 分离式 AgentX 镜像更新至 v0.31.0 #4057、[Klaud Cold] Update GB300 MiniMax-M3 Dynamo vLLM AgentX image to v0.31.0 / [Klaud Cold] 将 GB300 MiniMax-M3 Dynamo vLLM AgentX 镜像更新至 v0.31.0 #4049 和 [Klaud Cold] Update dsv4-fp4-b200-dynamo-sglang-agentic-disagg SGLang image to v0.5.21 / [Klaud Cold] 将 dsv4-fp4-b200-dynamo-sglang-agentic-disagg 的 SGLang 镜像更新至 v0.5.21 #3957。mainSHA 的规划运行，而 main 的更新频率高于 6 小时一轮，因此几乎从未生效；即使生效也只是调整顺序。DSR1 B200（b200-nscale 缺少权重）和qwen3.5-fp8-b300-dynamo-sglang-agentic-disagg因此被重试了 4 到 10 次。现在冷却期会跨所有基准提交读取每次近期尝试的已验证结果：容量推迟仍在下一轮重试，其他结果需等待 24 小时。duplicate，例如 9 月 18 日至 28 日的 8 个[Klaud Cold]PR（[Klaud Cold] Update qwen3.5-fp4-b300-sglang (+mtp) SGLang image to nightly-dev-cu13-20260925-8ca82118 #3441、[Klaud Cold] Update qwen3.5-fp4-b300-sglang-agentic-mtp SGLang image to nightly-dev-cu13-20260925-8ca82118 #3454、[Klaud Cold] Update GLM-5.2 B300 SGLang AgentX (FP4 + FP8) image to nightly-dev-cu13-20260918-20518d85 / 将 GLM-5.2 B300 SGLang AgentX（FP4 + FP8）镜像更新至 nightly-dev-cu13-20260918-20518d85 #3276、[Klaud Cold] Update GLM-5.2 B200 SGLang AgentX (FP4 + FP8) image to nightly-dev-cu13-20260918-20518d85 / 将 GLM-5.2 B200 SGLang AgentX（FP4 + FP8）镜像更新至 nightly-dev-cu13-20260918-20518d85 #3275、[Klaud Cold] Update qwen3.5-fp8-b200-sglang (+mtp) SGLang image to nightly-dev-cu13-20260925-8ca82118 #3438、[Klaud Cold] Update qwen3.5-fp8-h200-sglang (+mtp) to SGLang nightly-dev-cu13-20260928 / [Klaud Cold] 将 qwen3.5-fp8-h200-sglang（含 MTP）更新至 SGLang nightly-dev-cu13-20260928 #3535、[Klaud Cold] Add B300 DeepSeek V4.1 Flash SGLang DSpark AgentX / 新增 B300 DeepSeek V4.1 Flash SGLang DSpark AgentX 配方 #3242、[Klaud Cold] Update qwen3.5-fp8-gb300-dynamo-sglang-mtp SGLang image to nightly-dev-cu13-20260918-20518d85 / 将 qwen3.5-fp8-gb300-dynamo-sglang-mtp 的 SGLang 镜像更新至 nightly-dev-cu13-20260918-20518d85 #3255）。现在规划器会从重叠索引中剔除 7 天无活动的非 Klaud PR，Klaud 自己的候选 PR 始终保留。按今天的数据，索引中 204 个开放 PR 会剔除 110 个。规划任务也从每 6 小时改为每 2 小时运行一次。在今天 18:11 UTC 的运行中，通过评审的 22 个候选里有 15 个因容量被推迟，更频繁的轮次可以抓住短暂的空闲窗口。每轮上限仍为 5 个，现有的容量检查和配置族认领仍会限制负载。每轮评审成本约 2 至 6 美元。
本 PR 未解决： b200-nscale 上缺少 DSR1 权重（
/scratch/models/DeepSeek-R1-0528、DeepSeek-R1-0528-NVFP4-v2），在恢复权重之前这些升级仍为 readiness-blocked；现在它们每天只重试一次，而不是每轮都重试。风险： 长期无活动的人工 PR 不再锁住配置族，因此 Klaud 的刷新可能先合并，迫使该 PR 重新 rebase。
AI model disclosure
Prepared with
anthropic/claude-opus-5-5, which investigated the Klaud runs and PRs, wrote the code and tests, and drafted this description. No delegated agents were used.