Skip to content

Commit 4badac6

Browse files
authored
fix(klaud): verify older and aggregate baselines, free abandoned work / fix(klaud): 验证旧版与聚合基线并释放放弃的工作 (#4137)
* fix(klaud): regenerate pre-flag producers, record deferral reasons, survive install flakes - Producers older than the generator's --no-evals flag rejected it, so their baselines could never be verified and nine multi-node families were deferred every wave; retry without the flag and keep only throughput rows. - select now records why each candidate's baseline was deferred in selection.json and the step summary instead of discarding the error. - Both Klaud workflows retry the Claude CLI install once when the native binary is missing. - An attempt whose agent step never ran is retried next wave instead of cooling down for 24 hours. - 早于生成器 --no-evals 参数的 producer 会拒绝该参数,导致其基线无法验证, 九个多节点配置族每一轮都被推迟;现在会不带该参数重试,并只保留吞吐量行。 - select 现在会在 selection.json 和步骤摘要中记录每个候选基线被推迟的原因, 而不是丢弃错误。 - 两个 Klaud 工作流在缺少原生二进制时会重试一次 Claude CLI 安装。 - Agent 步骤从未运行的尝试会在下一轮重试,而不是冷却 24 小时。 * fix(klaud): match aggregate multinode baselines as the app publishes them The app mirrors an aggregate engine into both role columns, so a one-engine aggregate row publishes decode_num_workers equal to prefill's, while the matrix gives decode zero workers. Older result writers also published a one-worker aggregate as a single-node row. Either way no public row matched its producer entry, deferring nine agentic agg families with "No verified public baseline points". 应用会把聚合引擎镜像到 prefill 和 decode 两列,因此聚合行发布的 decode_num_workers 与 prefill 相同,而矩阵中 decode 为零个 worker;较旧的结果 写入器还会把单 worker 聚合发布为单节点行。两种情况下公开行都无法匹配其 producer 条目,导致九个 agentic agg 配置族因"No verified public baseline points"被推迟。 * feat(klaud): free abandoned candidate PRs, branches and claims each wave Sessions that hit the 6h job limit left draft PRs, klaud/auto branches and family claims behind, and recovery could not reconcile them, so their families stayed blocked indefinitely. Before selection, recovery now treats a candidate as abandoned once no planner session or final sweep is active: it cancels the candidate's leftover smoke runs, closes a draft PR that only Klaud committed to, deletes leftover branches of PRs Klaud itself closed and releases claims held by finished parents. Handoff, maintainer-touched and ready PRs are left alone. Recovery also bounds its targeted-run listing to the parent's lifetime, which exceeded GitHub's 1000-result cap for older parents. 达到 6 小时作业上限的会话会遗留草稿 PR、klaud/auto 分支和配置族声明,且恢复 流程无法收尾,导致这些配置族被无限期阻塞。现在恢复流程会在选择前,将没有活动 规划会话或最终 sweep 的候选视为已放弃:取消其残留的冒烟运行,关闭只有 Klaud 提交过的草稿 PR,删除 Klaud 自己关闭的 PR 遗留的分支,并释放已结束父运行持有 的声明。交接、维护者改动过或已就绪的 PR 保持不变。恢复流程还会将定向运行列表 限定在父运行的存续期内,此前较旧父运行的列表超过了 GitHub 1000 条结果上限。 * feat(klaud): delete finished recovery leases during the abandoned-work sweep Each recovery pass leaves a klaud/recovery-<parent>-<candidate> ref behind and none was ever deleted (568 today). When no other planner is still reconciling, the sweep now deletes leases whose parent run finished and whose candidate is not active; a later pass recreates any it still needs. 每次恢复都会留下 klaud/recovery-<父运行>-<候选> 引用,且从未删除(目前有 568 个)。 当没有其他规划运行仍在收尾时,清扫现在会删除父运行已结束且候选不再活动的租约; 之后的恢复如仍需要会重新创建。
1 parent 77a21c0 commit 4badac6

7 files changed

Lines changed: 386 additions & 34 deletions

File tree

‎.github/workflows/klaud-candidate.yml‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -78,9 +78,14 @@ jobs:
7878
- name: Install Claude Code 2.1.282
7979
id: claude_cli
8080
run: | # zizmor: ignore[adhoc-packages] Claude CLI and its native packages are pinned to 2.1.282
81-
npm install --prefix "$RUNNER_TEMP/claude-code" --no-audit --no-fund @anthropic-ai/claude-code@2.1.282
8281
claude_cli="$RUNNER_TEMP/claude-code/node_modules/.bin/claude"
83-
test "$("$claude_cli" --version)" = "2.1.282 (Claude Code)"
82+
for attempt in 1 2; do
83+
rm -rf "$RUNNER_TEMP/claude-code"
84+
npm install --prefix "$RUNNER_TEMP/claude-code" --no-audit --no-fund @anthropic-ai/claude-code@2.1.282 &&
85+
[ "$("$claude_cli" --version 2>&1)" = "2.1.282 (Claude Code)" ] && break
86+
[ "$attempt" = 2 ] && exit 1
87+
sleep 15
88+
done
8489
echo "path=$claude_cli" >> "$GITHUB_OUTPUT"
8590
- name: Let Klaud Cold own the candidate PR
8691
id: claude

‎.github/workflows/klaud-plan.yml‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -136,9 +136,14 @@ jobs:
136136
id: claude_cli
137137
if: steps.prepare.outputs.has_candidates == 'true'
138138
run: | # zizmor: ignore[adhoc-packages] Claude CLI and its native packages are pinned to 2.1.282
139-
npm install --prefix "$RUNNER_TEMP/claude-code" --no-audit --no-fund @anthropic-ai/claude-code@2.1.282
140139
claude_cli="$RUNNER_TEMP/claude-code/node_modules/.bin/claude"
141-
test "$("$claude_cli" --version)" = "2.1.282 (Claude Code)"
140+
for attempt in 1 2; do
141+
rm -rf "$RUNNER_TEMP/claude-code"
142+
npm install --prefix "$RUNNER_TEMP/claude-code" --no-audit --no-fund @anthropic-ai/claude-code@2.1.282 &&
143+
[ "$("$claude_cli" --version 2>&1)" = "2.1.282 (Claude Code)" ] && break
144+
[ "$attempt" = 2 ] && exit 1
145+
sleep 15
146+
done
142147
echo "path=$claude_cli" >> "$GITHUB_OUTPUT"
143148
- name: Check for overlapping open PRs
144149
id: review

‎inferencex-e2e/infx/klaud/__main__.py‎

Lines changed: 24 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@
3535
)
3636

3737
PRODUCERS = "producers.json"
38-
RETRY_NEXT_WAVE = frozenset({"capacity-deferred"})
38+
RETRY_NEXT_WAVE = frozenset({"capacity-deferred", "agent-not-started"})
3939

4040

4141
def observation_key(row: dict) -> tuple:
@@ -203,10 +203,10 @@ def recorded_outcome(repository: str, artifact: dict) -> str | None:
203203
with tempfile.TemporaryDirectory() as scratch:
204204
try:
205205
download_json(repository, artifact, Path(scratch))
206-
outcome = json.loads((Path(scratch) / "candidate-diagnostics.json").read_text())[
207-
"candidate-outcome"
208-
]
209-
return CandidateOutcome.model_validate(outcome).outcome
206+
diagnostics = json.loads((Path(scratch) / "candidate-diagnostics.json").read_text())
207+
if diagnostics.get("action-outcome") == "skipped":
208+
return "agent-not-started"
209+
return CandidateOutcome.model_validate(diagnostics["candidate-outcome"]).outcome
210210
except (
211211
OSError,
212212
ValueError,
@@ -662,6 +662,7 @@ def select(directory: Path, max_candidates: int, execution_file: Path | None = N
662662
deferred = "capacity-unavailable"
663663
capacity_deferred = []
664664
baseline_deferred = []
665+
baseline_reasons: dict[str, str] = {}
665666
baseline_mismatches = []
666667
preflights = {}
667668
families = {decision.family for decision in review.decisions if decision.decision != "proceed"}
@@ -700,16 +701,24 @@ def select(directory: Path, max_candidates: int, execution_file: Path | None = N
700701
),
701702
baseline,
702703
)
703-
except VerificationError:
704+
except (VerificationError, ReadError) as error:
704705
baseline_deferred.append(candidate["id"])
706+
baseline_reasons[candidate["id"]] = str(error)
705707
families.add(decision.family)
706708
continue
707-
except (OSError, ValueError, KeyError, TypeError, subprocess.SubprocessError, ReadError):
709+
except subprocess.SubprocessError as error:
708710
baseline_deferred.append(candidate["id"])
711+
baseline_reasons[candidate["id"]] = type(error).__name__
712+
families.add(decision.family)
713+
continue
714+
except (OSError, ValueError, KeyError, TypeError) as error:
715+
baseline_deferred.append(candidate["id"])
716+
baseline_reasons[candidate["id"]] = f"{type(error).__name__}: {str(error)[:300]}"
709717
families.add(decision.family)
710718
continue
711719
if missing:
712720
baseline_deferred.append(candidate["id"])
721+
baseline_reasons[candidate["id"]] = "baseline-point-mismatch"
713722
baseline_mismatches.append(
714723
{
715724
"id": candidate["id"],
@@ -759,6 +768,7 @@ def select(directory: Path, max_candidates: int, execution_file: Path | None = N
759768
"deferred-reason": deferred,
760769
"capacity-deferred-candidates": capacity_deferred,
761770
"baseline-deferred-candidates": baseline_deferred,
771+
"baseline-deferral-reasons": baseline_reasons,
762772
"baseline-mismatch-candidates": baseline_mismatches,
763773
**review.model_dump(by_alias=True),
764774
},
@@ -779,7 +789,13 @@ def select(directory: Path, max_candidates: int, execution_file: Path | None = N
779789
f" {len(capacity_deferred)} reviewed candidates deferred by the latest capacity check."
780790
)
781791
if baseline_deferred:
782-
summary += f" {len(baseline_deferred)} candidates lacked a verifiable full baseline."
792+
summary += f" {len(baseline_deferred)} candidates lacked a verifiable full baseline: " + (
793+
"; ".join(
794+
f"{count}× {reason[:120]}"
795+
for reason, count in Counter(baseline_reasons.values()).most_common()
796+
)
797+
+ "."
798+
)
783799
if baseline_mismatches:
784800
summary += (
785801
f" {len(baseline_mismatches)} of them have frozen baseline points"

‎inferencex-e2e/infx/klaud/lifecycle.py‎

Lines changed: 149 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,18 +4,23 @@
44

55
import json
66
import os
7+
import re
78
import subprocess
89
import tempfile
910
from pathlib import Path
1011
from urllib.parse import urlencode
1112

1213
from . import claims, github
1314
from .github import VerificationError
14-
from .models import CandidateOutcome, OwnedCandidate, Ownership, utc
15+
from .models import CandidateOutcome, OwnedCandidate, Ownership, identity, utc
1516
from .reporting import translated
1617
from .validation import verify_sweep
1718

1819
BOT = "Klaud-Cold"
20+
COMMITTER = "github-actions[bot]"
21+
ACTIVE_STATUSES = ("requested", "waiting", "pending", "queued", "in_progress")
22+
CANDIDATE_ID = r"[0-9a-f]{16}-[0-9a-f]{16}"
23+
RECONCILE_JOB = "Reconcile interrupted candidates"
1924
SWEEP_LABELS = {
2025
"full-sweep-enabled",
2126
"non-canary-full-sweep-enabled",
@@ -87,7 +92,8 @@ def pulls(self) -> list[dict]:
8792
return pulls
8893

8994
def runs(self) -> list[dict]:
90-
query = {"per_page": 100, "created": ">=" + self.parent["created_at"]}
95+
end = self.parent["updated_at"] if terminal(self.parent) else "*"
96+
query = {"per_page": 100, "created": f"{self.parent['created_at']}..{end}"}
9197
targeted = github.items(
9298
self.repository,
9399
"actions/workflows/e2e-tests.yml/runs?"
@@ -105,7 +111,14 @@ def runs(self) -> list[dict]:
105111
sweeps = github.items(
106112
self.repository,
107113
"actions/workflows/run-sweep.yml/runs?"
108-
+ urlencode({**query, "event": "pull_request", "branch": self.branch}),
114+
+ urlencode(
115+
{
116+
"per_page": 100,
117+
"created": ">=" + self.parent["created_at"],
118+
"event": "pull_request",
119+
"branch": self.branch,
120+
}
121+
),
109122
"workflow_runs",
110123
)
111124
pulls = self.pulls()
@@ -531,6 +544,7 @@ def recover() -> None:
531544
raise VerificationError("Untrusted ownership parent")
532545
results = {}
533546
blocked = set()
547+
live = set()
534548
for owner, _ in sorted(owners, key=lambda pair: pair[0].run_id, reverse=True):
535549
parent = parents[owner.run_id]
536550
for candidate in owner.candidates:
@@ -540,11 +554,13 @@ def recover() -> None:
540554
results[key] = False
541555
if not terminal(parent):
542556
blocked.add(candidate.family)
557+
live.add(candidate.id)
543558
continue
544559
session = Session(repository, parent, candidate, recovering=True)
545560
try:
546561
if not claims.recovery_lease(repository, parent["id"], candidate):
547562
blocked.add(candidate.family)
563+
live.add(candidate.id)
548564
continue
549565
changed = reconcile(session)
550566
claims.release_family(repository, candidate, parent["id"])
@@ -582,6 +598,136 @@ def recover() -> None:
582598
current = github.read(repository, f"actions/runs/{owner.run_id}")
583599
if terminal(current) and current["run_attempt"] == parents[owner.run_id]["run_attempt"]:
584600
github.write(repository, f"actions/artifacts/{artifact['id']}", "DELETE")
601+
try:
602+
live |= sweep_abandoned(repository, live)
603+
except (OSError, ValueError, KeyError, TypeError, subprocess.SubprocessError) as error:
604+
print(f"::warning::Klaud abandoned-work sweep skipped: {type(error).__name__}")
605+
else:
606+
prefixes = {candidate.split("-", 1)[0] for candidate in live}
607+
blocked = {family for family in blocked if identity(family)[:16] in prefixes}
585608
(Path(os.environ["RUNNER_TEMP"]) / "klaud-recovery.json").write_text(
586609
json.dumps(sorted(blocked))
587610
)
611+
612+
613+
def owned_activity(repository: str) -> tuple[set[str], dict[str, list[int]], set[int]]:
614+
live: set[str] = set()
615+
targeted: dict[str, list[int]] = {}
616+
planners: set[int] = set()
617+
for status in ACTIVE_STATUSES:
618+
query = urlencode({"status": status, "per_page": 100})
619+
for run in github.items(
620+
repository, f"actions/workflows/e2e-tests.yml/runs?{query}", "workflow_runs"
621+
):
622+
match = re.fullmatch(rf"e2e Test - klaud-\d+-({CANDIDATE_ID})", run["display_title"])
623+
if match and run["actor"]["login"] == BOT:
624+
targeted.setdefault(match[1], []).append(run["id"])
625+
for run in github.items(
626+
repository, f"actions/workflows/run-sweep.yml/runs?{query}", "workflow_runs"
627+
):
628+
if match := re.fullmatch(rf"klaud/auto-({CANDIDATE_ID})", run["head_branch"] or ""):
629+
live.add(match[1])
630+
for run in github.items(
631+
repository, f"actions/workflows/klaud-plan.yml/runs?{query}", "workflow_runs"
632+
):
633+
for job in github.items(
634+
repository, f"actions/runs/{run['id']}/jobs?per_page=100", "jobs"
635+
):
636+
if match := re.search(rf"Run candidate ({CANDIDATE_ID})", job["name"]):
637+
live.add(match[1])
638+
elif job["name"] == RECONCILE_JOB and job["status"] != "completed":
639+
planners.add(run["id"])
640+
return live, targeted, planners
641+
642+
643+
def bot_committed(commits: list[dict]) -> bool:
644+
return bool(commits) and all(
645+
(commit.get("committer") or {}).get("login") == COMMITTER for commit in commits
646+
)
647+
648+
649+
def sweep_abandoned(repository: str, live: set[str]) -> set[str]:
650+
sessions, targeted, planners = owned_activity(repository)
651+
live = live | sessions
652+
pulls = github.items(repository, "pulls?state=open&per_page=100")
653+
abandoned = []
654+
for pull in pulls:
655+
match = re.fullmatch(rf"klaud/auto-({CANDIDATE_ID})", pull["head"]["ref"])
656+
if not match or match[1] in live:
657+
continue
658+
if (
659+
pull["draft"]
660+
and pull["user"]["login"] == BOT
661+
and (pull["head"]["repo"] or {}).get("full_name") == repository
662+
and not any(label["name"] == "klaud-handoff" for label in pull["labels"])
663+
and bot_committed(
664+
github.items(repository, f"pulls/{pull['number']}/commits?per_page=100")
665+
)
666+
):
667+
abandoned.append(pull)
668+
else:
669+
live.add(match[1])
670+
for candidate, runs in targeted.items():
671+
if candidate not in live:
672+
for run in runs:
673+
github.write(repository, f"actions/runs/{run}/cancel", "POST")
674+
for owner in claims.family_owners(repository):
675+
candidate = owner.candidates[0]
676+
if candidate.id not in live and terminal(
677+
github.read(repository, f"actions/runs/{owner.run_id}")
678+
):
679+
claims.release_family(repository, candidate, owner.run_id)
680+
if planners <= {int(os.environ.get("GITHUB_RUN_ID", "0"))}:
681+
finished: dict[int, bool] = {}
682+
for lease in github.items(repository, "git/matching-refs/heads/klaud/recovery-"):
683+
match = re.fullmatch(rf"refs/heads/klaud/recovery-(\d+)-({CANDIDATE_ID})", lease["ref"])
684+
if not match or match[2] in live:
685+
continue
686+
parent = int(match[1])
687+
if parent not in finished:
688+
finished[parent] = terminal(github.read(repository, f"actions/runs/{parent}"))
689+
if finished[parent]:
690+
github.write(repository, "git/" + lease["ref"], "DELETE")
691+
for pull in abandoned:
692+
github.write(
693+
repository,
694+
f"issues/{pull['number']}/comments",
695+
"POST",
696+
{
697+
"body": translated(
698+
"**abandoned** · The owning Klaud session ended without cleanup. Closed, "
699+
"with any leftover smoke runs cancelled, so a new candidate can take this "
700+
"family.",
701+
"**abandoned** · 所属 Klaud 会话已结束但未完成清理。已关闭并取消残留的冒烟运行,"
702+
"新的候选可以接手该配置族。",
703+
)
704+
},
705+
)
706+
github.write(repository, f"pulls/{pull['number']}", "PATCH", {"state": "closed"})
707+
github.write(repository, f"git/refs/heads/{pull['head']['ref']}", "DELETE")
708+
open_heads = {pull["head"]["ref"] for pull in pulls}
709+
for ref in github.items(repository, "git/matching-refs/heads/klaud/auto-"):
710+
branch = ref["ref"].removeprefix("refs/heads/")
711+
match = re.fullmatch(rf"klaud/auto-({CANDIDATE_ID})", branch)
712+
if not match or match[1] in live or branch in open_heads:
713+
continue
714+
history = github.items(
715+
repository,
716+
"pulls?"
717+
+ urlencode(
718+
{
719+
"state": "all",
720+
"head": repository.split("/", maxsplit=1)[0] + ":" + branch,
721+
"per_page": 100,
722+
}
723+
),
724+
)
725+
if history:
726+
latest = max(history, key=lambda pull: pull["created_at"])
727+
closer = github.read(repository, f"issues/{latest['number']}").get("closed_by") or {}
728+
if latest["user"]["login"] != BOT or closer.get("login") != BOT:
729+
continue
730+
elif not bot_committed([github.read(repository, f"commits/{ref['object']['sha']}")]):
731+
continue
732+
github.write(repository, f"git/refs/heads/{branch}", "DELETE")
733+
return live

‎inferencex-e2e/infx/klaud/reporting.py‎

Lines changed: 16 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -869,8 +869,9 @@ def public_point(entry: dict) -> dict:
869869
"conc": int(entry["conc"]),
870870
"image": normalized_image(entry["image"]),
871871
}
872+
aggregate = multi and not point["disagg"] and not entry["decode"].get("num-worker")
872873
for role in ("prefill", "decode"):
873-
topology = entry[role] if multi else entry
874+
topology = entry["prefill" if aggregate else role] if multi else entry
874875
point.update(
875876
{
876877
f"{role}_tp": topology["tp"],
@@ -886,14 +887,20 @@ def _matches_public_point(row: dict, point: dict) -> bool:
886887
"""Compare a public row with a generated identity using canonical image spelling."""
887888
from .models import normalized_image
888889

889-
for key, value in point.items():
890-
observed = row.get(key)
891-
if key == "image":
892-
if not isinstance(observed, str) or normalized_image(observed) != value:
893-
return False
894-
elif observed != value:
895-
return False
896-
return True
890+
encodings = [point]
891+
if point["is_multinode"] and not point["disagg"] and point["prefill_num_workers"] == 1:
892+
encodings.append(
893+
{**point, "is_multinode": False, "prefill_num_workers": 0, "decode_num_workers": 0}
894+
)
895+
return any(
896+
all(
897+
isinstance(row.get(key), str) and normalized_image(row[key]) == value
898+
if key == "image"
899+
else row.get(key) == value
900+
for key, value in encoding.items()
901+
)
902+
for encoding in encodings
903+
)
897904

898905

899906
def matrix_points(matrix: dict) -> list[dict]:

0 commit comments

Comments
 (0)