feat(luce-bench): in-tree bench harness + multi-turn agent_recorded + LLM judge

[easel]

Summary

Adds the standalone luce-bench/ Python eval harness package (vendored into the repo), plus a multi-turn agent_recorded replay area graded by a Claude Sonnet LLM judge. Also wires up a card-bundle drift check script and CI workflow updates.

Files

• luce-bench/ — new top-level package (pyproject, src layout, tests, fixtures, docs)
  • src/lucebench/areas/ — eval areas: smoke, agent, agent_recorded, ds4_eval, forge, gsm8k, hellaswag, humaneval, longctx, truthfulqa_mc1
  • src/lucebench/grading/llm_judge.py — Sonnet judge with on-disk cache
  • src/lucebench/fixtures/ — vendored eval fixtures (incl. forge_eval scenarios)
  • tests/ — ~20 test modules covering graders, runners, snapshot, cards
• scripts/check_card_bundle_drift.sh — new drift CI helper
• .github/workflows/ci.yml — wires luce-bench tests + card bundle drift check

Dependencies

None. This PR is self-contained: pure-Python harness with no source-level references to the server, lucebox CLI, or docker-stack PRs. Independent of #334, #335, #336, #338, #339, #340, #341.

Test plan

• cd luce-bench && pytest passes locally
• CI green on the new card-bundle-drift job
• Dry-run multi-turn agent_recorded against a live server with judge mocked
• Confirm default --areas all does not enable the LLM-judge path (cost containment)

Judge cost estimate: ~$0.30-$1.50 per full 48-case pass on Anthropic Sonnet.

Co-Authored-By: Claude Opus 4.7 noreply@anthropic.com

[cubic-dev-ai]

16 issues found across 116 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="luce-bench/src/lucebench/fixtures/agent_prompts/codex_apply_patch.md">

<violation number="1" location="luce-bench/src/lucebench/fixtures/agent_prompts/codex_apply_patch.md:325">
P2: Grammar for `Hunk` does not allow multiple `@@` context-scoping lines, contradicting the documented feature and examples. The production `Hunk := "@@" [ header ] NEWLINE { HunkLine }` expects only HunkLines (starting with space, `-`, or `+`) after the `@@` header, but the text explicitly says "use multiple `@@` statements to jump to the right context" and shows consecutive `@@` lines (e.g., `@@ class BaseClass` then `@@ def method():`). This inconsistency will confuse agent implementors or cause strict parsers to reject valid patch syntax.</violation>
</file>

<sub>Note: This PR contains a large number of files. cubic only reviews up to 100 files per PR, so some files may not have been reviewed. cubic prioritizes the most important files to review.
On a pro plan you can use ultrareview for larger PRs.
Tip: cubic used a learning from your PR history. Let your coding agent read cubic learnings directly with the cubic MCP.

Re-trigger cubic</sub>

[cubic-dev-ai]

2 issues found across 6 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="luce-bench/src/lucebench/areas/agent_recorded.py">

<violation number="1" location="luce-bench/src/lucebench/areas/agent_recorded.py:583">
P2: Unvalidated restart-timeout env parsing can crash the area with ValueError. Invalid `LUCEBENCH_AGENT_RECORDED_RESTART_TIMEOUT` should degrade to a restart failure/default timeout, not terminate execution.</violation>

<violation number="2" location="luce-bench/src/lucebench/areas/agent_recorded.py:915">
P2: Sequential-mode row stores final response in `model_response_cold`, so cold/final response fields are inconsistent. Use the first turn response for `model_response_cold` and keep final output in `model_response_final`.</violation>
</file>

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: Sequential-mode row stores final response in model_response_cold, so cold/final response fields are inconsistent. Use the first turn response for model_response_cold and keep final output in model_response_final.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At luce-bench/src/lucebench/areas/agent_recorded.py, line 915:

<comment>Sequential-mode row stores final response in `model_response_cold`, so cold/final response fields are inconsistent. Use the first turn response for `model_response_cold` and keep final output in `model_response_final`.</comment>

<file context>
@@ -683,14 +753,211 @@ def _run_one_multi_turn_case(
+        "cache_eligible_turns": len(eligible_turns),
+        "cache_hit_turns": cache_hit_turns,
+        "cache_speedup_x": None,
+        "model_response_cold": final_content,
+        "model_response_final": final_content,
+        "judge": judge_result,
</file context>

Suggested change

	"model_response_cold": final_content,
	"model_response_cold": (cold_row.get("content") or "") + (cold_row.get("reasoning_content") or ""),

[cubic-dev-ai]

P2: Unvalidated restart-timeout env parsing can crash the area with ValueError. Invalid LUCEBENCH_AGENT_RECORDED_RESTART_TIMEOUT should degrade to a restart failure/default timeout, not terminate execution.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At luce-bench/src/lucebench/areas/agent_recorded.py, line 583:

<comment>Unvalidated restart-timeout env parsing can crash the area with ValueError. Invalid `LUCEBENCH_AGENT_RECORDED_RESTART_TIMEOUT` should degrade to a restart failure/default timeout, not terminate execution.</comment>

<file context>
@@ -504,15 +580,16 @@ def _restart_lucebox_service(*, log_prefix: str = "[agent_recorded]") -> tuple[b
     """
     if shutil.which("systemctl") is None:
         return (False, "systemctl not on PATH")
+    timeout_s = float(os.environ.get("LUCEBENCH_AGENT_RECORDED_RESTART_TIMEOUT", "180"))
     try:
         result = subprocess.run(  # noqa: S603 - operator-approved invocation
</file context>

Suggested change

	timeout_s = float(os.environ.get("LUCEBENCH_AGENT_RECORDED_RESTART_TIMEOUT", "180"))
	try:
	timeout_s = float(os.environ.get("LUCEBENCH_AGENT_RECORDED_RESTART_TIMEOUT", "180"))
	except ValueError:
	return (False, "invalid LUCEBENCH_AGENT_RECORDED_RESTART_TIMEOUT")

[cubic-dev-ai]

1 issue found across 2 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="luce-bench/src/lucebench/areas/agent_recorded.py">

<violation number="1" location="luce-bench/src/lucebench/areas/agent_recorded.py:583">
P2: Unvalidated restart-timeout env parsing can crash the area with ValueError. Invalid `LUCEBENCH_AGENT_RECORDED_RESTART_TIMEOUT` should degrade to a restart failure/default timeout, not terminate execution.</violation>

<violation number="2" location="luce-bench/src/lucebench/areas/agent_recorded.py:915">
P2: Sequential-mode row stores final response in `model_response_cold`, so cold/final response fields are inconsistent. Use the first turn response for `model_response_cold` and keep final output in `model_response_final`.</violation>
</file>

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>