Skip to content

Add eval-as-a-gate scoring - #225

Merged
Siddhant-K-code merged 1 commit into
mainfrom
feat/eval-gate-210
Aug 16, 2026
Merged

Siddhant-K-code merged 1 commit into
mainfrom
feat/eval-gate-210

Conversation

@Siddhant-K-code

Copy link
Copy Markdown
Owner

Summary

  • add named raw-value eval criteria for cost, errors, completion status, redundant reads, tool calls, context use, duration, and lint findings
  • support versioned baselines, regression tolerance, JSON/table output, custom Python scorers, and GitHub Actions step summaries
  • preserve the legacy eval subcommands while making agent-strace eval [SESSION_ID] the direct CI gate
  • wire the existing composite action to the named-gate workflow safely
  • document .agent-evals.yaml and bump agent-strace from 0.83.0 to 0.84.0

Closes #210

Verification

  • PYTHONPATH=src python -m unittest tests.test_eval_gate tests.test_eval tests.test_eval_extensions -q — 96 tests passed
  • Ona test task — 1,680 Python tests and 3 VS Code extension tests passed
  • python -m compileall -q src/agent_trace && git diff --check
  • wheel build — agent_strace-0.84.0-py3-none-any.whl

Co-authored-by: Codex <noreply@openai.com>
@Siddhant-K-code Siddhant-K-code added the enhancement New feature or request label Aug 16, 2026
@Siddhant-K-code

Copy link
Copy Markdown
Owner Author

PR review completed: checked backward compatibility, direct/legacy CLI routing, baseline direction and tolerance semantics, GitHub Actions argument handling, storage compatibility, and coverage. No blocking findings. Merge remains gated on all five GitHub CI jobs.

@Siddhant-K-code
Siddhant-K-code merged commit 89c077f into main Aug 16, 2026
5 checks passed
@Siddhant-K-code
Siddhant-K-code deleted the feat/eval-gate-210 branch August 16, 2026 16:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: eval-as-a-gate — score sessions against named criteria and fail CI on regression

1 participant