feat(eval): cross-harness eval/benchmark framework (cli-anything-eval)#366

Open

omerarslan0 wants to merge 25 commits into

HKUDS:mainfrom

omerarslan0:feat/eval-benchmark-framework

Commits on Jun 14, 2026

Feature/cli matrix multi approach (#355)

yuh-yang
and
claude
authored

Commits on Jun 22, 2026

feat(eval): scaffold cli-anything-eval package with core contracts
omerarslan0
committed
feat(eval): add safe_write_json io helper
omerarslan0
committed
feat(eval): add baseline load + regression comparison
omerarslan0
committed
test(eval): strengthen baseline regression assertions
omerarslan0
committed
feat(eval): add report markdown + baseline projection + leaderboard
omerarslan0
committed
fix(eval): rank leaderboard rows by success rate in the renderer
omerarslan0
committed
feat(eval): add task discovery, execution and summary
omerarslan0
committed
fix(eval): skip tasks before creating dirs; guard test sys.path insert
omerarslan0
committed
feat(eval): add run_eval orchestration with reports and baseline
omerarslan0
committed
style(eval): consolidate imports in runner and runner tests
omerarslan0
committed
feat(eval): add cross-harness suite discovery and leaderboard
omerarslan0
committed
fix(eval): treat empty harness list as run-none in run_suite
omerarslan0
committed
feat(eval): add console script and per-harness command factory
omerarslan0
committed
docs(eval): add cli-anything-eval package README
omerarslan0
committed
refactor(audacity): migrate eval runner onto shared cli-anything-eval
omerarslan0
committed
feat(gimp): add eval tasks using shared cli-anything-eval
omerarslan0
committed
fix(gimp): narrow eval precheck to ImportError
omerarslan0
committed
feat(blender): add eval tasks using shared cli-anything-eval
omerarslan0
committed
refactor(blender): place eval registration before __main__ guard
omerarslan0
committed
feat(libreoffice): add eval tasks using shared cli-anything-eval
omerarslan0
committed
docs(eval): add plugin command, HARNESS.md methodology, and roadmap update
omerarslan0
committed
fix(eval): align HARNESS.md example, add io encoding, build blender argv directly
omerarslan0
committed
fix(eval): discover harnesses via distributions for editable installs
omerarslan0
committed
test(eval): fix sys.path leak and monkeypatch target in metadata discovery test
omerarslan0
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(eval): cross-harness eval/benchmark framework (cli-anything-eval)#366

feat(eval): cross-harness eval/benchmark framework (cli-anything-eval)#366
omerarslan0 wants to merge 25 commits into
HKUDS:mainfrom
omerarslan0:feat/eval-benchmark-framework

Commits on Jun 14, 2026

Feature/cli matrix multi approach (#355)

Commits on Jun 22, 2026

feat(eval): scaffold cli-anything-eval package with core contracts

feat(eval): add safe_write_json io helper

feat(eval): add baseline load + regression comparison

test(eval): strengthen baseline regression assertions

feat(eval): add report markdown + baseline projection + leaderboard

fix(eval): rank leaderboard rows by success rate in the renderer

feat(eval): add task discovery, execution and summary

fix(eval): skip tasks before creating dirs; guard test sys.path insert

feat(eval): add run_eval orchestration with reports and baseline

style(eval): consolidate imports in runner and runner tests

feat(eval): add cross-harness suite discovery and leaderboard

fix(eval): treat empty harness list as run-none in run_suite

feat(eval): add console script and per-harness command factory

docs(eval): add cli-anything-eval package README

refactor(audacity): migrate eval runner onto shared cli-anything-eval

feat(gimp): add eval tasks using shared cli-anything-eval

fix(gimp): narrow eval precheck to ImportError

feat(blender): add eval tasks using shared cli-anything-eval

refactor(blender): place eval registration before main guard

feat(libreoffice): add eval tasks using shared cli-anything-eval

docs(eval): add plugin command, HARNESS.md methodology, and roadmap update

fix(eval): align HARNESS.md example, add io encoding, build blender argv directly

fix(eval): discover harnesses via distributions for editable installs

test(eval): fix sys.path leak and monkeypatch target in metadata discovery test