feat(eval): cross-harness eval/benchmark framework (cli-anything-eval)

[omerarslan0]

Description

Generalizes the previously Audacity-only eval prototype into a shared, installable cli-anything-eval package — a cross-harness eval/benchmark framework. This implements the roadmap item "Benchmark suite for agent task completion rates."

The framework provides a convention-based task contract (TASK + run + optional verify/precheck/requires/prompt), a runner with pass/fail/error/skipped statuses, baseline regression detection, JSON + Markdown reports, a cross-harness leaderboard, and a CLI (cli-anything-eval run|suite). A verify(ctx) grader seam keeps tasks forward-compatible with future agent-driven grading while running deterministically today.

Adoption in this PR: Audacity is migrated onto the shared package via a backward-compatible shim; seed eval tasks are added for GIMP, Blender, and LibreOffice; a /cli-anything:eval plugin command and a HARNESS.md methodology section are added.

Type of Change

• New Feature — adds new functionality to an existing harness or the plugin

(Also migrates the Audacity harness and seeds three others onto the new framework.)

Highlights

• New top-level package cli-anything-eval/ exposing the cli_anything.eval namespace (PEP 420; depends only on click; imports nothing from any harness).
• Status model: success_rate = passed / (pass + fail + error); skipped (unmet backend requires/precheck) is reported separately and excluded from the denominator — a deliberate, documented divergence from the test-suite "fail when backend missing" rule, so a benchmark's completion rate is not polluted by missing backends.
• Backward compatible: cli_anything.audacity.eval.runner keeps its public API; the existing cli-anything-audacity eval command and test_eval.py are unchanged in behavior.
• Harness eval subcommands register behind an except ImportError guard, so a harness still works if the eval package is absent.
• Discovery works under both normal site-packages and PEP 660 editable installs.

For Existing CLI Modifications

• Audacity eval tests pass (test_eval.py 3/3); the 4 Audacity eval tasks run via the shared runner.
• No test regressions — Blender unit suite still 165/165 after a Windows-encoding fix in core/render.py.
• registry.json — not applicable (no version/description/requirement change to registry entries).

General Checklist

• Code follows existing patterns and conventions
• --json flag is supported on the new commands
• Commit messages follow the conventional format (feat:, fix:, docs:, test:)
• I have tested my changes locally

Test Results

cli-anything-eval package:   29 passed
audacity test_eval.py:        3 passed   (backward-compatibility)
blender unit (test_core.py): 165 passed   (no regression from render.py encoding fix)

cross-harness leaderboard (cli-anything-eval suite):
| Rank | Harness     | Passed/Attempted | Skipped | Success Rate |
| ---- | ----------- | ---------------- | ------- | ------------ |
| 1    | audacity    | 4/4              | 0       | 100%         |
| 2    | blender     | 1/1              | 1       | 100%         |
| 3    | gimp        | 1/1              | 0       | 100%         |
| 4    | libreoffice | 1/1              | 1       | 100%         |

(skipped rows are backend-dependent tasks — blender render_execute, libreoffice export_pdf — correctly skipped when the binary is absent.)

Follow-ups (out of scope for this PR)

• Optional extras_require["eval"] instead of a hard dependency (the registration is already ImportError-guarded).
• A CI workflow to run the suite and publish the leaderboard.
• A real (key-gated) agent driver behind the prompt/verify seam.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 792aa264f6

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Do not silently ignore missing baselines

When a caller passes a baseline path that does not exist (for example a typo with --baseline ... --fail-on-regression), this guard leaves comparison as None, so the run succeeds and the regression gate is bypassed instead of surfacing the bad baseline path. Since load_baseline() already raises FileNotFoundError, the existence check should not suppress that error unless this is specifically the update-baseline creation path.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Add an active camera before rendering

When Blender is installed, this task builds a scene with only a cube; create_scene() starts with no cameras and the generated bpy script clears Blender's default scene before adding project cameras, so the render has no active camera and produces no PNG (or reports a Blender render error). The eval task should add an active camera (and ideally lighting) before calling render_scene() so environments with Blender don't fail this benchmark.

Useful? React with 👍 / 👎.

[omerarslan0]

@yuh-yang This PR is on the larger side (48 files) and touches shared/core infrastructure — it adds a new shared cli-anything-eval package and migrates the Audacity harness onto it (plus seeds GIMP/Blender/LibreOffice). Because of that surface area, could you please give it a detailed review rather than a quick pass when you have time? Happy to clarify any design decision or split it into smaller PRs if that helps. Thanks!