Skip to content

evals: wire successPredicates through suite-run hydration (Convex testCase schema + startSuiteRunWithRecorder) #2353

Description

@coderabbitai

Summary

Suite-path delivery of successPredicates on EvalTestCase is intentionally deferred from PR #2352 (#2352 (comment)).

Quick-run and single-case paths already gate on successPredicates today via RunTestCaseRequest.testCaseOverrides.successPredicates. Suite runs cannot yet, because:

  1. The Convex testCase schema does not include the field.
  2. startSuiteRunWithRecorder rebuilds config.tests from persisted Convex testCase rows, so adding the field to RunEvalsRequestSchema alone would be accepted-but-silently-ignored.

Work required

  • Backend (mcpjam-backend, branch eval-predicate-metadata): Add successPredicates to the Convex testCase schema and update storage/retrieval accordingly.
  • Inspector: In startSuiteRunWithRecorder / createTestCase (~L703-720 of mcpjam-inspector/server/services/evals-runner.ts), thread tc.successPredicates into the rebuilt EvalTestCase objects, mirroring how matchOptions is already handled.
  • Schema: Add successPredicates to RunEvalsRequestSchema and ensure it survives the round-trip through Convex.

References

/cc @chelojimenez

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions