Summary
Suite-path delivery of successPredicates on EvalTestCase is intentionally deferred from PR #2352 (#2352 (comment)).
Quick-run and single-case paths already gate on successPredicates today via RunTestCaseRequest.testCaseOverrides.successPredicates. Suite runs cannot yet, because:
- The Convex
testCase schema does not include the field.
startSuiteRunWithRecorder rebuilds config.tests from persisted Convex testCase rows, so adding the field to RunEvalsRequestSchema alone would be accepted-but-silently-ignored.
Work required
- Backend (mcpjam-backend, branch
eval-predicate-metadata): Add successPredicates to the Convex testCase schema and update storage/retrieval accordingly.
- Inspector: In
startSuiteRunWithRecorder / createTestCase (~L703-720 of mcpjam-inspector/server/services/evals-runner.ts), thread tc.successPredicates into the rebuilt EvalTestCase objects, mirroring how matchOptions is already handled.
- Schema: Add
successPredicates to RunEvalsRequestSchema and ensure it survives the round-trip through Convex.
References
/cc @chelojimenez
Summary
Suite-path delivery of
successPredicatesonEvalTestCaseis intentionally deferred from PR #2352 (#2352 (comment)).Quick-run and single-case paths already gate on
successPredicatestoday viaRunTestCaseRequest.testCaseOverrides.successPredicates. Suite runs cannot yet, because:testCaseschema does not include the field.startSuiteRunWithRecorderrebuildsconfig.testsfrom persisted ConvextestCaserows, so adding the field toRunEvalsRequestSchemaalone would be accepted-but-silently-ignored.Work required
eval-predicate-metadata): AddsuccessPredicatesto the ConvextestCaseschema and update storage/retrieval accordingly.startSuiteRunWithRecorder/createTestCase(~L703-720 ofmcpjam-inspector/server/services/evals-runner.ts), threadtc.successPredicatesinto the rebuiltEvalTestCaseobjects, mirroring howmatchOptionsis already handled.successPredicatestoRunEvalsRequestSchemaand ensure it survives the round-trip through Convex.References
eval-predicate-metadata/cc @chelojimenez