Publish an evaluator that carries its own rubric - #6
Merged
m7md7sien merged 1 commit intoAug 20, 2026
Merged
Conversation
Both publish loops selected on 'source' alone, and validation guarantees a rubric written under 'definition' comes with no source. So a ref to a rubric decoded, validated, reported nothing and published nothing, and the eval was then created against an evaluator the service had never been told about. The whole feature was inert. Every test for it stopped at decoding, which is why none of them noticed. A test now asserts a carried rubric reaches the publish set. Also: the rescue is gated on the document actually using a ref, structurally, on both routes. The CLI's no-ref fast path skipped nesting while deploy still applied it, so a hand-written entry carrying rubric keys deployed and was then refused by every command that read it -- the same asymmetry a third time. The gate is structural rather than a byte scan, so a ref mentioned in a comment no longer changes an unrelated entry's meaning. The one-resolver guard now scans the whole extension; it walked only its own package, which is not where a second caller would appear.
m7md7sien
merged commit Aug 20, 2026
2c0509e
into
feat/azure-ai-evaluations-consolidated
1 check passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Final review round found the rubric feature was inert: both publish loops selected on
sourcealone, and validation guarantees a rubric underdefinitioncomes with no source, so a\to a rubric published nothing and the eval bound an evaluator the service had never seen. Every test stopped at decoding.Also closes the
\asymmetry for a third time: the CLI's no-\fast path skipped the rescue while deploy still applied it, so a hand-written entry carrying rubric keys deployed and was refused by every command that read it. Both routes now gate on the same structural check.Both fixes are mutation-tested -- each regression test was confirmed to fail without its fix.
gofmt, vet (untagged, hero, live) and the full suite are green.