Declare the azure.ai.eval schema, and make $ref mean one thing - #2
Merged
m7md7sien merged 23 commits intoAug 20, 2026
Merged
Conversation
Six Foundry extensions ship a schemas/ directory and are registered in schemas/v1.0/azure.yaml.json by an if/then block on host. The evaluations extension shipped neither, so host: azure.ai.eval was unrecognised: host is a free string with an examples list rather than an enum, so it did not error -- nothing validated the service entry, the $ref, or anything under it. This adds azure.ai.eval.json describing the service body -- the dataset and evaluator catalogs and the evals defined over them -- built from the Go types rather than from the spec, and registers it in both v1.0 and alpha the way the five thin siblings do. Two examples are validated with the ajv recipe from the agents schemas README; both pass. It also pins an asymmetry found while writing this. azd core resolves $ref recursively over the whole service config and the eval service target calls it, so azd up accepts an include anywhere in azure.eval.yaml. Every CLI command instead reads the file straight off disk through a strict decoder, so the same document fails with unknown key "$ref". Resolving it there is not just a matter of calling ResolveFileRefs: azd rebases only project and instructions path values, so a declaration spliced in from another directory would keep a source: that resolves against the wrong base. Which keys an extension may have rebased is the azd team question this work surfaces, and the new test is what should fail when that answer arrives.
Evaluator entries are now either an inline definition or a $ref to a file
holding one, which is what agents and projects do and what FileRef means:
replace an inline definition with a reference to an external file. The
declaration is the definition rather than a pointer to one, so a spliced entry
carries no further paths and the rebasing question does not arise.
FileRef is cloned locally rather than referenced across extensions, following
azure.ai.projects. Nothing references the agents copy from another extension,
and a raw-URL $ref would couple this schema to their file layout.
datasets[].source becomes datasets[].file. A .jsonl is a data artifact to
upload, not a definition to splice, so it keeps a path key -- but source meant
three different things in one file (a data path, a definition path, and the
inline object saying where an eval reads rows), and only the last is left.
The oneOf needs an explicit discriminator: EvaluatorDecl carries the
service-owned rubric keys so additionalProperties has to stay true, which means
it would otherwise also match a FileRef. ajv caught that; not: {required:
[$ref]} settles it. Both examples validate.
A dataset entry points at a .jsonl whose rows are uploaded. It is an artifact to publish, not a definition, which is why it is not a $ref -- and calling it source: made that harder to see, because source: also named an evaluator file and, on an eval, the inline object saying where its rows come from. Three meanings for one key, in one file. Only the eval one is left, which is what the surrounding prose already claimed the key meant. The near-miss suggester works on edit distance 1, so the typo fixture moves from sourse/source to fil/file; a transposition like fiel is distance 2 and would have made the test assert a suggestion the code does not offer.
Three independent reviews converged on the same thing: the schema had been
written for the FileRef model before the Go code moved, so it described
evaluator entries that DecodeEvalConfig rejects. A schema that is ahead of its
code is worse than no schema -- an editor accepts a document the CLI then
refuses. Evaluators go back to {name, source, version} and the FileRef
definition comes out until the migration lands.
The numeric bounds were also stricter than the code. minimum: 1 rejected the
zero that every one of these fields treats as unset: runnable.go rejects only
MaxSamples < 0, trace_window.go only MaxTurns < 0 and LookbackHours < 0.
lookback_hours gains the upper bound the code does enforce
(project.MaxLookbackHours, ten years).
max_samples deliberately gets no upper bound. MinSampleSize/MaxSampleSize
gate the generate flag, not the config field, which runnable.go bounds only
below -- stating 15..1000 here would make the schema stricter than the code
in the other direction.
The hero assertion was checking for the old key, which made it vacuous: it
asserted a registered dataset writes no "source:" in a file that no longer
contains that word for any reason. It checks "file:" now. All of tests/ is
behind live/hero build tags, so go test ./... never compiled it; both tags
now vet clean.
Core owns the resolver but does not run it before dispatch -- it hands each extension the entry with $ref still in it. The service target has always called foundry.ResolveFileRefs and this path never did, so an include deployed cleanly and then failed every azd ai eval command with unknown key "$ref". One file meaning two different things depending on which command opened it. LoadEvalConfig now resolves against the configuration's own directory, which is the base its relative paths are already documented to use, and strips the directive before the strict decode. A configuration with no $ref is handed to the decoder untouched rather than round-tripped through a map. The round trip would cost the decoder its line numbers, and a mistyped key reported at the wrong line is worse than the resolution is worth for the case that does not need it. The old test pinned the asymmetry as known-broken; it now covers the invariant that survives -- a directive reaching the strict decoder means resolution was skipped, and naming it is still the right answer. Renamed to say that. Reported by the owner of $ref in azd. The dataset file: half of the same review is already handled: serviceRelativeDir recovers the base from the $ref value on deploy, and on this path the base is the file's own directory.
The cascade for where the configuration lives was --path, then the path init recorded in the azd environment, then ./evals. The $ref on the azure.ai.eval service was written by init and never read back, so azure.yaml and the CLI were two answers to one question. The recorded level hid it on the machine that ran init. An azd environment is not in the repository, though: check the project out somewhere else -- a colleague, a build agent -- and level two is empty, so a configuration the project declares under ./config is reported missing by every command while azd up deploys it from the same $ref. The old comment on this function already described the symptom: "while azure.yaml's $ref pointed at it correctly the whole time." The declaration now sits between the recorded path and the default. It ranks below the recorded one because that is what --path wrote on this machine, and above the default because a project that declares where its configuration lives should be believed over a convention. Best-effort by design: outside an azd project there is no azure.yaml to read, which is ordinary rather than a failure, unlike a recorded path that could not be read. From the $ref owner's review of what the agents extension does correctly -- resolving both root and nested references. Nested was already handled by the shared resolver; the root reference was the half that was only ever written.
Reading the declaration was half a fix. The $ref names a file -- checkOneConfig's own comment says so, "azure.yaml $refs a single file by name" -- and taking only its directory then looked for azure.eval.yaml beside it. A project declaring ./config/nightly.yaml was still reported missing by every CLI command while azd up deployed it from the same $ref. A configuration location is now the directory or the file itself. EvalConfigPath returns a location that names a file unchanged and applies the naming convention to one that names a directory; EvalDirOf gives the directory relative paths resolve against. A location that does not exist yet reads as a directory, which is what init is handed before it writes anything. The both-names guard only applies to a directory: a location that already names the file is the project saying which one it means, so there is nothing to disambiguate. Artifact directories now derive from EvalDirOf rather than the location, so init scaffolds datasets/ and evaluators/ beside a declared file instead of inside it.
ensureServiceWired matched on name and host alone, so a service pointing at the
old file counted as correctly wired. init --path ./quality wrote the
configuration to one place and left azure.yaml deploying another.
That was inert while nothing read the entry. It stopped being inert when the
directory cascade started reading it: the stale value now answers on any machine
that did not run this init, because the recorded path lives in the azd
environment and an azd environment does not travel with the repository. So this
is a hazard the cascade change created out of a latent bug, and it belongs with
it.
AddService reloads from disk and replaces the entry, preserving the event
dispatcher and the ${VAR} templates already in the file, so writing it again is
how it gets repointed. The caller is told: a repoint changes what azd up deploys
and should not happen quietly.
Resolving on the CLI read path introduced a quiet regression: init and generate read, modify and save the same file, so what they saved was the resolved document. The author's include was replaced by its content, the file it named was orphaned, and the paths inside that file were left resolving against the wrong directory -- a source: ./quality.json written beside evaluators/quality.yaml came back pointing at the project root. Nothing reported any of it, because from the writer's point of view it had saved what it read. Reading for use and reading for edit are now separate. OpenEvalConfig resolves; OpenEvalConfigForEdit hands the file to the decoder as written, and the two commands that save use it. That needs $ref to survive the strict decoder, so EvaluatorDecl models it. Name becomes omitempty for the same reason: an entry that is only a $ref has no name until the file it names supplies one. The old test asserted the decoder refuses a directive, on the reasoning that one reaching it meant resolution had been skipped. That only held while every reader resolved. It now covers what replaced it -- the directive decodes and round trips -- and the schema's oneOf: [EvaluatorDecl, FileRef] goes back, since the code accepts what it describes. EvaluatorDecl is closed, so a $ref entry matches the FileRef branch alone and the oneOf needs no discriminator this time. Both examples validate with the branch exercised.
One conflict, and it was the same bug fixed twice. Both sides had found that matching a service entry on name and host alone reported the wiring present after init --path moved the configuration, leaving azure.yaml pointing at the file that was left behind. Took the base's design over mine. It refuses with ServiceRefPointsElsewhere and leaves the decision to the reader, rather than repointing azure.yaml quietly as mine did -- the project file is the author's, and a command that rewrites it without being asked is the more surprising of the two. Its sameRefTarget also compares the values as paths, so ./evals/azure.eval.yaml and evals/azure.eval.yaml are one answer, which my exact-string compare got wrong. That left wiringUpdated, RepointedServiceLine and a duplicate serviceRef helper unreferenced. Removed: the unused linter fires on dead unexported code, and the messages package has a reachability test that fails on a constructor nothing calls. The base shipped its fix without tests, so mine were retargeted at it rather than deleted -- the path comparison, the empty-ref case, and that the refusal names both configurations.
An evaluator's rubric could only be named as a file. A `` splices the referenced file's keys into the entry, so a `` to a rubric produced `unknown key dimensions` -- the shape the spec documents did not load. The rubric now rides on the declaration under `definition`, which is one of the two shapes the publish path already accepted. It is a named key rather than a catch-all on purpose: a catch-all would absorb every misspelling in the entry and publish it to the service as rubric content. Naming a rubric and writing one out are rejected together rather than one silently winning, and a definition is fingerprinted like a file so inline rubrics get the change detection they lacked.
…ated' into feat/eval-schema-registration
It documented `options.eval_model`, which is not a key this decoder has ever had, so following the README produced `unknown key options`. It also named the config `azure.yaml` rather than `azure.eval.yaml`. Correcting the keys alone would only reset the clock on the same drift, so the example is now decoded by a test. Confirmed it fails on the old content rather than passing vacuously.
The spec's example points a `` at a rubric downloaded by `generate`, which is a bare `{type, dimensions}`. Resolution splices those keys into the entry, so they landed beside `name` and were rejected: the documented example did not load.
They are moved under `definition` instead. Wrapping the file would have been the smaller change and the wrong one -- the tool writes that file, so the config has to read what the tool writes.
Scoped to entries the author wrote as a ``, noted before resolution removes the directive. Doing it for every entry would be a catch-all by another name, filing a misspelling as rubric content and publishing it to the service instead of reporting it.
EvalConfigFromService resolves includes itself rather than going through LoadEvalConfig, so the rescue landed on the CLI path only. That recreated the exact asymmetry this work started from, in mirror image: an include every CLI command accepted and 'azd up' refused with unknown key 'dimensions'. Both routes now agree, and a test reads the same include each way and compares. Confirmed it fails without the deploy-path call rather than passing vacuously.
…ated' into feat/eval-schema-registration
The rescue marked evaluator entries by index in the pre-resolution document. That cannot see the evaluators inside a config that is itself behind a ref -- which is the layout the README and the spec both document, so the documented shape still failed to deploy with unknown key 'dimensions'. It now keys on the entry carrying 'dimensions', the same key normalizeRubricBody insists on before it will treat a document as a definition. That drops the pre-scan entirely and works at any nesting depth. The gate is what separates this from a catch-all: a ref to something that is not a rubric is still reported rather than filed away as publishable content. Confirmed by removing the gate and watching that test go green.
Resolving a ref round-trips the document through a map, which expands aliases. An alias that expanded to nothing would silently drop the judge model shared across evaluators rather than fail, so the test asserts the parameters are equal and non-empty.
The CLI and 'azd up' reach a configuration by different routes and each resolved for itself, so every ref rule had to be added twice. Twice it was not: an include azd up accepted and every CLI command refused, and later the reverse. Everything between obtaining the map and using it now lives in resolveEvalRefs. A test counts the callers of ResolveFileRefs, because a second one is how the split comes back and it comes back silently.
…ated' into feat/eval-schema-registration
The base added scored_version_test.go against 'source:' while this branch renamed the dataset key to 'file:'. The merge was textually clean and semantically not: writeCatalog wrote a key the model no longer knows. Worth noting for the owner: the config failing to parse did not surface. localDatasetPath read it as 'not local' and the run was labelled with the recorded version -- the false provenance that test exists to prevent.
m7md7sien
changed the title
…ated' into feat/eval-schema-registration
m7md7sien
merged commit Aug 20, 2026
06282dc
into
feat/azure-ai-evaluations-consolidated
1 check passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Registers
azure.ai.evalwith azd's schema system, and makes$refmean the same thingon every path into the configuration.
Why
Six Foundry extensions ship a
schemas/directory and are registered inschemas/v1.0/azure.yaml.jsonby anif/thenblock onhost. This one shipped neither.hostis a free string with anexampleslist rather than anenum, sohost: azure.ai.evaldid not error — it was simply unrecognised, and nothing validated theservice entry, the
$ref, or anything under it.This came out of review feedback on the design spec asking whether the
$refapproach hadbeen closed with the azd team. It had not, because it was never opened: there was no
declared contract to close on. Declaring one is what surfaced everything below.
What is here
schemas/azure.ai.eval.jsondescribing the service body — the dataset and evaluatorcatalogs and the evals defined over them. Built from the Go types, not from the spec, so
it describes what the code actually accepts.
schemas/v1.0/andschemas/alpha/, following the five thinsiblings: one
if/thenblock plus the host added toexamples.schemas/examples/, the$refform and the inline form. Both validatewith the
ajvrecipe from the agentsschemas/README.md.$reffixes below, each with a test.$refnow means one thingazd core resolves
$reffor nobody — it hands each extension the entry with the directivestill in it. We called the shared resolver on the deploy path only, so
azd upaccepted aninclude that every CLI command then refused with
unknown key "$ref": one file, twomeanings, decided by which command opened it.
Fixing that naively reintroduced the same class of bug twice more, which is the substance
of this PR:
resolved document — inlining the include, orphaning the referenced file and losing its
comments. Reading to use and reading to modify are now separate.
$refnaming a rubric — the shape the spec documents and the shapegeneratedownloads — splices the file's keys into the entry, where the strict decoder rejected
them. They are now nested under
definition, gated on the key that marks a rubric so amisspelling is still reported rather than published to the service as rubric content.
which cannot see evaluators inside a config that is itself behind a
$ref— the layoutthe README documents. It is keyed on shape now and works at any depth.
initand never read back, so a fresh clonereported the configuration missing.
The deeper cause was two independent resolvers, one per path. There is now one, and
TestRefsAreResolvedInOnePlacefails the build if a second appears.Also
file:, notsource:.source:meant three different thingsacross the model.
options.eval_model, which has never been a key of thisdecoder — following the README produced
unknown key "options". It is now extracted anddecoded by a test rather than trusted.
reshapes the document.
Note on an unrelated pre-existing failure
azure.ai.agents/schemas/examples/complex.azure.yamldoes not validate against the currentschemas — its
toolboxes[]entries are strings whereToolbox.jsonrequires objects.Confirmed to fail on a worktree without this change, so it is pre-existing and untouched
here. Nothing catches it because
schema-ci.ymltriggers only onpaths: schemas/**andruns
jsonlintplus a doc-gen check; no CI runs theajvexamples for any extension.Validation
ajv:ref.azure.yamlPASS,inline.azure.yamlPASS.gofmtclean;go vetclean untagged and under bothheroandlive; fullgo test ./...green.vacuously.
For the azd owner
A design-level summary of the
$refcontract we implemented to, and the one decision wewould most like challenged, is in the review memo accompanying this PR.