memory:promote — production-vs-fixture-precision-fix - #772
Open
github-actions[bot] wants to merge 2 commits into
Open
memory:promote — production-vs-fixture-precision-fix#772github-actions[bot] wants to merge 2 commits into
github-actions[bot] wants to merge 2 commits into
Conversation
…fix) Precision regressions in mesh-scan routing evals can be fixed at two distinct loci: the real production tool descriptions in src/a2a/peer-bridge.js (affects every deployed agent) or the eval harness's synthetic caller prompt (affects only that fixture). PR #748 fixed the former for Issue #742 (3x-disjoint cell); the sibling no-hedge-single-peer-directive lesson fixed the latter for Issue #744 (6x-confusable cell). Cross-linked both docs, and flagged PR #748's reviewer-noted risk: its two added tool-description directives pull in opposite directions (don't-guess vs. must-delegate) and were verified only by a substring-match unit test, not a real eval-matrix re-run. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
Author
|
Triager classification —
No code change proposed; I don't edit code from this role. |
PR #772 stuck in mergeStateStatus=UNKNOWN for 3752min.
Contributor
Author
|
🤖 dev-mesh janitor: pushed an empty commit to nudge GitHub's mergeable recalc. If the state stays UNKNOWN after a few minutes, this PR likely needs a human. |
This was referenced Jul 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
[mesh-scan] perf-regression: precision regressed (-13.3%)([mesh-scan] perf-regression: precision regressed (-13.3%) #742)workflows/production-vs-fixture-precision-fix.md(new pattern)What this lesson captures
Issue #742 (3x-disjoint cell) and Issue #744 (6x-confusable cell, see
[[no-hedge-single-peer-directive]]) are sibling -13.3% precisionregressions, not duplicates — and they were fixed at two genuinely
different loci:
caller prompt (
eval/perf/harness.mjsbuildRoutingMesh) — only affectsthat fixture's agent.
descriptions in
buildTools()(src/a2a/peer-bridge.js) — thelist_peers/delegate_to_peerMCP descriptions every real deployed agentreads via the peer bridge, not just an eval fixture.
The lesson generalizes: trace a routing regression to its cause before
picking a fix locus — production tool descriptions when the bias is generic
(any worker can hedge/under-delegate), the eval fixture prompt when it's
specific to that eval's synthetic mesh.
It also flags two things PR #748 shipped with, per its own reviewer:
list_peerstext ("don'tguess in favor of a loosely-matching peer") and the new
delegate_to_peertext ("delegate even if you think you know theanswer") pull in opposite directions for a loosely-matching task, risking
swapping a precision fix for a recall regression (or vice versa) on a
different eval cell.
description strings landed (
assert.match), not that routing behavioractually changed. Confirming the regression is really fixed needs a real
scripts/eval-perf.mjsre-run on the 3x-disjoint cell, checking precisionand recall together.
Cross-linked
[[no-hedge-single-peer-directive]]↔[[production-vs-fixture-precision-fix]]so either doc surfaces the other.Validation
node scripts/validate-quick-memory.mjs dev-mesh/curator/memory/quick.json→ valid (no quick.json changes; workflow files only)workflows/instead for exactly that reason)Review checklist