Skip to content

Isolate docs eval tools and retain final evidence - #677

Merged
nicholasdbrady merged 4 commits into
mainfrom
nicholasdbrady-issue-631-final-evidence
Jul 30, 2026
Merged

nicholasdbrady merged 4 commits into
mainfrom
nicholasdbrady-issue-631-final-evidence

Conversation

@nicholasdbrady

@nicholasdbrady nicholasdbrady commented Jul 30, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • expose only each selected MCP server's exact fully qualified documentation tools; retain deferTools: "never", one config, built-in MCP disablement, and server permission
  • enforce exact configured tool membership in runner, scorer, and Azure proof
  • retain bounded post-result metadata: line/type/ephemeral plus structured server states containing canonical name/status, every supplied identity/status alias, exact error-field names/presence, total counts, truncation, and shape validity
  • mcp.tools.list_changed requires exactly the selected server identity, no status, no error/failure fields, and ephemeral=true
  • status changes allow only selected connected/ready or builtin github-mcp-server disabled, with consistent aliases and no error fields
  • loaded summaries require every entry named and typed, selected server present, no foreign/duplicate/error entries, and each entry exactly selected connected/ready or builtin disabled
  • reject conflicting identity/status aliases and alias arrays beyond runner-representable cardinality; scorer independently enforces all lifecycle rules
  • fail closed on semantic, unknown, non-ephemeral, failing lifecycle, duplicate-result, or truncated post-result evidence
  • compute Azure selected-search proof before other validity short circuits and retain it on invalid/timeout rows without changing row validity

Artifact 8755138625 cannot reveal prior exact post-result types because all affected excerpts truncate before the result; this PR closes that gap conservatively.

Validation

  • python -m pytest tests\test_docs_eval_evidence.py -q — 322 passed, 2 POSIX-specific tests skipped on Windows
  • python -m ruff check scripts\run_docs_eval.py scripts\eval_scorer.py tests\test_docs_eval_evidence.py — passed
  • runner/scorer tests cover allowed selected/builtin cases and foreign, unnamed, duplicate, error-bearing, shape-invalid, alias-conflicting, and oversized-alias cases
  • exact selected-tool isolation and Azure invalid/timeout proof tests retained

Do not merge or rerun the full matrix until reviewed.

Expose only exact selected MCP tools, retain bounded post-result metadata with conservative semantics, and credit Azure search proof independently of row validity.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 2fce2ecd-5a59-4d80-96ff-a5e9e09bc60d
Fail closed on truncated post-result tails, require selected or builtin lifecycle identities, and enforce exact configured tool membership in scorer and Azure proof.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 2fce2ecd-5a59-4d80-96ff-a5e9e09bc60d
Retain structured identity, status alias, and error-field evidence so runner and scorer reject foreign, unnamed, duplicate, conflicting, or malformed MCP lifecycle postambles.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 2fce2ecd-5a59-4d80-96ff-a5e9e09bc60d
Assert scorer rejects foreign and error-bearing loaded server summaries alongside foreign and unnamed lifecycle metadata.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 2fce2ecd-5a59-4d80-96ff-a5e9e09bc60d
@nicholasdbrady
nicholasdbrady merged commit fddee8a into main Jul 30, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant