Skip to content

feat: annotate every MCP tool with title + behavioural hints - #125

Merged
dzianisv merged 1 commit into
mainfrom
feat/tool-annotations
Aug 8, 2026
Merged

feat: annotate every MCP tool with title + behavioural hints#125
dzianisv merged 1 commit into
mainfrom
feat/tool-annotations

Conversation

@dzianisv

@dzianisv dzianisv commented Aug 8, 2026

Copy link
Copy Markdown
Member

Clears the first of the two hard blockers named in worklog/mcp-distribution-channels.md (#124): we emitted no tool annotations at all, failing the listing requirement on every tool in every directory.

What changed

tools/list used to emit only {name, description, inputSchema}. It now emits a top-level title plus a full annotations object on every tool. Attached in the ListToolsRequestSchema handler — the single choke point where the extension backend, the chrome-use --devtools backend, and the set_remote meta-tool converge.

Field names/semantics taken from the current spec (ToolAnnotationsSchema in the installed SDK: title, readOnlyHint, destructiveHint, idempotentHint, openWorldHint), not from memory.

Classification

Done by reading each implementation. 41 tool names classified (58 registry entries incl. alias spellings); 29 on the wire in the core profile. Counts over the exposed set: readOnly 17 · destructive 8 · openWorld 12 · idempotent 28.

Three calls worth review — all three contradict what the tool's name suggests:

Tool Call Why
click not read-only, open-world Dispatches a real DOM click — submits forms, follows links to any host, opens tabs. The extension arms a new-tab tracker specifically for it.
wait_for_condition not read-only, destructive, open-world Reads like an observer, but evaluates a caller-supplied JS expression in the page. Arbitrary code execution; it can fetch().
secrets_manager is read-only Reads like a mutator, but only supports list/read of credential metadata. Never writes, never returns plaintext.

Ambiguous ones and how I called them: press_key is destructive (Enter submits, Ctrl+W closes a tab); web_fetch is read-only but open-world (reads only, arbitrary host); upload_file is not read-only and open-world (pushes local file bytes into a page that may ship them anywhere); drag is destructive (can reorder or delete).

Unclassified tools from a future backend get the most cautious combination the spec allows, so a client prompts the user rather than silently shipping an unannotated capability.

Test — scripts/e2e-tool-annotations.mjs

Wired into test and test:ci. Given the #1856 false-green deletions, it proves itself in three parts:

  • A. contract — every exposed tool is classified, and no classification contradicts the implementation (a mutating tool cannot claim readOnly; an observer cannot be marked mutating; readOnly+destructive is rejected).
  • B. self-check — the part-A checks are shown to go red on 7 deliberately broken inputs. A check that cannot fail is itself a failure.
  • C. wire — boots a real server process, drives it with a real MCP client over streamable HTTP, asserts annotations are actually on the wire. Only Chrome is faked, and it advertises the genuine 27-tool core profile.

Nothing skips silently; missing prerequisites throw.

Fail-then-pass evidence

Injected a real unannotated tool into CHROME_USE_TOOLS, rebuilt:

Error: Missing MCP tool annotations for: dummy_unannotated_tool. Add an entry to TOOL_ANNOTATIONS in src/tool-annotations.ts.
    at assertAnnotationCoverage (dist/tool-annotations.js:231:15)
RED exit code = 1
=== dummy removed, rebuilt ===
GREEN exit code = 0

Also proved it catches misclassification, not just absence — flipping click back to readOnlyHint: true:

Error: Tool annotation contract failed:
  - click: marked read-only, but it mutates page or browser state
misclassification exit = 1
restored exit = 0

tools/list wire output (verbatim, part C)

[
  { "name": "take_screenshot", "title": "Take Screenshot",
    "annotations": { "title": "Take Screenshot", "readOnlyHint": true, "destructiveHint": false, "idempotentHint": true, "openWorldHint": false } },
  { "name": "click", "title": "Click Element",
    "annotations": { "title": "Click Element", "readOnlyHint": false, "destructiveHint": false, "idempotentHint": true, "openWorldHint": true } },
  { "name": "wait_for_condition", "title": "Wait For Condition",
    "annotations": { "title": "Wait For Condition", "readOnlyHint": false, "destructiveHint": true, "idempotentHint": false, "openWorldHint": true } },
  { "name": "secrets_manager", "title": "Read Saved Credential Metadata",
    "annotations": { "title": "Read Saved Credential Metadata", "readOnlyHint": true, "destructiveHint": false, "idempotentHint": true, "openWorldHint": false } },
  { "name": "web_fetch", "title": "Fetch Web Page",
    "annotations": { "title": "Fetch Web Page", "readOnlyHint": true, "destructiveHint": false, "idempotentHint": true, "openWorldHint": true } }
]

Build + full suite

$ npm run build
> tsc && node scripts/prepare-cli-package.mjs      # clean

$ npm run test:ci
FULL SUITE EXIT = 0
relay-race ok · relay-roundtrip ok · cli-relay ok · cli-autospawn ok · http ok
uuid-only-auth ok · tool-annotations ok (41 exposed, 29 on the wire)
tools-list-budget ok (3008ms) · browser-cli ok · cli-package PASS · devtools-flag ok

Release / version

Version deliberately left at 0.3.2. Annotations are runtime behaviour and are not carried in the registry manifest, so no npm release is needed to make them take effect — and this avoids the mcpb-vendors-npm release chain that CI correctly rejected on #124's first attempt. The live registry entry needs no republish.

Drive-by

Untracks dist/server.js, a build artifact committed by accident (dist/ is gitignored). Leaving it tracked would have meant committing a stale build alongside this change.

`tools/list` previously emitted only {name, description, inputSchema}. Both
the Anthropic Connectors directory and the OpenAI plugins directory require a
`title` plus accurate readOnly/destructive/openWorld hints on every tool, so
we failed that check on every tool in every directory — the cheapest of the
two hard blockers named in worklog/mcp-distribution-channels.md (#124).

Annotations are attached in the `ListToolsRequestSchema` handler, the single
choke point where the extension backend, the chrome-use `--devtools` backend,
and the `set_remote` meta-tool all converge on the wire.

Field names and semantics were taken from the current spec
(ToolAnnotationsSchema in @modelcontextprotocol/sdk types.d.ts: title,
readOnlyHint, destructiveHint, idempotentHint, openWorldHint) rather than
from memory, plus the top-level `Tool.title`.

Classification was done by reading each implementation, not by pattern-matching
the tool name. Three calls worth flagging:

- `click` is NOT read-only. It dispatches a real DOM click, which can submit a
  form, follow a link to any host, or open a tab — the extension arms a new-tab
  tracker specifically for it. Also open-world.
- `wait_for_condition` reads like an observer but evaluates a caller-supplied
  JavaScript expression in the page. Arbitrary code execution, so not read-only
  and open-world (it can `fetch()`).
- `secrets_manager` reads like a mutator but only supports `list`/`read` of
  credential *metadata*, never writes and never returns plaintext. It is
  read-only.

Unclassified tools from a future backend get the most cautious combination the
spec allows (writes / destructive / non-idempotent / open-world) so a client
prompts the user, rather than silently going unannotated.

Test: scripts/e2e-tool-annotations.mjs, wired into `test` and `test:ci`.
Given the #1856 false-green deletions, it proves itself in three parts:
  A. contract — every exposed tool is classified, and no classification
     contradicts the implementation (a mutating tool cannot claim readOnly,
     an observer cannot be marked mutating, readOnly+destructive is rejected).
  B. self-check — the part-A checks are shown to go red on seven broken
     inputs, so a check that cannot fail is caught.
  C. wire — boots a real server process and drives it with a real MCP client
     over streamable HTTP; only Chrome is faked, advertising the genuine
     27-tool core profile. Asserts the annotations are actually on the wire.
Nothing skips silently; missing prerequisites throw.

Version deliberately left at 0.3.2: annotations are runtime behaviour and are
not carried in the registry manifest, so no npm release is required and the
mcpb-vendors-npm release chain that CI rejected in #124 is not triggered.

Also untracks dist/server.js, a build artifact committed by accident (dist/ is
gitignored); leaving it would have meant committing a stale build.
@dzianisv
dzianisv merged commit 18bb9de into main Aug 8, 2026
1 check passed
@dzianisv
dzianisv deleted the feat/tool-annotations branch August 8, 2026 04:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant