Skip to content

datasets: seed the first two agent-trace entries - #72

Open
gautamgb wants to merge 1 commit into
GenAI-Security-Project:mainfrom
gautamgb:dataset/agent-trace-entries
Open

gautamgb wants to merge 1 commit into
GenAI-Security-Project:mainfrom
gautamgb:dataset/agent-trace-entries

Conversation

@gautamgb

Copy link
Copy Markdown
Contributor

Summary

The follow-up promised in #62: the two seed entries and index.csv, now that the schema and validator have merged.

  • DSGAI-TRACE-mcp-declared-effect-flip-stale-autoapproval - a tool re-declares destructiveHint: true after an approval policy has already cached it into an auto-approve class, and the next call runs destructively with no human asked.
  • DSGAI-TRACE-mcp-output-schema-expansion-context-overshare - a tool's output schema widens under a stable server identity, and a schema-agnostic context append carries three unentitled fields into the model context, the user-facing answer, and the observability sink.

Both are derived_from_public_documentation: fully synthetic sessions built around a precondition the MCP specification itself establishes (annotations untrusted from an untrusted server, no version field on a tool definition, tools/list_changed as a SHOULD, outputSchema obligations attaching to whatever schema is current), plus CVE-2025-54136 for the first entry, with the entry stating plainly what the CVE does and does not establish about the non-adversarial mechanism it illustrates.

index.csv is rebuilt against main as it stands now, so it includes #67's entry alongside these two - three rows total.

Verification

$ python validate.py
OK: 3 entries + example.json validated against schema.

$ python build_index.py
Wrote index.csv with 3 entries.

Rebased on the current main tip in a clean worktree (not cherry-picked over a stale base), so there's nothing here that predates #67's entry or the merged schema.

Two traces of a tool contract changing under an agent that had already bound
it. One: a tool re-declares destructiveHint true after an approval policy
cached it into an auto-approve class, and the next call runs destructively
with no human asked. Two: a tool's output schema widens under a stable
server identity, and a schema-agnostic context append carries three
unentitled fields into the model context, the user-facing answer and the
observability sink.

Both sessions are fully synthetic. The preconditions are cited to the MCP
specification - destructiveHint's own definition and default, annotations
untrusted from an untrusted server, a tool definition carrying no version,
tools/list_changed as a SHOULD - and to CVE-2025-54136, where a once-granted
approval of an MCP entry survived a later change to what was approved; both
entries state plainly what the CVE does and does not establish about this
trace's mechanism.

index.csv regenerated against the merged schema PR and GenAI-Security-Project#67's entry, three
rows total.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant