datasets: define the agent-trace schema and seed the first entries

[gautamgb]

Summary

Defines the data format for datasets/agentdataflow_toolexchange_traces/, which has been open for contributions with a <!-- TODO: Define schema once initial data is contributed --> where its format should be, and ships the tooling that enforces it.

Split as requested: this PR is the format. The two seed entries and index.csv move to a follow-up that depends on it.

File
schema.json	JSON Schema draft 2020-12
validate.py	Schema, span graph, provenance, citation form, date, and secret-scan checks
build_index.py	Index generator, same pattern as exploit_dataset; index.csv ships with the entries
example.json	Worked template, validated by validate.py
README.md	Data Format section written; Scope and Sanitization sections unchanged
.github/workflows/dataset-agentdataflow-validate.yml	Runs validate.py and checks index.csv is current, path-filtered to this dataset

What changed since the last review

The self-citation is gone. The previous revision backed every trace precondition with Zenodo record 10.5281/zenodo.21449149 and two mcpindex.ai pages. That record is my own work, and the description should have said so. It called it a public CC BY 4.0 contract-drift corpus and left the ownership out. Those citations are out of example.json here, and out of both entries in the follow-up.

Preconditions are now cited to the Model Context Protocol specification (2025-06-18), its schema reference, and CVE-2025-54136:

• destructiveHint is defined in the MCP schema reference (ToolAnnotations) as a plain server-declared boolean, true meaning the tool may perform destructive updates and false meaning it is additive-only, defaulting to true. Re-declaring it is within what the server is always allowed to do; nothing about it is a client-verified fact.
• The specification's trust warning is conditional, and the entry now quotes the condition in full: "clients MUST consider tool annotations to be untrusted unless they come from trusted servers." A server the client has already bound is, by construction, one it currently trusts. The earlier draft dropped that clause and overstated the finding; what actually holds is narrower and, I think, stronger - the spec doesn't say whether that trust gets re-evaluated when the annotation it was based on later changes, and the trace shows a system where it doesn't.
• A tool definition carries no version field, and the only change signal is notifications/tools/list_changed, a SHOULD from servers that declared listChanged. A client holding a binding from an earlier tools/list has neither a guaranteed notification nor a version to compare.
• outputSchema obligations attach to the current schema on both sides: servers MUST conform to it, clients SHOULD validate against it. A client validating against what the server publishes now cannot see that the schema is wider than the one the agent bound to.
• CVE-2025-54136: an MCP entry a collaborator had already accepted could be swapped for a different command with no warning and no re-prompt (Cursor 1.2.4 and below, fixed in 1.3). Both entries that cite it now say plainly what it does not establish: the CVE requires an attacker with repository or local write access, so it backs an adversarial route to the same end-state, not the non-adversarial mechanism the trace models.

No prevalence claim survives anywhere. Nothing in the dataset now says how often any of this happens in the wild, only what the protocol permits and that one instance of the class reached shipped software.

Three schema and validator changes so the same class of mistake is harder to make, by me or by the next contributor:

• provenance.evidence[].supports is required. A citation cannot be dropped in without stating the narrow claim it backs.
• provenance.tier's derived_from_public_observation is renamed to derived_from_public_documentation. Most of what backs these two entries is a normative reading of a specification, what a protocol permits, not an observation of what happened, and pooling those under a tier named "observation" overstates what the evidence supports. The definition now says the source can be normative or empirical, as long as it's independently checkable.
• The README now states the ownership rule inline: a source the contributor owns or controls does not establish a precondition, whatever it says, and this is a review-time obligation - validate.py cannot check it. The earlier draft pointed at a repository-wide vendor-neutrality policy that does not exist as a document today; I said the rule plainly in both the README and the schema description instead.

validate.py also checks that a citation looks like the kind it claims to be (doi, cve, ghsa, url), catches a span that names itself as its own parent, checks ISO 8601 on every span timestamp in addition to date_added/date_observed (hand-rolled, so it works without the optional rfc3339-validator package), scans for internal-hostname suffixes (.internal/.corp/.lan/.intra) and credential-shaped key=value assignments in addition to the original patterns, flags two or more addresses in one string that look like real routable public IPv4s (RFC 5737 documentation ranges excluded on purpose), makes a backwards t_offset_ms an error (it was a silent warning), and prints a warning when the shared taxonomy file is missing (it used to no-op silently). It passes with entries/ empty, so this PR stands on its own, and the new CI workflow runs it on every push and PR to the dataset so that's true of the merged tree too, not only of whoever ran it locally.

Schema direction, still your call

data_validation/schemas/agentdataflow_trace.schema.json exists and describes this dataset. So does the dataset-local schema here, and #67 takes the opposite position on which is authoritative.

The repo already has two schema systems and they already disagree, before either PR. datasets/exploit_dataset/ ships its own schema.json with seven live entries validated against it, and those same entries fail data_validation/schemas/exploit.schema.json. Taking the merged entries/AML.T0010.json against the central schema:

6 errors
  <root>                 'name' is a required property          (local schema uses 'title')
  <root>                 'source' is a required property        (local schema uses 'source_urls')
  exploit_id             'AML.T0010' does not match '^EXP-\d{4}$'
  impact                 [...4 values...] is not of type 'string'
  prerequisites          [...1 value...] is not of type 'string'
  detection_indicators   [...4 values...] is not of type 'string'

The central schemas describe an earlier design that the dataset-local ones superseded in practice. The dataset-local one is the one with data behind it, and that is the precedent this PR follows.

What I aligned: the category enum uses the central schema's values (delegation_chain -> multi_agent_delegation, plugin_exchange -> plugin_data_exchange, memory_io -> memory_read_write).

What I kept, and why:

Central stub	Here	Why
trace_id ^TRACE-\d{4}$	DSGAI-TRACE-<slug>	Sequential identifiers collide when contributors work in parallel. exploit_dataset abandoned EXP-\d{4} for the same reason.
trace_data, array of untyped objects	spans, typed	An untyped array cannot express actor, operation, data class, or where a finding lands, which is most of what makes a trace analyzable.
type: benign / adversarial	disposition, plus unintentional_failure	Most data-security failures have no adversary. Forcing them into benign or adversarial loses the majority case.
security_observations: string	array of strings	One falsifiable claim per item, so they can be counted and checked.
sensitivity_annotations: top-level array	per-span data_classes + sensitivity	Sensitivity is a property of a step. A trace-level array cannot say which step was sensitive.

The three options from the earlier description still stand:

1. Leave the drift. Dataset-local schemas are authoritative and the central ones remain historical stubs. Cheapest, and it matches what already happens.
2. I open a follow-up replacing data_validation/schemas/agentdataflow_trace.schema.json with a pointer to the dataset-local one, and file an issue recording the same drift for exploit_dataset.
3. Reconcile properly: one schema per dataset, centrally held, dataset folders reference it. Much larger, and your call whether it earns its cost.

I suggested 2 and I am happy to do the work. I have no objection to #67 being the PR you take this from instead. If the answer is the central schema, the entries re-target it and the evidence rules above apply either way.

Provenance tiers

Unchanged in purpose from the earlier description, renamed as noted above. The people holding real agent traces mostly cannot publish them, because production telemetry contains exactly what the anonymization rule forbids. A dataset that accepts only production telemetry stays empty; one that accepts anything fills with plausible fiction.

• observed_production - sanitized telemetry from a real deployment
• observed_lab - sandbox, test environment, CTF, red-team exercise
• derived_from_public_documentation - the mechanism rendered as a trace, its precondition carried as independently checkable evidence, normative or empirical
• hypothetical - illustrative only, excluded from any statistic

Every tier except hypothetical requires at least one evidence item, enforced by validate.py, and each item has to state what it supports.

Two smaller choices, both easy to reverse

1. sanitization.attestation is const: true, so an entry cannot merge with the anonymization box quietly unticked, and validate.py scans for credential-shaped, PII-shaped, and now internal-hostname-shaped values independently of the attestation. The scan covers the whole entry: summary, notes, collection_method, and evidence locator are where a real hostname actually gets typed, because that is the prose someone writes while looking at a live trace. It is pattern matching and not a guarantee; the README now says so plainly rather than implying full coverage. Related to The PR template asks contributors to certify a validation script that cannot fail #61.

2. mitre_atlas_mapping is release-pinned: {atlas_release, techniques: [{id, name_at_release}]} rather than a flat list of identifiers. Correct eleven MITRE ATLAS technique citations #59 and ATLAS mappings needing a judgement call, and an offer of the verifier behind #59 #60 make the case better than I could. Happy to flatten it to match exploit_dataset if you would rather have cross-dataset consistency than the pin.

Verification

$ python validate.py
OK: example.json validated against schema. entries/ is empty.

$ python build_index.py
Wrote index.csv with 0 entries.

With the follow-up entries branch checked out on top:

$ python validate.py
OK: 2 entries + example.json validated against schema.

$ python build_index.py
Wrote index.csv with 2 entries.

Sixteen deliberately broken copies, each one error and a non-zero exit, no false positives on the real files: a span naming itself as its own parent, a backwards t_offset_ms (now an error, was a warning), a cve-typed citation that is not a CVE id, a url-typed citation that is not an absolute URL, an evidence item with no supports, an internal hostname (prod-db-07.corp.internal) in a span summary, a credential-shaped assignment (password=hunter2) in prose, two distinct addresses that look like real routable IPv4s in one string, a malformed span timestamp, plus the original leaked bearer token, RFC1918 address, email, home path, non-ISO date_added, off-enum category, missing evidence on a derived-tier entry, attestation: false, and filename/trace_id mismatch. A missing taxonomy file now prints a warning instead of silently skipping the check, and RFC 5737 documentation-range IPs (192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24) were confirmed NOT to trip the new IP scan, since those are the correct way to write a fake-but-obviously-fake address in an example.

data_validation/run_all_checks.py is still a stub, so the dataset-local validator is what ran, matching the exploit_dataset pattern, and it now also runs in CI on every PR to this dataset.

Checklist

• Data is anonymized (no PII, credentials, or org names). example.json is fully synthetic, payloads are typed placeholders by convention, and validate.py enforces it across the whole file. The only third party named anywhere is the vendor in the cited CVE, which is already public.
• Entries reference at least one DSGAI ID where applicable. Required by schema, cross-checked against _shared/dsgai_taxonomy.json, and every span-level finding must map to a risk the trace declares.
• Validation scripts pass. Output above.
• One entry per file for dataset contributions. entries/<trace_id>.json, filename agreement enforced by the validator.

---

LICENSE still reads NOASSERTION and #44 is in draft. Which license should I take this contribution as being accepted under? Everything here is original work written for this repo, so I do not think anything is blocked either way.

Separately: every validator in data_validation/validators/ is currently a stub with a # TODO (schema_validator, anonymization_scanner, dsgai_mapping_check, completeness_check). The validate.py here implements roughly what those four promise, and now runs in CI, so lifting it into data_validation/ would close #61's underlying complaint. Separate PR, if you want it.

[gautamgb]

Revised the description and force-pushed a corrected commit before anyone had to read the first one.

Self-review turned up three things: data_validation/schemas/agentdataflow_trace.schema.json already exists and I had missed it (now addressed head-on in the description, along with evidence that the same drift already affects exploit_dataset); the secret scan only walked span payloads and left the free-text fields unscanned; and "format": "date" was decorative, since jsonschema does not enforce format without a format checker, so a garbage date_added passed. All three are fixed and the negative suite is now twelve cases.

[emmanuelgjr]

Thank you for this work. The schema design is thoughtful, and validate.py is solid: it has no network or subprocess calls and scans the whole entry for secrets. We verified the corpus figures you cite (123/25, 992/2,503, 62.4%) against the Zenodo record's aggregates.json, and they check out. Two things block merging.

1. Vendor-neutrality policy. This is a global, vendor-neutral OWASP repository. Merged files credit contributors by name only. Contributor-owned sources such as your own corpus, product or website can't serve as evidence unless they are independently established. Zenodo record 10.5281/zenodo.21449149 ("mcpindex Drift Report") lists you as its sole creator, and the entries also link to https://mcpindex.ai/drift-report and https://mcpindex.ai/methodology. The PR description calls it "a public CC BY 4.0 contract-drift corpus" without saying it's yours.

Requested changes:

• Remove the mcpindex.ai links from all entries and the example.
• Remove the self-authored Zenodo corpus as provenance.evidence. Instead, back each precondition with independent public sources (for example the MCP specification's destructiveHint semantics, published advisories, or third-party research). If you can't, mark the traces hypothetical.
• Keep contributor.name as your name; that's fine.

2. Schema direction needs a maintainer decision first. This PR and #67 both define the format for agentdataflow_toolexchange_traces/. They conflict in the README and take opposite positions on the central data_validation/schemas/agentdataflow_trace.schema.json. Maintainers will decide between dataset-local and central schemas (your options 1–3) before either PR merges. When you update, please split the PR: first the schema, validator and README, then the entries in a follow-up. That keeps review of the format decision separate from review of the data.

Thanks again, and happy to re-review once updated.

[gautamgb]

Both points taken, and I went back through my own work harder than the first pass. Updated, and the PR is now split.

1. Vendor neutrality. You are right and I should have disclosed it. The Zenodo record is my own work, and describing it in the PR as a public CC BY 4.0 contract-drift corpus without saying whose it was is the part that should not have happened. The DOI and both mcpindex.ai links are gone from example.json and from both entries.

Every precondition is now carried by a source you can check without me. destructiveHint is defined in the MCP schema reference as a plain server-declared boolean, true for destructive and false for additive-only, default true - so re-declaring it is within what the server is always allowed to do. A tool definition carries no version field, and the only change signal is notifications/tools/list_changed, a SHOULD from servers that declared listChanged. outputSchema obligations attach to the current schema on both sides: servers MUST conform, clients SHOULD validate. And CVE-2025-54136 (GHSA-24mc-g4xr-4395): an MCP entry a collaborator had already accepted could be swapped for a different command with no warning and no re-prompt, Cursor 1.2.4 and below, fixed in 1.3.

I also caught a real problem with my own first pass at this, past what you flagged: the trust-warning citation dropped the specification's own conditional. It says untrusted unless the server is trusted, and the server in this trace is one the client already bound and trusts, so the annotation isn't forbidden - what's missing is any statement that the trust gets re-checked when the annotation later changes, which is the actual mechanism the trace shows. I rewrote that evidence item to say the narrower, correct thing instead of the broader claim I'd written the first time. Both entries citing the CVE now also say what it does not establish: it needs an attacker with write access, so it backs an adversarial path to the same end-state, not the non-adversarial one the traces model.

Every count is gone with the corpus, and I renamed provenance.tier's derived_from_public_observation to derived_from_public_documentation, because most of what backs these entries now is a normative reading of a specification - what a protocol permits - and calling that "observation" claims more than it should. The tier still takes either kind of source, normative or empirical, as long as it's independently checkable.

Two format changes came out of this. provenance.evidence[].supports is required now, and a citation without the narrow claim it backs fails the schema. The README also states the ownership rule inline: a source the contributor owns or controls does not establish a precondition, that is a review-time call, and validate.py cannot check it. I checked the repo for the vendor-neutrality policy I had pointed at in the earlier draft. It does not exist as a written document; the only two hits for that phrase were the two places I had just written it myself.

2. Split. This PR is now README.md, schema.json, validate.py, build_index.py, example.json, plus a new CI workflow that runs validate.py on every push and PR to this dataset. The README said evidence was enforced by validate.py, and nothing had actually been running it in CI. validate.py used to exit 1 on an empty entries/, which would have made a format-only PR fail its own check; it now validates example.json on its own and passes. The two entries and index.csv are on a follow-up branch that I will open once you have picked a direction, since they have to target whichever schema wins.

While I was in there I also fixed four things in the validator that had nothing to do with your review but should not have shipped: a span could name itself as its own parent and pass; the scan had no pattern for internal hostnames or credential-shaped key=value text even though the README promised it; a backwards t_offset_ms was a silent warning, not a failure; and a missing taxonomy file made the taxonomy check disappear with no message instead of telling you it did not run. All four are covered by new break-test cases alongside the original twelve.

On the direction itself I have nothing to add to the three options in the description, and no objection to #67 being the one you take it from. If the answer is the central schema, the entries re-target it and the evidence rules above hold either way.

One thing still open from last time: LICENSE reads NOASSERTION and #44 is in draft. Which license should I take this contribution as being accepted under?

[emmanuelgjr]

Re-reviewed the revised PR end to end. Verified locally: validate.py passes with entries/ empty; a deliberately broken entry (self-parent span, credential-shaped assignment, internal hostname) fails with exit 1 and precise messages; no reference to the removed corpus or its links remains anywhere in the diff; the CI workflow is SHA-pinned, path-filtered, contents: read, plain pull_request. The rewritten evidence is exactly what we asked for — narrow claims against the MCP specification and CVE-2025-54136, each stating what it does and does not establish — and the ownership rule now written into the README says the policy better than we had it written anywhere. Thank you for taking the vendor-neutrality point the way you did.

Answers to your two open questions:

1. Schema direction: option 2. Dataset-local schemas are authoritative — that is the precedent exploit_dataset set, and yours is the design with enforcement behind it. Please go ahead with the follow-up replacing data_validation/schemas/agentdataflow_trace.schema.json with a pointer to the dataset-local schema, and file the issue recording the same drift for exploit_dataset. The entries follow-up (both traces + index.csv) is welcome as soon as this merges; the derived_from_public_documentation tier as you've redefined it is accepted. Keep mitre_atlas_mapping release-pinned — #59/#65 settled that argument in favor of the pin.

2. License. This contribution is accepted under the repository's license, CC BY-SA 4.0 (the LICENSE file is the CC BY-SA summary; GitHub's NOASSERTION is a detection artifact). Should #44 land, code files (validate.py, build_index.py, the workflow) would be relicensed Apache-2.0 — reply here confirming you're fine with that and the record is complete either way.

One optional follow-up: the break-test cases you describe aren't committed. With CI now running the validator, a small tests/ of known-bad fixtures would keep them from rotting.

Merging. #67 will be asked to re-target TRACE-0001 to this format — its adversarial, non-MCP trace is variety this dataset wants.