Skip to content

v1.91.139.0 fix(evals): docsync observer accepts read-only git describe - #3276

Merged
capy-ai[bot] merged 3 commits into
mainfrom
capy/docsync-observer-accepts-read
Oct 18, 2026
Merged

capy-ai[bot] merged 3 commits into
mainfrom
capy/docsync-observer-accepts-read

Conversation

@garrytan

Copy link
Copy Markdown
Owner

The paid rule case ship-docsync-store went red on #3272 (run 39153452209, head a844821) for a reason that PR didn't cause. The /document-release child agent ran git describe --tags --always. That command is read-only, but it wasn't in the fixture's declared docs observation interface, so the observer failed the case with "command outside declared docs observation interface". GStack Minder's call (2026-10-18): it's a legitimate read, so this PR adds it to the interface. The child prompt is not steered away from it.

Change

Observer. test/helpers/docsync-observer.ts gains DOCS_DESCRIBE_FLAGS and a describe branch in the literal-command check. The observer prose in the docs interface now lists the same rule.

git describe form Verdict
--tags, --always, --abbrev=<digits>, --exact-match, in any combination, with at most one revision accepted
--dirty, --broken refused: on git 2.55 they take index.lock and rewrite .git/index
--contains, --all, a bare or non-numeric --abbrev, two revisions, -- HEAD refused

Tests. test/docsync-command-grammar.test.ts adds:

  • the failing run's 36 Bash commands, inlined, which must all pass the observer;
  • every accepted form, run against a fixture with an annotated tag and a stale file timestamp, asserting that neither the repository snapshot nor the raw bytes of .git/index change;
  • negative controls that must still fail the observer: git tag v9 HEAD, git update-index --refresh, git notes add -m x HEAD, --dirty, redirection (> tag.md) and composition (; git tag v9);
  • a companion check proving that --dirty really does write the index.

Without the observer change, the capture test and the read-only-forms test both fail.

Gate-integrity edits

Edit Effect
test/helpers/docsync-observer.ts: read-only git describe added to the docs observation interface (code allowlist and prose) widens one observer allowlist to the forms above; nothing else is tightened or loosened

No threshold, assertion, trial count, case kind or prompt instruction changes.

Checks

  • Focused paid rerun: ship-docsync-store passed 1/1 for $3.83. The child ran git describe --tags --always twice and the observer accepted both. It ran through test-paid-shards.ts --tier gate --case ship-docsync-store with the CI-pinned bun, node and Claude CLI 2.1.292.
  • Static checks: bun run typecheck is clean and typecheck:test reports no new diagnostics.
  • Guards: parity-suite, the SKILL.md size guards, the spawnSync tripwire, free-fixtures, the module-size ratchet and the test-of-test ratchet pass, 31/31.
  • Remote pregate: 2,583 tests, 0 failed.
  • Expected paid selection: docsync-spawned plus the 12 ship-docsync-* cases, and no judges.

Open in Capy

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@trunk-io

trunk-io Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

😎 This pull request was merged.

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

E2E Evals: ✅ PASS

[test:paid] VERDICT GREEN — lane gate/pr, attempt 1
  rule 13/13 · behavior 0/0 · judge 0/0 · quarantined 0 (0 failing the lane)
  SKIPPED 0 · INFRA 0 · INCOMPLETE 0 · unattributed 0 · ACTION REQUIRED 0
  wall 6m17s · cost $20.77

13 executed, 0 reused rule/judge records | 0 manual accepted (unscored; no score-cache credit) | $20.77 rule/judge cost | reconcile exit: 0

Profile: pr / mode pr; selected behaviors: 13, judges: 0. Deferred to scheduled/release coverage: 0 behaviors and 0 changed prompt files. Deferred checks did not run and receive no PR-pass credit.

Rule and judge shards
Shard Automated result Manual/unscored Executed Reused Status Cost
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-late-result 1/1 0 1 0 ✅ $0.9
e2e-docsync-spawned/skill-e2e-docsync-spawned 1/1 0 1 0 ✅ $1.19
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-missing-asset 1/1 0 1 0 ✅ $1.33
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-recovery 1/1 0 1 0 ✅ $0.94
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-missing-marker 1/1 0 1 0 ✅ $0.54
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-stale-after 1/1 0 1 0 ✅ $1.02
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-timeout-unsettled 1/1 0 1 0 ✅ $0.56
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-stale-before 1/1 0 1 0 ✅ $1.01
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-current 1/1 0 1 0 ✅ $3.28
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-failure 1/1 0 1 0 ✅ $2.15
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-launch-failure 1/1 0 1 0 ✅ $0.48
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-store 1/1 0 1 0 ✅ $3.76
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-completion 1/1 0 1 0 ✅ $3.61
Fail-closed reconciliation
[test:paid] report: 4/4 slices, 13 planned shards, tier=gate, attempt 1
[test:paid] coverage: profile=pr mode=pr; selected E2E=13, judges=0
[test:paid] deferred: 0 broad behaviors, 0 changed prompts without quick live coverage; these are not PR passes
  slice 1  passed            354s  test/skill-e2e-ship-docsync.test.ts#ship-docsync-completion
  slice 1  passed            368s  test/skill-e2e-ship-docsync.test.ts#ship-docsync-store
  slice 2  passed            301s  test/skill-e2e-ship-docsync.test.ts#ship-docsync-current
  slice 2  passed            226s  test/skill-e2e-ship-docsync.test.ts#ship-docsync-failure
  slice 2  passed            125s  test/skill-e2e-ship-docsync.test.ts#ship-docsync-stale-before
  slice 3  passed            126s  test/skill-e2e-ship-docsync.test.ts#ship-docsync-stale-after
  slice 3  passed            113s  test/skill-e2e-ship-docsync.test.ts#ship-docsync-recovery
  slice 3  passed             98s  test/skill-e2e-ship-docsync.test.ts#ship-docsync-late-result
  slice 3  passed             98s  test/skill-e2e-docsync-spawned.test.ts
  slice 3  passed            121s  test/skill-e2e-ship-docsync.test.ts#ship-docsync-missing-asset
  slice 3  passed             61s  test/skill-e2e-ship-docsync.test.ts#ship-docsync-timeout-unsettled
  slice 3  passed             70s  test/skill-e2e-ship-docsync.test.ts#ship-docsync-missing-marker
  slice 4  passed             47s  test/skill-e2e-ship-docsync.test.ts#ship-docsync-launch-failure
[test:paid] collector final outcomes: 13 executed, 0 reused; 13 passed, 0 failed, 0 manual accepted (unscored; no score-cache credit) (13 attempt records from 13 collectors; every record counts)
[test:paid] deselected testcases, zero credit (132):
[test:paid]    132  sibling case of a case shard (its own shard runs it)
[test:paid] VERDICT GREEN — lane gate/pr, attempt 1
  rule 13/13 · behavior 0/0 · judge 0/0 · quarantined 0 (0 failing the lane)
  SKIPPED 0 · INFRA 0 · INCOMPLETE 0 · unattributed 0 · ACTION REQUIRED 0
  wall 6m17s · cost $20.77
[test:paid] exercised-path map (shadow, non-blocking): 0 MAP_GAP line(s) over 13 executed case(s) with recorded commands
[test:paid] report: every planned shard accounted and passed

Sliced lane: planner → duration-packed executors → fail-closed report. Behavior cases run a pre-registered 3-trial panel (PASS at 2/3 with no contract violation); a PASS 2/3 is shown with its failed trial, never as a clean pass. Reused results retain their original provenance and expiry.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@github-actions github-actions Bot changed the title fix(evals): docsync observer accepts read-only git describe v1.91.139.0 fix(evals): docsync observer accepts read-only git describe Oct 18, 2026
@capy-ai
capy-ai Bot merged commit 3a64ab8 into main Oct 18, 2026
58 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant