Skip to content

feat: seal codex and scheduled-agents baselines, and fix the test collision - #69

Merged
imran-siddique merged 1 commit into
mainfrom
fix/test-module-collision-and-sealing
Aug 1, 2026
Merged

feat: seal codex and scheduled-agents baselines, and fix the test collision#69
imran-siddique merged 1 commit into
mainfrom
fix/test-module-collision-and-sealing

Conversation

@imran-siddique

Copy link
Copy Markdown
Contributor

Two independent changes, one per commit. Splitting them across PRs would have meant two CI rounds for work that touches the same test files, so they are here together and reviewable separately.

1. Baseline sealing for the two engines that lacked it

claude-code has sealed its baseline since #64. Codex and scheduled-agents did not, so anything able to write their state could add a component to the approved set and the check would report "nothing added, nothing subtracted" from then on, indefinitely and quietly.

That gap mattered most for scheduled-agents, whose entire subject is the things that run when nobody is watching.

Both now seal on write via core.save_baseline and check on read. A broken seal is reported ahead of any drift result, in both the hook message and verify: if the baseline was altered then the comparison against it means nothing, and a reassuring "no changes" underneath would be worse than no result at all.

Both approve commands now print the baseline digest with instructions to record it off the machine. That is the only part of this that survives an attacker who owns the state directory, since anyone who can write the file can also reseal it.

Sealing writes without checking reads would have been decorative, so the tests cover the check rather than the wiring: an edited baseline is detected, verify states integrity before drift, and the scheduled-agents hook warns instead of reporting no drift when the baseline was tampered with. That last test plants a PreToolUse hook command into the baseline, which is the realistic attack on that surface.

2. Two test collisions, one root cause each

Four engines each define a module named capture, and three of their suites were named test_capture.py.

Module name. sys.path insertion meant import capture resolved to whichever suite pytest collected first, so a root-level run tested each suite against another engine's code. Each suite now loads its engine by path under a unique module name.

File basename. With that fixed, collection still failed because pytest could not derive distinct module names from three identical filenames. Renamed to test_claude_code_capture.py, test_codex_capture.py and test_scheduled_capture.py, matching the copilot suite.

A single root-level pytest now works for the first time in this repo: 184 passed. Per-suite runs are unchanged, which is how CI invokes them.

Test plan

Suite Result
claude-code 59 passed
agentrust-codex 29 passed (+4)
scheduled-agents 26 passed (+5)
copilot 25 passed
capture-core 45 passed
all in one run 184 passed
  • ruff check --target-version py39 clean across every path this PR touches. The 21 errors a repo-wide run reports are all in sentinel, comply54 and decisionassure, untouched here.
  • Vendored sync check passes across all four engines.

The one codex failure is the pre-existing TRACE conformance check that fix/codex-trace-pins addresses.

What is left after this

TRACE emission for the Copilot check, so it earns an integration.yaml honestly, and publishing agentrust-capture-core to PyPI. The second needs credentials and is yours; nothing breaks until it happens, because every engine carries the vendored fallback.

…lision

Two independent changes. Splitting them across PRs would have meant two CI rounds
for work that touches the same test files, so they are separate commits here.

## Baseline sealing for the two engines that lacked it

claude-code has sealed its baseline since #64. Codex and scheduled-agents did not,
so anything able to write their state could add a component to the APPROVED set and
the check would report "nothing added, nothing subtracted" from then on.

That gap mattered most for scheduled-agents, whose whole subject is the things that
run when nobody is watching.

Both now seal on write via core.save_baseline and check on read. A broken seal is
reported AHEAD of any drift result, in the hook message and in verify, because if
the baseline was altered then the comparison against it means nothing and a
reassuring "no changes" underneath would be worse than no result at all. Both
commands print the baseline digest with the instruction to record it off the machine,
which is the only part that survives an attacker who owns the state directory.

Sealing writes without checking on read would have been decorative, so there are
tests for the check, not just the write: an edited baseline is detected, verify
states integrity before drift, and the scheduled-agents hook warns instead of
reporting no drift when the baseline was tampered with. That last test plants a
PreToolUse hook command in the baseline, which is the realistic attack on that
surface.

## Test module and file collisions

Four engines each define a module named `capture`, and three of their suites were
named test_capture.py. Two separate problems, both fixed.

sys.path insertion meant `import capture` resolved to whichever suite pytest
collected first, so a root-level run tested each suite against another engine's
code. Each suite now loads its engine by path under a unique module name.

The shared test_capture.py basename then still broke collection, since pytest could
not derive distinct module names. Renamed to test_claude_code_capture.py,
test_codex_capture.py and test_scheduled_capture.py, matching the copilot suite.

A single root-level pytest run now works for the first time: 184 passed. Per-suite
runs are unchanged, which is how CI invokes them.

Suites: claude-code 59, codex 29, scheduled-agents 26, copilot 25, core 45. The one
codex failure is the pre-existing TRACE conformance check that fix/codex-trace-pins
addresses.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Imran Siddique <imran.siddique@opaque.co>
@imran-siddique
imran-siddique merged commit a3c9b19 into main Aug 1, 2026
20 checks passed
@imran-siddique
imran-siddique deleted the fix/test-module-collision-and-sealing branch August 1, 2026 04:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant