Skip to content

test(agent): collect tests/verify_*.py so CI runs the VULN regression suite - #39

Open
schenksj wants to merge 3 commits into
mainfrom
fix/collect-verify-regression-tests
Open

schenksj wants to merge 3 commits into
mainfrom
fix/collect-verify-regression-tests

Conversation

@schenksj

@schenksj schenksj commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Closes #38.

What

  • Collect the regression suite. vulnhunter-agent/pyproject.toml now sets python_files = ["test_*.py", "verify_*.py"]. Before this change, none of the 11 tests/verify_NNN_*.py files (VULN-001…013, 51 tests) were collected by bare pytest, so CI never ran them. Keeping the verify_NNN names, rather than renaming the files, preserves their link to the findings.
  • Make two verify_012 tests able to fail. test_bearer_header_redacted and test_access_token_query_redacted used ghp_-prefixed tokens. The raw-token-prefix pass masks those no matter what, so the Bearer and query-param passes had no effective test coverage. Disabling either pass in agent/_url.py left the whole suite green. They now use opaque tokens.
  • Dropped an unused import pytest in verify_012 left over from Add client_secret redaction pass to credential scrubber #17.
  • Stop verify_013 from leaking the process umask. Its tests call os.umask(0o022) and never restored the previous value. That didn't matter while the file wasn't collected, but now the changed umask would carry into every test that runs after it. An autouse fixture now saves and restores the umask. Checked under umask 077: a probe test that runs after verify_013 fails without the fixture and passes with it.

Verification

Result
pytest on main 1139 passed, 0 collected from verify_*
pytest on this branch 1190 passed (+51, all verify_*), 1 skipped
Mutation: Bearer pass disabled test_bearer_header_redacted fails (passed before this change)
Mutation: access_token query pass disabled test_access_token_query_redacted fails (passed before this change)

All newly collected tests pass, so this adds coverage without breaking CI.

🤖 Generated with Claude Code

… suite

pyproject restricted python_files to test_*.py, so the 11 verify_NNN_*.py
security regression files (51 tests) were never collected by bare pytest
or CI. Add verify_*.py to python_files.

While enabling them, two verify_012 cases were found to be vacuous: the
bearer-header and access_token-query tests used ghp_-prefixed tokens, which
the raw-prefix pass masks regardless, so disabling either pass left them
green. Switch them to opaque tokens so each exercises its own pass, and drop
an unused pytest import.

Closes #38

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@schenksj
schenksj requested a review from a team as a code owner October 5, 2026 15:28
@CLAassistant

CLAassistant commented Oct 5, 2026 •

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ schenksj
❌ xxxxxxxxxxxxx
You have signed the CLA already but the status is still pending? Let us recheck it.

schenksj and others added 2 commits October 5, 2026 11:44
verify_013 forces os.umask(0o022) and never restored it. Harmless while the
file was uncollected; now that CI runs verify_*.py, the changed umask leaked
into every later test in the process. Save and restore it with an autouse
fixture.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@schenksj
schenksj requested a review from mihirv0ra October 5, 2026 15:51

@srujan-2025 srujan-2025 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 3ff5477. No blocking findings. Independently verified that collection adds all 51 security regression tests and the full agent suite passes (1,190 passed, 1 skipped). Confirmed the updated redaction tests detect disabled Bearer/query-token passes, and the umask fixture restores the caller’s setting even after test failure. The changes look correct and appropriately scoped.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

vulnhunter-agent: tests/verify_*.py regression suite is never collected by pytest/CI

4 participants