test: add false-positive regression corpus#34
Merged
fahadsiddiqui merged 1 commit intoJul 21, 2026
Conversation
jesse-quinn
force-pushed
the
test/false-positive-corpus
branch
from
July 19, 2026 03:16
669f13c to
87803f5
Compare
Add tests/false-positive-corpus.test.ts and a fixtures module covering realistic benign snippets that must produce zero findings against the full default rule set (15 built-in rules plus 30 external coding-only rules). The corpus was built empirically: every snippet was scanned before inclusion and only genuinely zero-finding strings are kept. Realistic near-misses that DO fire under the current rules are documented as KNOWN_FALSE_POSITIVES and kept out of the corpus rather than changing any rule - this pins today's behaviour, it does not fix rule bugs. A set of true-positive canaries (one per built-in category) asserts detection still fires so the corpus self-checks that the scanner was loaded correctly. Test-only change; no src/, scripts/, hooks/, or dist/ touched.
fahadsiddiqui
force-pushed
the
test/false-positive-corpus
branch
from
July 21, 2026 19:50
87803f5 to
07cd9b4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #20
What
Adds a false-positive regression corpus to the test suite. New files:
tests/false-positive-corpus.test.tstests/fixtures/false-positive-corpus.tsThe test scans each corpus snippet with the full default rule set - the 15
built-in rules plus the 30 external coding-only rules loaded from
data/regex_list_1.json(vialoadExternalRulesFromJson(..., { codingOnly: true }),mirroring
tests/rules.test.ts) - and asserts zero findings. Any future rulechange that starts flagging one of these benign strings now fails CI.
Test-only PR.
src/,scripts/,hooks/, anddist/are untouched - andbecause no
src/changed,dist/did not need a rebuild.Corpus (49 benign snippets, all verified zero-finding today)
eyJsegments)Covered traps include:
api_key = os.environ["API_KEY"],apiKey: process.env.API_KEY,Authorization: Bearer <token>,token: "changeme", UUIDs and git SHAs, npmintegrity/ lockfile base64 blobs,sha256:digests,kubectl/gcloudcommand lines, Secret Manager references (
projects/x/secrets/y/versions/z) incode and diffs, and markdown authentication docs.
Self-check canaries
CANARIESholds 8 real-shaped secrets (one per built-in category family: email,JWT, Bearer, AWS key, OpenAI key, GitHub token, hardcoded assignment, PEM private
key) and asserts detection does fire - so the zero-finding assertions can't
pass vacuously against a mis-loaded or empty scanner.
Current-behavior notes discovered while building the corpus
The corpus was built empirically (each snippet scanned before inclusion). Several
realistic-but-benign strings do fire under today's rules; rather than change any
rule, they are documented in
KNOWN_FALSE_POSITIVESand kept out of the corpus,each asserted to still fire so a future refinement is intentional and visible:
AKIAIOSFODNN7EXAMPLE- Amazon's own documented example access key ID - matchesaws-api-key.api_key = "YOUR_API_KEY_HERE"- the 17-char placeholder is >=16 word chars in quotes, sogeneric-code-secret-assignmentfires (the corpus uses the shorter"YOUR_KEY"to stay clean).sk-xxxxxxxxxxxxxxxxxxxx-sk-+ 20 placeholder chars matchesopenai-api-key.secret_access_key/access_key_idin prose -external-aws-credentials-contextis a keyword-only alternation, so the term alone fires with no value present.https://login.microsoftonline.com/common/oauth2/v2.0/token- the public OAuth endpoint URL matchesexternal-microsoft-office-365-oauth-context./etc/passwdline (root:x:0:0:...) - matchesexternal-password-etc-passwd.ssh-rsa AAAA... user@hostline - matchesexternal-ssh-rsa-public.npm install && npm run build && npm testall pass (225 tests).