Skip to content

test: add false-positive regression corpus#34

Merged
fahadsiddiqui merged 1 commit into
datumbrain:mainfrom
jesse-quinn:test/false-positive-corpus
Jul 21, 2026
Merged

test: add false-positive regression corpus#34
fahadsiddiqui merged 1 commit into
datumbrain:mainfrom
jesse-quinn:test/false-positive-corpus

Conversation

@jesse-quinn

@jesse-quinn jesse-quinn commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Fixes #20

What

Adds a false-positive regression corpus to the test suite. New files:

  • tests/false-positive-corpus.test.ts
  • tests/fixtures/false-positive-corpus.ts

The test scans each corpus snippet with the full default rule set - the 15
built-in rules plus the 30 external coding-only rules loaded from
data/regex_list_1.json (via loadExternalRulesFromJson(..., { codingOnly: true }),
mirroring tests/rules.test.ts) - and asserts zero findings. Any future rule
change that starts flagging one of these benign strings now fails CI.

Test-only PR. src/, scripts/, hooks/, and dist/ are untouched - and
because no src/ changed, dist/ did not need a rebuild.

Corpus (49 benign snippets, all verified zero-finding today)

Category Count
Prose mentioning secret terms (CLI / docs) 6
Environment-variable references (no literal secret) 9
Placeholder / example values 7
JWT-lookalikes (not three dot-separated eyJ segments) 4
Hashes, UUIDs, git SHAs, lockfile blobs 11
CLI / infra commands and secret-manager references 11
Markdown auth docs 1

Covered traps include: api_key = os.environ["API_KEY"], apiKey: process.env.API_KEY,
Authorization: Bearer <token>, token: "changeme", UUIDs and git SHAs, npm
integrity / lockfile base64 blobs, sha256: digests, kubectl / gcloud
command lines, Secret Manager references (projects/x/secrets/y/versions/z) in
code and diffs, and markdown authentication docs.

Self-check canaries

CANARIES holds 8 real-shaped secrets (one per built-in category family: email,
JWT, Bearer, AWS key, OpenAI key, GitHub token, hardcoded assignment, PEM private
key) and asserts detection does fire - so the zero-finding assertions can't
pass vacuously against a mis-loaded or empty scanner.

Current-behavior notes discovered while building the corpus

The corpus was built empirically (each snippet scanned before inclusion). Several
realistic-but-benign strings do fire under today's rules; rather than change any
rule, they are documented in KNOWN_FALSE_POSITIVES and kept out of the corpus,
each asserted to still fire so a future refinement is intentional and visible:

  • AKIAIOSFODNN7EXAMPLE - Amazon's own documented example access key ID - matches aws-api-key.
  • api_key = "YOUR_API_KEY_HERE" - the 17-char placeholder is >=16 word chars in quotes, so generic-code-secret-assignment fires (the corpus uses the shorter "YOUR_KEY" to stay clean).
  • sk-xxxxxxxxxxxxxxxxxxxx - sk- + 20 placeholder chars matches openai-api-key.
  • Bare secret_access_key / access_key_id in prose - external-aws-credentials-context is a keyword-only alternation, so the term alone fires with no value present.
  • https://login.microsoftonline.com/common/oauth2/v2.0/token - the public OAuth endpoint URL matches external-microsoft-office-365-oauth-context.
  • A standard /etc/passwd line (root:x:0:0:...) - matches external-password-etc-passwd.
  • A public ssh-rsa AAAA... user@host line - matches external-ssh-rsa-public.

npm install && npm run build && npm test all pass (225 tests).

@jesse-quinn
jesse-quinn force-pushed the test/false-positive-corpus branch from 669f13c to 87803f5 Compare July 19, 2026 03:16
Add tests/false-positive-corpus.test.ts and a fixtures module covering
realistic benign snippets that must produce zero findings against the full
default rule set (15 built-in rules plus 30 external coding-only rules).

The corpus was built empirically: every snippet was scanned before inclusion
and only genuinely zero-finding strings are kept. Realistic near-misses that
DO fire under the current rules are documented as KNOWN_FALSE_POSITIVES and
kept out of the corpus rather than changing any rule - this pins today's
behaviour, it does not fix rule bugs. A set of true-positive canaries (one per
built-in category) asserts detection still fires so the corpus self-checks that
the scanner was loaded correctly.

Test-only change; no src/, scripts/, hooks/, or dist/ touched.
@fahadsiddiqui
fahadsiddiqui force-pushed the test/false-positive-corpus branch from 87803f5 to 07cd9b4 Compare July 21, 2026 19:50
@fahadsiddiqui
fahadsiddiqui merged commit 31081cf into datumbrain:main Jul 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

False-positive regression corpus in tests

2 participants