feat: add Slack, GitLab, npm, Twilio, SendGrid, GCP service-account, and DB connection-string detectors#35
Open
jesse-quinn wants to merge 1 commit into
Conversation
…tion-string detectors Expand BUILTIN_RULES with nine detectors requested in datumbrain#13, each with a verified token format, positive and negative tests, and a rebuilt dist/: - slack-token (xoxb/xoxp/xoxa/xoxr and xapp), high/secret - gitlab-personal-access-token (glpat-), high/secret - gitlab-ci-job-token (glcbt-, GA since GitLab 16.9), high/secret - npm-access-token (npm_), high/secret - twilio-api-key-sid (SK + 32 hex), high/secret - twilio-account-sid (AC + 32 hex, public identifier), low/internal-data, partial-mask - sendgrid-api-key (SG.<22>.<43>), critical/secret - gcp-service-account-key (JSON envelope: type service_account + private_key), critical/secret - database-connection-string-credentials (scheme://user:password@host, password required), critical/secret, partial-mask Patterns are ReDoS-safe (no nested quantifiers; bounded lazy spans) and conservative to avoid firing on prose, placeholders, and credential-less URIs.
jesse-quinn
force-pushed
the
feat/expand-builtin-detectors
branch
from
July 19, 2026 03:16
d70a180 to
49edfca
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #13.
Expands
BUILTIN_RULESinsrc/scanner/detectors.tswith nine new detectors covering the provider gaps called out in the issue. Every rule ships positive and negative tests intests/rules.test.ts, the patterns are ReDoS-safe (no nested quantifiers over variable input; the one multiline rule uses a bounded lazy span), anddist/is rebuilt and committed alongsidesrc/.Per the CONTRIBUTING "Adding a detection rule" guidance, each token format below was verified against current public documentation (not memory), cross-checked against secret-scanning references (gitleaks / GitGuardian / Trivy) for exact charset and length.
Rules added
slack-tokenxoxb-/xoxp-), legacy workspace/refresh (xoxa-/xoxr-), app-level (xapp-) — docs.slack.dev/authentication/tokens; segment structure from gitleaksgitlab-personal-access-tokenglpat-+ 20+[A-Za-z0-9_-]— docs.gitlab.com/security/tokensgitlab-ci-job-tokenglcbt-(CI/CD job token, GA since GitLab 16.9) — docs.gitlab.com/security/tokensnpm-access-tokennpm_+ 36 base62 — docs.npmjs.com/about-access-tokens, npm token format announcementtwilio-api-key-sidSK+ 32 hex (API Key SID, pairs with a secret) — twilio.com/docs/glossary/what-is-a-sidtwilio-account-sidAC+ 32 hex — a public identifier, not a secret alone; low severity,partial-mask— same Twilio SID docsendgrid-api-keySG.<22>.<43>([A-Za-z0-9_-]) — GitGuardian SendGrid detector + gitleaks (SendGrid's own docs treat the key as opaque; the 22/43 segment lengths are the scanner consensus)gcp-service-account-key"type": "service_account"co-occurring with a"private_key"field — cloud.google.com IAM keysdatabase-connection-string-credentialsscheme://user:password@hostfor postgres(ql)/mysql/mongodb(+srv)/redis(s)/amqp(s) — libpq URIs, MySQL, MongoDB, RabbitMQ URI spec, RFC 3986 §2.1/§3.2.1Design notes
AC…) is an identifier, not a secret — classifiedlow/internal-datawithpartial-mask(per the issue's guidance). The API Key SID (SK…) ishigh/secret. The Auth Token / API Key Secret have no prefix and are not reliably detectable standalone, so they are intentionally out of scope."type": "service_account"and a"private_key"field within a bounded window (either field order), so it fires on the JSON envelope even when the PEM value is elided ("private_key": "REDACTED"), yet stays conservative — prose about service accounts (a service_account has a private_key field) does not match.:password@userinfo.@and/are illegal unencoded in RFC 3986 userinfo, so the class boundaries are reliable and the rule never fires on credential-less URIs (postgres://localhost:5432/mydb,postgres://user@host) or docs URLs. Redacted withpartial-maskso the scheme/host tail survives while the credentials are masked.Overlap with existing rules (does not fix engine behavior — see #4)
database-connection-string-credentialsoverlaps with the existingemail-addressrule: theuser:password@hosttail also matches the email pattern, so both fire on the same input. Both findings redact; the overlap is benign and left to the engine's overlap handling tracked in Overlapping matches corrupt redacted text and double-count findings #4.gcp-service-account-keyoverlaps withprivate-keywhen the PEM is present —private-keycatches the embedded key, the new rule catches the JSON envelope. Both arecritical; overlap redaction is Overlapping matches corrupt redacted text and double-count findings #4's domain.generic-code-secret-assignment, which requires a quoted value — the unquotedNPM_TOKEN=npm_…/GITLAB_TOKEN=glpat-…forms are covered only by the new rules.data/regex_list_1.jsoninclude aslack_api_tokenand apgSQL Connection Informationpattern that can overlap withslack-token/ the DB rule when external rules are enabled; also governed by Overlapping matches corrupt redacted text and double-count findings #4.Deliberately left for a follow-up
The issue's final bullet —
.env-style unquoted assignments (PASSWORD=…,DB_PASS=…) — is not included here. It is a change to the existinggeneric-code-secret-assignmentrule (which currently requires quotes) rather than a new self-contained provider detector, and broadening it interacts with the overlap behavior in #4. Hence "Addresses #13" rather than "Fixes #13".Test evidence
npm install && npm run build && npm test: 243 tests pass (up from 159 onmain) — +84 cases across the 9 new rules, each with realistic positive matches and negative cases (placeholders likeglpat-xxxx/SG.xxx.yyy, prose mentioning the provider, credential-less URIs, docs URLs).npm run lintclean;dist/rebuilt and committed withsrc/.scripts/prompt-guard.jsblocks a prompt containing a DB connection string + SendGrid key.Note: the example fixtures are synthetic (all-zero /
EXAMPLEplaceholders). Two of them (Slack, SendGrid) are assembled at runtime viajoinso the full string never appears as a single literal in source — this only avoids upstream secret-scanner false positives on the fixtures; the runtimeexamplesvalues and tests are unchanged in behavior.