Skip to content

feat: add Slack, GitLab, npm, Twilio, SendGrid, GCP service-account, and DB connection-string detectors#35

Open
jesse-quinn wants to merge 1 commit into
datumbrain:mainfrom
jesse-quinn:feat/expand-builtin-detectors
Open

feat: add Slack, GitLab, npm, Twilio, SendGrid, GCP service-account, and DB connection-string detectors#35
jesse-quinn wants to merge 1 commit into
datumbrain:mainfrom
jesse-quinn:feat/expand-builtin-detectors

Conversation

@jesse-quinn

@jesse-quinn jesse-quinn commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Addresses #13.

Expands BUILTIN_RULES in src/scanner/detectors.ts with nine new detectors covering the provider gaps called out in the issue. Every rule ships positive and negative tests in tests/rules.test.ts, the patterns are ReDoS-safe (no nested quantifiers over variable input; the one multiline rule uses a bounded lazy span), and dist/ is rebuilt and committed alongside src/.

Per the CONTRIBUTING "Adding a detection rule" guidance, each token format below was verified against current public documentation (not memory), cross-checked against secret-scanning references (gitleaks / GitGuardian / Trivy) for exact charset and length.

Rules added

id severity category format verified against
slack-token high secret Slack bot/user (xoxb-/xoxp-), legacy workspace/refresh (xoxa-/xoxr-), app-level (xapp-) — docs.slack.dev/authentication/tokens; segment structure from gitleaks
gitlab-personal-access-token high secret glpat- + 20+ [A-Za-z0-9_-]docs.gitlab.com/security/tokens
gitlab-ci-job-token high secret glcbt- (CI/CD job token, GA since GitLab 16.9) — docs.gitlab.com/security/tokens
npm-access-token high secret npm_ + 36 base62 — docs.npmjs.com/about-access-tokens, npm token format announcement
twilio-api-key-sid high secret SK + 32 hex (API Key SID, pairs with a secret) — twilio.com/docs/glossary/what-is-a-sid
twilio-account-sid low internal-data AC + 32 hex — a public identifier, not a secret alone; low severity, partial-mask — same Twilio SID doc
sendgrid-api-key critical secret SG.<22>.<43> ([A-Za-z0-9_-]) — GitGuardian SendGrid detector + gitleaks (SendGrid's own docs treat the key as opaque; the 22/43 segment lengths are the scanner consensus)
gcp-service-account-key critical secret JSON envelope: "type": "service_account" co-occurring with a "private_key" field — cloud.google.com IAM keys
database-connection-string-credentials critical secret scheme://user:password@host for postgres(ql)/mysql/mongodb(+srv)/redis(s)/amqp(s) — libpq URIs, MySQL, MongoDB, RabbitMQ URI spec, RFC 3986 §2.1/§3.2.1

Design notes

  • Twilio Account SID (AC…) is an identifier, not a secret — classified low / internal-data with partial-mask (per the issue's guidance). The API Key SID (SK…) is high/secret. The Auth Token / API Key Secret have no prefix and are not reliably detectable standalone, so they are intentionally out of scope.
  • GCP service-account key requires both "type": "service_account" and a "private_key" field within a bounded window (either field order), so it fires on the JSON envelope even when the PEM value is elided ("private_key": "REDACTED"), yet stays conservative — prose about service accounts (a service_account has a private_key field) does not match.
  • Database connection strings require the :password@ userinfo. @ and / are illegal unencoded in RFC 3986 userinfo, so the class boundaries are reliable and the rule never fires on credential-less URIs (postgres://localhost:5432/mydb, postgres://user@host) or docs URLs. Redacted with partial-mask so the scheme/host tail survives while the credentials are masked.

Overlap with existing rules (does not fix engine behavior — see #4)

  • database-connection-string-credentials overlaps with the existing email-address rule: the user:password@host tail also matches the email pattern, so both fire on the same input. Both findings redact; the overlap is benign and left to the engine's overlap handling tracked in Overlapping matches corrupt redacted text and double-count findings #4.
  • gcp-service-account-key overlaps with private-key when the PEM is present — private-key catches the embedded key, the new rule catches the JSON envelope. Both are critical; overlap redaction is Overlapping matches corrupt redacted text and double-count findings #4's domain.
  • The new provider rules do not double-fire with generic-code-secret-assignment, which requires a quoted value — the unquoted NPM_TOKEN=npm_… / GITLAB_TOKEN=glpat-… forms are covered only by the new rules.
  • The opt-in external JSON rules in data/regex_list_1.json include a slack_api_token and a pgSQL Connection Information pattern that can overlap with slack-token / the DB rule when external rules are enabled; also governed by Overlapping matches corrupt redacted text and double-count findings #4.

Deliberately left for a follow-up

The issue's final bullet — .env-style unquoted assignments (PASSWORD=…, DB_PASS=…) — is not included here. It is a change to the existing generic-code-secret-assignment rule (which currently requires quotes) rather than a new self-contained provider detector, and broadening it interacts with the overlap behavior in #4. Hence "Addresses #13" rather than "Fixes #13".

Test evidence

  • npm install && npm run build && npm test: 243 tests pass (up from 159 on main) — +84 cases across the 9 new rules, each with realistic positive matches and negative cases (placeholders like glpat-xxxx / SG.xxx.yyy, prose mentioning the provider, credential-less URIs, docs URLs).
  • npm run lint clean; dist/ rebuilt and committed with src/.
  • Smoke test via scripts/prompt-guard.js blocks a prompt containing a DB connection string + SendGrid key.

Note: the example fixtures are synthetic (all-zero / EXAMPLE placeholders). Two of them (Slack, SendGrid) are assembled at runtime via join so the full string never appears as a single literal in source — this only avoids upstream secret-scanner false positives on the fixtures; the runtime examples values and tests are unchanged in behavior.

…tion-string detectors

Expand BUILTIN_RULES with nine detectors requested in datumbrain#13, each with a
verified token format, positive and negative tests, and a rebuilt dist/:

- slack-token (xoxb/xoxp/xoxa/xoxr and xapp), high/secret
- gitlab-personal-access-token (glpat-), high/secret
- gitlab-ci-job-token (glcbt-, GA since GitLab 16.9), high/secret
- npm-access-token (npm_), high/secret
- twilio-api-key-sid (SK + 32 hex), high/secret
- twilio-account-sid (AC + 32 hex, public identifier), low/internal-data, partial-mask
- sendgrid-api-key (SG.<22>.<43>), critical/secret
- gcp-service-account-key (JSON envelope: type service_account + private_key), critical/secret
- database-connection-string-credentials (scheme://user:password@host, password required), critical/secret, partial-mask

Patterns are ReDoS-safe (no nested quantifiers; bounded lazy spans) and
conservative to avoid firing on prose, placeholders, and credential-less URIs.
@jesse-quinn
jesse-quinn force-pushed the feat/expand-builtin-detectors branch from d70a180 to 49edfca Compare July 19, 2026 03:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expand built-in detectors: Slack, GitLab, npm, Twilio, SendGrid, GCP service accounts, DB connection strings

1 participant