detection.md

How detection works

A KeyHog scan is a pipeline. Files come in one side, findings go out the other. In between, four stages:

files → [chunker] → [prefilter] → [detector match] → [post-process] → findings

Most chunks that fail the cheap prefilter stop there, which keeps full regex evaluation focused on plausible inputs. This is not an unconditional hard drop: a rejected chunk that looks encoded can enter a bounded decode-only recovery pass (recursively decoding up to a max_decode_depth, defaulting to 10), so an encoded secret is not lost merely because its plaintext anchor is absent from the original bytes.

Detection mechanisms

KeyHog does not use one universal test for "secret-like." It composes several mechanisms, and their roles are deliberately different:

Mechanism	Role	Can create a candidate?
Service-anchored detector regex	Matches a vendor or credential-specific shape from detector TOML	Yes
Companion patterns	Finds related fields near a primary match; required = true gates acceptance, while optional companions enrich confidence or verification	Confirms an existing candidate
Structured and multiline extraction	Reassembles assignments and strings that syntax splits across lines or nodes	Yes
Decode-through transforms	Scans supported encoded representations while preserving source attribution. Reverse and Caesar admission uses the active detector TOMLs' decode_transforms prefixes	Yes
Bounded static program recovery	Evaluates recognized side-effect-free JavaScript XOR, explicit-key AES-256-CBC, and CryptoJS/OpenSSL passphrase expressions when every operand is embedded and immutable	Yes
Generic assignment bridge	Extracts values beside credential-role keys when no vendor shape exists	Yes
Shannon entropy	Measures byte-distribution uncertainty for opaque generic values	Yes, on the entropy-discovery path
BPE token efficiency	Rejects language-like values that compress into common subword tokens; eligible candidates use it by default, and detector TOML can tune or disable it	No; precision gate
English bigram discriminator	Distinguishes random alphabetic tokens from pronounceable identifiers, dictionary placeholders, and low-diversity masks inside specific shape and context gates	No; admits or rejects an extracted candidate within those gates
Shape, placeholder, path, and context policy	Rejects examples, references, prose, identifiers, and context-specific noise; entropy owners compile their isolated-token floors and lengths from their detector TOML plausibility table	No; precision gates
Checksums and structural validators	Proves or rejects formats that carry intrinsic validity bits or grammar	Adjusts acceptance/confidence
On-device MoE scoring	Scores ambiguous candidates using local features; never sends content away	Adjusts confidence
Live verification	Optionally asks the owning service whether a surviving credential is active	Adds a verdict after detection

Regex, generic extraction, entropy, and decode-through therefore find different candidate classes. Named regexes and generic assignment extraction create candidates; companions, validators, BPE, English bigram evidence, shape/context policy, and confidence then confirm, reject, or score them. Verification runs only after a candidate survives detection and reporting policy.

Structured extraction accepts balanced Helm actions as render-time syntax. It replaces each action with an inert YAML value while retaining every literal source byte. A Jupyter notebook truncated at end of file is repaired only by closing its open string and container delimiters. Any other syntax error remains a counted coverage gap.

ML participation is detector-owned through [detector.ml]. lift can raise a structural score but cannot veto a match, blend combines model and structural evidence, and authoritative lets the model decide an otherwise ambiguous channel. In the shipped corpus every regex-pattern channel is currently lift; 919 of 923 entropy channels disable ML, and only the four generic entropy owners use authoritative. The model is therefore not a general regex false-positive veto. Its current feature record identifies detector and pattern-versus-entropy channel, but not the exact matched pattern. Pattern-local weak-anchor policy remains outside model conditioning until that provenance is carried end to end. The retrainer refuses mismatched detector/channel records and now requires positive and negative held-out support for every blend or authoritative channel before writing another model.

Before model inference, a cheap probabilistic gate may assign 0.1 confidence to an unaccompanied generic candidate that lacks secret-like randomness. It cannot short-circuit a service-regex match or a candidate backed by a matched companion, including a weakly anchored service pattern: those candidates continue into their detector-owned ML mode. This distinction prevents a corpus-wide randomness shortcut from overriding stronger detector-local evidence.

Static program recovery is a decode mechanism, not arbitrary code execution. KeyHog does not invoke Node.js or evaluate source. It recognizes a bounded grammar for cyclic byte-array XOR and Node-style AES-256-CBC decryption, resolves only literal numeric arrays, Base64-encoded JSON arrays, buffer literals, and empty-separator string joins, then checks binding consistency, UTF-8, AES block shape, and PKCS#7 padding before rescanning the recovered plaintext. Recovered XOR calls and Node AES ciphertext bindings are spliced back into bounded parent context, preserving assignment evidence and absolute source offsets. The CryptoJS dialect additionally requires an exact immutable require("crypto-js") alias, decrypt wrapper, literal passphrase and ciphertext bindings, an OpenSSL Salted__ envelope, and EVP_BytesToKey MD5 derivation. Dynamic values or unsupported syntax produce no derived candidate. The original source still follows the normal detector pipeline. The mechanism is disabled with decode recursion, including under --fast.

BPE is not a replacement name for entropy: it is an independent post-candidate signal. BetterLeaks calls the approach Token Efficiency; KeyHog uses the same broad BPE idea while keeping its own detector schema, thresholds, pipeline, and behavioral evidence.

Terminology matters here: BetterLeaks' public documentation names the feature Token Efficiency and describes BPE tokenization as a natural-language false positive filter; it does not present “BPD” as a separate score. KeyHog names its related mechanism BPE token efficiency, uses cl100k_base, measures UTF-8 bytes per token, and resolves the ceiling per detector. If “BPD” is being used informally to mean a bits/byte or bytes/token density, do not treat it as a third byte-density score. KeyHog also uses a separate English letter-bigram discriminator, but it does not measure bits per byte or bytes per token.

The English bigram discriminator evaluates lowercase ASCII alphabetic runs against an embedded 26 by 26 log-probability model. Digits and symbols end a run. Fewer than six alphabetic characters produce no randomness verdict. A random-token verdict requires a mean score at or below -6.85 and at least three distinct letters. Depending on the calling gate, that evidence can keep an otherwise identifier-shaped random credential or reject a confidently English placeholder. The model does not model arbitrary bytes, numeric keys, hexadecimal or Base64 alphabets as such, or short values. Its current thresholds are scanner-wide constants, not detector TOML fields.

Detector-owned tuning: what each setting changes

Detection policy belongs in the detector TOML whenever the choice is specific to a credential type. Scan-wide CLI/TOML values are operational overrides for controlled comparisons or a corpus-wide policy; they are not a second hidden detector definition. keyhog explain <detector-id> shows the policy declared by that detector TOML and its provenance; keyhog config --effective shows the resolved scan-wide policy.

Practical ownership rule: any numeric value that changes one secret family's recall, precision, shape admission, or scoring must be a named detector-TOML field. If the schema cannot express it, extend the typed schema and its explain/contract surfaces rather than adding a detector-specific literal in scanner code. Only true shared invariants, such as parser safety caps or a model's fixed vocabulary, remain global.

Global structural non-secrets are typed Tier-B data in rules/entropy-universal-rejections.toml. Plain prefixes and explicit prefix-plus-length rules, including the Ag Sealed Secrets ciphertext boundary, apply uniformly before detector plausibility. They are corpus-global structural invariants, not a second source of per-detector tuning.

Detector TOML field	If increased / enabled	If decreased / disabled
entropy_low	Requires more Shannon entropy for keyword-anchored generic values; fewer low-randomness passwords/tokens survive	Admits more values when the assignment key supplies evidence; shape, BPE, context, and confidence gates still apply
entropy_high	Tightens keyword-independent generic admission and raises the partial-confidence tier for entropy fallbacks	Admits more opaque candidates and grants the partial entropy score at a lower value
entropy_very_high	Tightens isolated, anchor-free admission and raises the full-confidence tier for entropy fallbacks	Expands the no-keyword search and grants the full entropy score at a lower value
sensitive_path_entropy_very_high	Raises the keyword-free bar even in sensitive files	Lowers the explicit sensitive-path bar for that detector, improving recall in .env/secret manifests
plausibility.keyword_free_operator_margin	Raises the detector-owned margin composed with the Tier-A entropy threshold	Lowers that explicit margin for the keyword-free role owner; no other detector may declare it
[detector.entropy_fallback]	Changes the emitted synthetic entropy finding identity and semantic class for that detector	Omitting it for an active entropy owner fails compilation; there is no scanner-global compatibility identity
entropy_roles	Claims one or more corpus entry paths: keyword-free, isolated-bare, or unclaimed-keyword	Omitting a role disables that path in a focused custom corpus; no built-in owner or threshold is substituted
decode_transforms.reverse_prefixes	Admits character-reversed candidates that can recover one of the declared plaintext prefixes	Omitting a prefix prevents reverse recovery for that prefix in a focused custom corpus
decode_transforms.caesar_prefixes	Admits only ROT-N shifts that can recover one of the declared plaintext prefixes	Omitting a prefix prevents Caesar recovery for that prefix in a focused custom corpus
[[detector.entropy_shapes]]	charset, optional grouping, diversity requirements, and a lower shape floor admit explicitly structured isolated credentials	Stricter structural requirements or a higher floor narrow the exception; omission is invalid for an active entropy owner
entropy_floor	A higher applicable length-bucket floor suppresses more low-entropy candidates for that detector	A lower floor preserves more human-chosen or structured credentials
plausibility.mixed_alnum_floor	Rejects more identifier-like alphanumeric runs	Preserves more low-randomness mixed-alphanumeric values
plausibility.symbolic_entropy_floor	Raises the minimum entropy for symbol-bearing credential assignments, including the bare auth= bridge	Preserves more anchored symbolic passwords through the same compiled detector policy
plausibility.second_half_entropy_floor	Rejects candidates with a less-random tail	Preserves more credentials whose entropy is front-loaded
plausibility.second_half_min_len	Applies the tail-entropy check to shorter values	Restricts the tail check to longer values
plausibility.unique_chars_min_len	Applies distinct-character requirements to shorter values	Restricts the diversity check to longer values
plausibility.min_unique_chars	Requires more distinct characters once the diversity check applies	Preserves lower-diversity credentials
plausibility.unanchored_hex_max_len	Allows longer unanchored all-hex values before treating them as non-secret key material	Rejects shorter unanchored all-hex values
plausibility.identical_char_max_len	Allows longer single-character repetitions	Rejects shorter single-character repetitions
plausibility.structured_dotted_min_len	Requires a longer isolated structured dotted token	Admits shorter structured dotted tokens after the other gates pass
plausibility.mixed_alnum_min_len	Requires a longer mixed alpha-numeric credential before the carve-out applies	Lets shorter anchored mixed tokens use the detector's mixed floor
plausibility.isolated_mixed_entropy_floor	Raises the floor for isolated contiguous or underscore-delimited mixed tokens	Preserves more low-randomness isolated mixed tokens
plausibility.isolated_symbolic_min_len	Requires a longer isolated symbol-rich credential for the short-candidate exception	Admits shorter symbol-rich candidates; exact declared lower-dash layouts still use their own shape rules
plausibility.isolated_symbolic_min_symbols	Requires more symbol bytes in the isolated symbolic exception	Admits candidates with fewer symbol bytes after the other gates pass
plausibility.isolated_symbolic_requires_non_underscore	Prevents underscore-only mixed tokens from bypassing their mixed entropy floor through the symbolic exception	Allows underscore to satisfy the symbolic exception by itself
plausibility.isolated_colon_left_min_len / isolated_colon_right_min_len	Requires longer sides around an isolated opaque:opaque separator	Admits shorter colon-separated opaque pairs
plausibility.leading_slash_base64_entropy_floor	Raises the floor for unanchored slash-led base64	Preserves more slash-led base64 candidates
plausibility.leading_slash_base64_min_len	Requires a longer unanchored slash-led base64 candidate	Admits shorter candidates after alphabet, padding, entropy, and shape checks pass
plausibility.reject_repeated_blocks	Rejects periodic mask values, including truncated repetitions in the bare auth= bridge	Allows that shape to continue through the remaining detector gates
plausibility.allow_alphabetic_credential	Admits anchored all-letter passwords/tokens after other gates	Requires alphabetic-only values to clear the ordinary entropy path
plausibility.reject_program_identifiers	Rejects pure source-language identifier shapes	Allows pure identifier-shaped values through the remaining gates
plausibility.reject_source_symbol_identifiers	Rejects digit-bearing mixed alphanumeric source-symbol shapes independently of the pure-identifier gate	Allows those mixed values to follow the detector's mixed_alnum_floor and mixed_alnum_min_len policy
plausibility.reject_dash_segmented_alnum	Rejects serial/product-key-like dash groups	Allows dash-segmented alphanumeric values through the remaining gates
entropy_policy_priority	Wins more overlapping generic keyword-policy claims	Yields shared keywords to a more specific detector; unique keywords are unchanged
bpe_max_bytes_per_token	A higher ceiling is looser: fewer compressible/word-like candidates are rejected	A lower ceiling is stricter: more language-like values are rejected, with corresponding recall risk
bpe_enabled = false	Not applicable	Skips token-efficiency rejection for detectors such as human-chosen passwords
decoded_hex_key_material_lengths	Adds only the declared pure-hex widths after transport decoding	Omitted widths remain decoded-digest negatives
canonical_hex_key_material	Generic detectors admit declared lengths only under exact keywords or vendor-prefixed suffixes; regex detectors use length-only entries because their matched pattern is the anchor	Omitted policy, scope, or length remains a digest-shaped negative; there is no service-wide width fallback
min_len / keyword_free_min_len	Longer values are required; short false positives fall, but short real credentials can also fall	Shorter credential shapes become eligible
max_len (entropy-policy owner)	Longer values remain eligible across generic assignment, entropy fallback, and explicit regex envelopes; increase only when the credential contract permits them	Overlength values are rejected whole with value_too_long before entropy or BPE
allowlist_paths, allowlist_values, stopwords	Adds detector-specific path, value-regex, or literal exclusions	Removing an exclusion makes that detector consider the matching path/value again; it does not affect other detectors
pattern required_literals	Routes a prefixless regex only after at least one AST-proven necessary ASCII literal is present. The declaration is the sole owner of non-prefix literal routing	Omitting it leaves the regex in its prefix, keyword-gated, or always-active route; an unsound declaration rejects the detector
public_identifier_assignment_markers	Classifies detector-local assignment-key fragments as public identifiers instead of credentials	Omission disables this suppression for that detector; there is no scanner-global blockchain/network marker list
min_confidence	Raises this detector's reporting floor	Lowers this detector's reporting floor; an operator override can still replace it
detector/pattern weak_anchor	Keeps generic shape/entropy gates active for a whole detector or an individual pattern; requires the owning detector's entropy_high and entropy_floor	Trusts unmarked patterns; use only when those patterns prove the credential shape
structural_password_slot	Applies password-slot placeholder policy to a free-form value captured from a syntactic credential slot	Leaves that detector outside the structural-password family
private_key_block	Makes the detector's enclosing key block suppress less-specific findings nested inside it	Treats the match as an ordinary, non-enclosing finding
[detector.credential_shape]	Declares exact prefix/length/shape constraints that a captured credential must satisfy	Omitting it leaves that detector without an additional credential-shape constraint

Resolution rules

These settings do not all use one generic “last value wins” rule:

• Generic keyword ownership: the highest entropy_policy_priority among detectors claiming the normalized assignment keyword owns entropy and BPE policy. Equal priorities use stable detector identity, independent of corpus order. Custom detector policy keywords join entropy discovery directly; they do not need to be repeated in [scan].secret_keywords.
• Final match resolution: the active compiled plan classifies named, phase-2 generic, entropy, and enclosing private-key findings. The reporting service string and detector-ID length do not change specificity. Unknown finding identities fail checked resolution instead of inheriting embedded or service-name behavior.
• Entropy entry roles: entropy_roles selects the detector that owns each corpus-level entry path. A compiled corpus may have at most one owner for each role. Missing roles remain disabled, and duplicate owners fail scanner construction. Role selection never depends on a detector ID spelling.
• Weak anchors: detector-level weak_anchor = true applies to every pattern, while the same field inside [[detector.patterns]] governs that exact regex. Each such detector owns entropy_high and length-bucketed entropy_floor. Scanner construction rejects an explicit weak anchor without that local policy. KeyHog never guesses this semantic choice from regex text, and min_confidence does not disable it. The compiled hot path uses a primitive detector-indexed floor program.
• BPE ceiling: every active entropy owner declares either bpe_max_bytes_per_token or bpe_enabled = false; omission fails detector validation and scanner construction. An explicitly supplied [scan].entropy_bpe_max_bytes_per_token or --entropy-bpe-max-bytes-per-token replaces every BPE-enabled entropy/generic detector ceiling; the CLI wins over the config file. bpe_enabled = false still disables the gate for that detector.
• Confidence floor: the scan floor defaults to 0.40. A detector TOML min_confidence replaces the scan floor for that detector, and an operator [detector.<id>].min_confidence replaces the detector-declared floor. Under --precision, the resolved global and per-detector floors are clamped to at least 0.85; neither source can weaken the precision preset.
• Entropy policy: every active entropy owner must declare its high, low, very-high, sensitive-path, mixed-alphanumeric, symbolic, tail-entropy, length, isolated-shape, and BPE policy. Scanner construction compiles these into concrete detector-indexed values; a missing field is an error, not a runtime default. Schema defaults remain only for non-owning programmatic detector values that never supply entropy policy. The scan-wide entropy_threshold is deliberately not a blanket replacement for all four bands. On the phase-2 generic bridge it tightens only when it exceeds the owning detector's high band. On the entropy scanner, a value above that high band tightens keyword and isolated candidates; a value below the keyword detector's low band loosens that keyword path, while values between the low and high bands leave its low floor in place. The isolated path keeps its mixed-alphanumeric floor unless the scan threshold exceeds the high band. Named-detector heuristic confidence uses the resolved scan threshold as its partial entropy tier and the scoring margin above it as its full tier; changing the setting can therefore change a named finding's confidence without changing whether its regex matched. These rules preserve the different evidence carried by an assignment key, an isolated opaque token, and an unanchored generic value. The owning entropy_high and entropy_very_high values also define the partial and full heuristic-confidence tiers for emitted entropy fallbacks. Detector ML policy then composes with that heuristic; an authoritative ML mode may replace it, while disabled, lift, and blend modes retain the documented heuristic semantics.
• Sensitive paths: sensitive_path_entropy_very_high is a required detector-local threshold for active entropy owners. Equaling entropy_very_high means no sensitive-path relaxation; a lower declared value is an explicit detector-owned recall choice.
• Credential plausibility: the required detector plausibility block owns its entropy floors, length carve-out, alphabetic admission, and repeated, identifier, and dash-segment rejection choices. There is no production-path fallback for an active entropy owner.
• Synthetic entropy identity: every active entropy owner declares [detector.entropy_fallback] with a semantic class (generic, password, token, or api-key), an entropy-* id, display name, and service. The compiled scanner uses the complete metadata from the active detector corpus for entropy-only findings. Omitting the block is a visible compile error; no scanner-global keyword classifier or compatibility identity can relabel a custom candidate.
• Isolated entropy shapes: generic entropy owners declare one data-driven shape with its character set, entropy floor, optional fixed-width grouping, diversity requirements, and special minimum length. For the shipped lower-alphanumeric app-password policy, candidate length is derived from four groups of four plus three separators; special_min_length controls the short-candidate revisit and must not exceed that derived length. The shape is used for anchorless synthetic entropy recovery; the anchored bluesky-app-password regex remains the source of the named Bluesky finding. A custom corpus without the shape has no isolated exception, rather than inheriting an embedded detector policy.
• Isolated symbolic credentials: the detector's plausibility.isolated_symbolic_min_len, plausibility.isolated_symbolic_min_symbols, and plausibility.isolated_symbolic_requires_non_underscore fields control the shorter symbol-rich exception. Contiguous and underscore-delimited mixed tokens stay under plausibility.isolated_mixed_entropy_floor when the owner requires a non-underscore symbol, and an exact declared lower-dash layout must satisfy its entropy_shapes policy instead of bypassing it as symbolic.

Token efficiency can carry more of the precision burden for a detector whose assignment key or regex already creates the candidate. That is the practical per-detector alternative to making Shannon entropy the decisive signal: use a permissive detector-owned entropy floor appropriate to the credential family, then let its BPE, shape, context, and confidence policy reject word-like noise. It is not equivalent to blindly replacing entropy with one global BPE number, and bpe_enabled alone never creates a candidate. Both configured gates still execute; the current pipeline has no entropy-or-BPE branch.

Detector-owned canonical_hex_key_material is the deliberate exception to the BPE and generic low-diversity/decode-as-data gates. Hexadecimal key bytes tokenize efficiently and use a small alphabet for the same mechanical reasons hexadecimal digests do, so the exact detector-owned contract supplies the discriminator: assignment scope plus length for generic detectors, or matched regex plus length for named detectors. Placeholder, degenerate-repeat, entropy, context, and reporting gates remain active. When ML is enabled, this exact TOML match is structural positive evidence and therefore preserves the detector heuristic floor; the model may raise its score but cannot erase a policy-proven key as if it were an unowned entropy candidate.

Scan-wide settings remain operational controls, but they do not all compose the same way. The operator-layer order and working TOML/CLI examples are in Configuration. Stable per-detector tuning belongs in the owning detector TOML and should be proved with that detector's positive, negative, evasion, backend-parity, and corpus contracts.

Settings, active corpus, and exact identity

KeyHog keeps detector content, resolved scan policy, and corpus provenance separate:

• The reported detector corpus digest binds the normalized corpus schema and the active detector specifications after composition and [detector.<id>] enabled = false removal. A matching disable therefore changes the digest. An unknown disabled ID warns and leaves this digest unchanged.
• The autoroute rules identity also describes the active detector specifications. Operator confidence-floor overrides are composed later so different scan presets can coexist in one calibration cache.
• The autoroute configuration identity binds the resolved scanner and operator policy. It includes the selected fast, deep, or precision preset, scan-wide and per-detector floors, the configured disabled-ID set, detector tuning inputs, worker and pipeline settings, backend/GPU policy, and profiling instrumentation.
• The corpus path and embedded, replace, or overlay label are provenance. They are reported in versioned output, but the path spelling is not detector content. Copying the same normalized corpus to another directory does not create a different content digest. A mode change changes the digest only when it changes the resulting active specifications.

The preset definitions and their override rules are in Configuration. --profile is performance instrumentation, not a named scan-policy profile.

Hardware changes execution, not detection policy. CPU, SIMD/Hyperscan, and GPU routes consume the same resolved detector and configuration identities. Autoroute accepts a candidate only when its canonical detection identities match the reference: chunk membership, detector id/name/service/severity, exact credential, stored hash, companion identity, source, file, line, byte offset, commit, author, date, entropy, confidence, and multiplicity. Mismatch diagnostics name only the differing fields and occurrence counts. They never expose raw values or deterministic value fingerprints.

Built-in suppression, confidence, decode, and scanner postprocessing are already part of those backend results. CLI allowlists and rules, policy floors, cross-source deduplication, verification, and output formatting run after selection. Missing or stale exact evidence is an error. Calibration never relaxes a detector to make a backend look faster.

Change	Finding-set effect	Routing/calibration effect
Copy the same normalized corpus to another path	None	Content identity is unchanged; reported source provenance changes
Change detector TOML, corpus schema, replacement/overlay membership, or a matching enabled = false override	Candidates, suppressions, confidence, or final findings may change	Active corpus/rules identity changes; recalibration is required
Change a preset, scan-wide policy, per-detector floor, configured disabled-ID set, workers, or GPU/runtime policy	Results or scan cost may change according to the setting	Configuration identity changes; calibration for the old identity is not reused
Change CPU, GPU, driver, or accelerator availability	None for the same resolved detector/configuration and input; a parity mismatch rejects that route	Host/device/runtime identity changes; old host evidence is not reusable
Use `--backend cpu	simd	gpu-cuda
Change input size, chunk count, source execution class, decoder-kind mask, decode candidate count or byte bucket, decoder uncertainty, or full-source-size availability	The input can change findings; backend choice must not	A different exact workload key is selected
Switch between a one-shot process and a ready daemon/watch runtime	Runtime lifetime must not change detector policy or canonical matches	Cold-aware and warm persistent-runtime routes may have different winners

Strict Backend Parity

KeyHog exposes three search-backend classes: pure Rust CPU, SIMD/Hyperscan (simd-regex), and GPU/VYRE region presence. Autoroute measures five concrete runtime peers when eligible: scalar CPU, Hyperscan CPU, CUDA, native Metal, and WGPU. Portable builds retain the pure-Rust trigger path without Hyperscan. keyhog calibrate-autoroute rejects any peer whose canonical match identity differs from the reference. It records the first real GPU dispatch plus warm trials: an ordinary process resolves against the cold-aware GPU cost, while a daemon that initialized its engines before readiness resolves against the warm GPU evidence. A missing or invalid decision is not autoroute evidence: KeyHog warns, uses the scalar correctness oracle to complete the scan, and records the exact recovered byte coverage and repair command.

When comparing settings, record the effective config, detector digest, input identity, backend, host/accelerator identity, and complete findings, not only elapsed time or finding count. A faster run with a different result set is a detection change or parity failure, not a routing win.

Stage 1 - chunker

A file becomes one or more chunks. A chunk is {data: str, metadata: {source_type, path, line_offsets, …}}. The chunker:

• Skips obvious binaries via magic-byte sniffing (PDF, PNG, zip, …).
• Skips files matching is_default_excluded_path (node_modules, .min.js, build/, etc.).
• Splits files larger than the 1 MiB window size into overlapping ~1 MiB windows so a single giant log file doesn't blow scratch memory. Each window carries its absolute base byte offset and base line so findings report the real file offset/line, not the per-window one. Cross-window secrets are reassembled in stage 4.
• Decodes UTF-16 BOM files into UTF-8 (PowerShell / .NET configs).

Specialized chunkers run too:

• Git history → one chunk per (commit × file × diff line)
• Docker images → one chunk per layer × file
• Web URLs → one chunk per response body / sourcemap / WASM strings
• S3 buckets → one chunk per object body
• GCS buckets → one chunk per object body
• Azure Blob containers → one chunk per blob body

Stage 2 - prefilter (the cheap pass)

Three gates, in order, each cheaper than the next:

1. Alphabet screen. A 256-bit mask of which bytes the corpus's detectors care about. A chunk with no relevant byte becomes a prefilter miss.

2. Bigram bloom. A 4096-bit bloom filter of 2-byte sequences from detector keyword prefixes. A chunk with no overlapping bigram becomes a prefilter miss. This cheaply recognizes source that carries no relevant anchor vocabulary.

After these screens, ordinary misses stop. Decode-shaped misses instead take the bounded decode-only path described above; transformed plaintext is then attributed back to the original source.

3. Backend trigger pass. The simd-regex backend compiles the detector corpus into Hyperscan databases when the simd feature is present; cpu-fallback uses the pure-Rust trigger path. One pass returns which detector IDs have a candidate match.

   GPU-capable builds add VYRE's resident fused literal-evidence backend. Its single dispatch returns region presence plus detector-derived localization positions; the shared host regexes still decide every finding. There is no universal model-name or byte threshold at which KeyHog silently switches to it. --backend auto requires an exact persisted calibration decision for the current binary, detector/config digest, host/device/driver, workload class, and size bucket. Calibration keeps a GPU route only when its canonical match identities equal the reference and it is the fastest eligible backend for that key.

Stage 3 - detector match

For each pattern-backed detector that the prefilter flagged, the full regex evaluates. The regex is detector.patterns[].regex in that detector's TOML, and its configured capture group becomes the candidate credential. Generic phase-2 detector TOMLs use keyword, length, entropy, token-efficiency, and shape policy for shapeless assignments or isolated opaque values. They may also carry explicit patterns for strongly structured envelopes such as JSON "secret", "token", or "apiKey" fields; both mechanisms remain owned by the same detector TOML instead of a central compatibility detector.

A detector's .toml carries:

• id, name, service, severity, keywords
• zero or more patterns, each with regex + group + optional description (required for service-anchored detectors; optional structured-envelope anchors for phase2-generic)
• optional typed companions; required entries gate acceptance, reinforcing entries add evidence, and forbidden entries suppress
• optional bounded detector_relations with requires, conflicts, or subsumes semantics across findings in the same source, file, and revision
• optional verify block: HTTP method, URL template, auth scheme, success status

Detectors fall into two camps:

• Service-anchored. Regex requires a service-specific keyword (AWS_SECRET_ACCESS_KEY=, stripe.com/v1/, dn_ Deepnote prefix). These have HIGH precision: the keyword itself is positive evidence, not just a hint.

• Generic / entropy discovery (generic-password, entropy-api-key, entropy-token). Triggered by entropy + assignment shape only - password = "...", secret: "...", JSON { "token": "..." }. Lower precision; suppression filters do most of the work.

  Surviving candidates also pass a BPE token-efficiency gate. Shannon entropy asks how evenly bytes are distributed; token efficiency asks how readily a fixed subword vocabulary compresses the value. Dotted API names and prose can have high Shannon entropy but tokenize into a few common pieces, while opaque secrets usually require many short tokens. The mechanisms are complementary, and generic detector TOMLs may own their token-efficiency ceiling through bpe_max_bytes_per_token. Opaque API-key/secret policies use their detector-owned ceiling, falling back to the scan-wide default of 2.2 UTF-8 bytes per token when they do not declare one; password/passphrase policies set bpe_enabled = false because human-chosen credentials may intentionally be word-like. Disabled policies skip tokenizer work entirely rather than using a magic oversized ceiling.

The entropy-generic, entropy-password, entropy-token, and entropy-api-key IDs are output classifications for entropy-discovered findings, not four additional detector TOML files. Their candidate policy is owned by the corresponding phase-2 TOMLs selected from the assignment context: generic-secret, generic-password, generic-keyword-secret, or generic-api-key. Use keyhog explain on those owning detector IDs when tuning entropy, BPE, length, or canonical-key policy.

The split matters for the post-process stage.

Stage 4 - post-process

Even a regex match isn't always a credential. Stage 4 filters:

• Known example fixtures (Stripe docs key, AWS docs key, RFC 7519 specimen JWT).
• Placeholder language - credentials containing YOUR_, INSERT, EXAMPLE, PLACEHOLDER, TODO, FIXME, etc.
• Shape gates.
  • Universal: punctuation_decorated_identifier - credentials starting with --, &, @, !, /, $ (CLI flags, pointers, SQL vars, shell vars, GraphQL refs) or ending in : / ! (UI labels, TypeScript non-null assertions).
  • Generic / entropy only: pure_identifier, word_separated_identifier, scheme_prefixed_uri, url_or_path_segment, contains_uuid_v4_substring. These shapes can be real credentials when paired with a service or protocol anchor, so named detector TOMLs and structural authorization detectors own those cases. A generic token=<uuid> remains an identifier; an Authorization: Bearer <uuid> value is a credential because the Bearer envelope supplies the missing evidence. Public salts and nonces are not generic secrets. A detector for a product whose field is genuinely secret despite that name must own the product syntax explicitly.
• Path-based suppressions - vendored bundles (node_modules/, wp-includes/, bower_components/), CI workflow files (where ${{ secrets.NAME }} references are syntactic, not credentials), i18n translation files, secret-scanner source files (the file IS a scanner; its regex literals shouldn't fire on itself).
• Cross-chunk reassembly. A secret split across window boundaries gets reassembled from the tail of chunk N + the head of chunk N+1.

A finding that survives stage 4 makes it to output.

Where the speed comes from

The alphabet screen and bigram bloom reject irrelevant chunks before regex confirmation. Literal triggers narrow the active detector set, and the scanner shares confirmation, suppression, and reporting tails across CPU and GPU backends. Windowing bounds scratch space for large inputs; caches avoid repeated compiler and index work.

End-to-end throughput depends on the detector/config digest, source shape, candidate density, decoding and verification policy, cache state, CPU, GPU, driver, and storage. Use keyhog calibrate-autoroute for routing evidence on the installed host and the repository benchmark harness for reproducible cross-version measurements; do not treat a throughput number from another machine or detector corpus as a routing threshold.

Where the precision comes from

Filter	What it catches
Known example fixtures	Stripe docs key, AWS docs key, RFC 7519 JWT
pure_identifier	getParameter, Benutzername, auth_decoders
word_separated_identifier	s3_secret_access_key (function name)
scheme_prefixed_uri	urn:foo:bar (URI literal, not creds)
url_or_path_segment	/api/v1/users/123 (REST path)
contains_uuid_v4_substring	TOKEN_LIST=636765a9-… (UUID identifier)
punctuation_decorated_identifier	--api-secret, &password, Password:
Vendored-minified-path	node_modules/jquery-3.6.0.min.js
CI workflow path	.github/workflows/ci.yml - ${{ secrets.X }}
i18n translation path	locale/de.po - translated password word

Each filter has a known-FP-cluster it was built to defuse. The Suppressions page enumerates them with examples.

What this looks like for one finding

file.env contains: AWS_SECRET_ACCESS_KEY=ev0BsFtSD7S/4VWYObxiEhME3hJBXeYzR43jgiB1

stage 1 - chunker:        emit chunk{ path: "file.env", data: "AWS_SECRET..." }
stage 2 - alphabet:       PASS (chunk has `=`, alphanumerics from the corpus)
stage 2 - bigram bloom:   PASS (`AW`, `WS`, `_S` are in the bloom)
stage 2 - simd-regex:     MATCH → triggers `aws-secret-access-key` + `generic-password`
stage 3 - regex eval:
  `aws-secret-access-key` detector pattern captures the 40-byte value
    captures `ev0BsFtSD7S/4VWYObxiEhME3hJBXeYzR43jgiB1`
  `generic-password` regex doesn't match (no `_password`/`_pwd` substring)
stage 4 - post-process:
  known-example check: no
  `looks_like_pure_identifier`: false (has digits + /)
  `looks_like_punctuation_decorated_identifier`: false
  → EMIT

That's one finding's life. Multiply by 10⁶ files and the throughput math is why each stage matters.