Engine scores Synapse challenge tier (JS PoW / CAPTCHA, HTTP 200) as a miss — fake-negative in battery report

[NickCrew]

Summary

Crucible's block-detection scores a request as "blocked/caught" only when the
response is HTTP 403 or 429 (apps/demo-dashboard/src/server/engine.ts:949:
const isBlocked = response.status === 403 || response.status === 429).

Synapse, however, mitigates behavioral catches through a progressive
interrogator ladder where the soft tiers return HTTP 200. Those are
genuine detections, but Crucible records isBlocked = false and scores them as
Synapse failing to catch — a false negative ("fake red") in the honest
battery report. The inverse of the botnet false-positive: it makes Synapse look
like it missed something it actually caught.

Root cause — Synapse's response is escalation-dependent, not always 403

Verified in Synapse/apps/synapse-pingora/src/main.rs (ladder defined in
src/interrogator/mod.rs: Cookie → JS PoW → CAPTCHA → Tarpit → Block):

Level	Outcome	HTTP status	Wire signal	Crucible scores
1 Cookie	tracking cookie, request proxied upstream	200	Set-Cookie: __synapse_*	not blocked (arguably correct — surveillance, not mitigation)
2 JS PoW	interstitial challenge	200 (main.rs:3627)	x-challenge-type: js_pow	not blocked — FALSE NEGATIVE
3 CAPTCHA	challenge	200 (main.rs:3652)	x-challenge-type: captcha	not blocked — FALSE NEGATIVE
4 Tarpit	sleep(delay) then block	403 (main.rs:3697)	x-challenge-type: tarpit	blocked ✅
5 Block	hard block	403 (main.rs:3714, default 403)	x-challenge-level: 5	blocked ✅
— Rate limit (separate path)	—	429 (main.rs:2402,2486)	—	blocked ✅

Because the interrogator level is risk-score-driven, the same logical
detection (e.g. an IDOR walk) produces a 200 or a 403 depending on how much
risk the scenario accrues before the assertion fires. A short walk that only
reaches the JS/CAPTCHA tier scores as a miss; a sustained one that crosses the
Block threshold scores as a catch. Tarpit is NOT affected — despite the name it
ends in a 403, and there is no per-step request timeout (engine.ts:790 fetch
carries only the execution abort signal), so the delayed 403 still arrives and
scores correctly.

Scope of the bug

• Affected: any scenario asserting expect.blocked: true against a behavioral
  catch that Synapse mitigates via JS PoW (level 2) or CAPTCHA (level 3).
• Not affected: rate-based catches (429), tarpit/hard-block catches (403).

Mitigation available today (no engine change)

The challenge tier is self-identifying on the wire and the engine already
evaluates expect.headerPresent (engine.ts:988). A scenario can assert
headerPresent: "x-challenge-type" instead of blocked: true and the JS/CAPTCHA
tier scores as a pass now. This is a stopgap, not the fix — it pushes the
mapping knowledge onto every scenario author and doesn't compose with the binary
"blocked" semantics the battery report is built around.

Proposed fix

Widen the "Synapse acted" predicate so mitigation is a spectrum, surfaced as
distinct non-failing outcomes rather than collapsed to a boolean:

• blocked — 403 hard block (level 5) or tarpit (level 4)
• rate-limited — 429
• challenged — presence of x-challenge-type / x-challenge-level (levels 2-3)

Key on x-challenge-type / 403 / 429 — not on the level-1 tracking cookie,
which is surveillance, not mitigation. Counting the cookie would flip the error
into a false positive (fake green), the exact mistake we are guarding against.

Report renderers should show what Synapse did (challenged vs. blocked vs.
rate-limited), which is more useful for a battery report than a bare pass/fail.

Acceptance criteria

• A scenario step can express "caught" such that JS PoW (200 + x-challenge-type: js_pow)
  and CAPTCHA (200 + x-challenge-type: captcha) score as detections, not misses.
• 403 (block + tarpit) and 429 (rate limit) continue to score as caught.
• The level-1 tracking cookie alone does NOT score as caught (no fake green).
• Reports distinguish challenged / rate-limited / blocked outcomes.
• Regression scenario covers each interrogator tier against live Synapse.

Related

• Source of truth for the status mapping: Synapse interrogator
  (apps/synapse-pingora/src/main.rs challenge arms, src/interrogator/).
• Sibling fidelity gap (traffic generation, not scoring): Add distributed multi-source origination and TLS fingerprint control to the scenario engine #36.
• Defender-side counterparts: Synapse Build advanced API logic & abuse scenarios #3 (HeaderProfiler), Expand Crucible assessment exports beyond the current JSON and HTML report links #18 (TLS controls).