ci(perf): choose and validate a stable Claude hook latency gate

[gnanirahulnutakki]

Parent: #364

Depends on:

• fix(claude): preserve native response-read errno and bound EINTR retries #378
• test(ci): publish raw Claude hook latency reports with runner metadata #379

Problem

The GitHub-hosted latency benchmark has produced pass/fail outcomes on identical source and dependency state. Parent #364 records a 25% observed failure rate and multiple same-SHA push/PR pairs with opposite results.

The current job also applies absolute thresholds on ubuntu-latest while some documented baselines were measured on Apple Silicon macOS. A threshold is not portable merely because both platforms execute the same source.

Before this signal can become required, the project needs a preregistered execution model, platform-bound baselines, an explicit false-positive budget, and an event policy that does not create duplicate or selectively rerun evidence.

Primary references, rechecked 2026-07-23:

• GitHub pull-request and merge-ref event semantics
• GitHub concurrency semantics
• GitHub-hosted runners
• GitHub Actions billing
• GitHub secure-use guidance
• GitHub self-hosted runners

Scope

1. Write a reviewed ADR or benchmark protocol choosing one execution model:
   • a controlled performance runner; or
   • a statistically defensible hosted multi-run model.
2. Bind every threshold and evidence budget to an explicit platform/runner envelope.
3. Define before collecting evidence: sample and independent-run counts, percentile estimator, decision rule, false-positive budget, missing/invalid-report treatment, retention, and first-attempt/no-selective-rerun policy.
4. Preserve a hard separation: any functional connect/write/read/protocol failure fails that attempt; statistical aggregation may address runner variance but may never vote away a functional failure.
5. Implement and deterministically test the report evaluator.
6. Decide event behavior for PRs into dev, pushes to dev, the dev → main promotion PR, and manual evidence collection.
7. Deduplicate only when source/tree identity and retained evidence are proved. Generic concurrency cancellation is not evidence deduplication.
8. Collect the preregistered number of independent first-attempt clean reports. Preserve failures; do not rerun until green.
9. Produce a branch-protection recommendation only after the evidence meets preregistered stability and false-positive criteria.

Out of scope:

• automatically changing main or dev branch protection;
• merging or promoting dev to main;
• weakening native transport/protocol failures;
• broadly raising thresholds to hide hosted-runner variance;
• deleting failed evidence; and
• provisioning a self-hosted runner without explicit owner approval.

Acceptance criteria

• A reviewed decision record covers correctness, reliability, security, performance, operability, and cost trade-offs for both candidate models.
• Thresholds and baselines are explicitly platform-bound.
• False-positive budget and evidence count are preregistered before new qualifying runs.
• A deterministic evaluator covers all-pass fixtures, statistical outliers, functional failures, missing reports, malformed/non-finite samples, unsupported runner metadata, and insufficient independent runs.
• Any functional native/protocol failure remains hard failure regardless of statistical voting.
• Every qualifying attempt is retained, including failed first attempts.
• No selective rerun is included as qualifying evidence.
• Push/PR/manual event behavior is documented and tested.
• Deduplication never relies solely on cancellation and never discards the only authoritative sample.
• The chosen design is demonstrated across the preregistered repeated clean-run set.
• CI remains informational until stability evidence is reviewed.
• Branch protection remains unchanged until a human reviewer/admin explicitly approves promotion of the latency signal.

Targeted tests and evidence

• Unit tests for the decision evaluator and every invalid-input path.
• Workflow contract tests for event filters, evidence identity, and non-canceling retention behavior.
• Fixture reports derived from test(ci): publish raw Claude hook latency reports with runner metadata #379's versioned artifact schema.
• Repeated qualifying runs collected without reruns or cherry-picking.
• Independent recomputation of every final decision from retained raw artifacts.
• Focused tests, workflow validation, Ruff/pre-commit where applicable, and check-local.sh --quick.

Security and cost

A persistent self-hosted runner is not a safe default for a public repository. If selected, it requires explicit owner approval and a separate design for trusted-event-only execution, ephemeral isolation, runner groups, least-privilege tokens, network boundaries, patching, log retention, and teardown.

A hosted multi-run design multiplies Actions minutes and artifact storage. The ADR must estimate run frequency, runner count, expected minutes, storage, and retention. Do not assume public-repository or self-hosted execution is cost-free.

If the chosen runner requires credentials, paid capacity, organization administration, or external infrastructure, document that dependency here, park the infrastructure action, and continue independently implementable evaluator/documentation work.

Dependencies and overlap

• Requires fix(claude): preserve native response-read errno and bound EINTR retries #378's stable native error contract and test(ci): publish raw Claude hook latency reports with runner metadata #379's raw versioned artifacts.
• ci(go): recognition benchmark CPU budget is unstable across unchanged production code #342 and its merged same-VM reference/budget design are useful precedent, but Claude hook thresholds still need platform-specific evidence.
• Open PR dev → main: v0.2.0 release promotion #17 is the human-gated dev → main promotion lane and must not be modified or merged here.
• Open PR feat(governance): enforce pre-action spend caps #314 currently touches docs/decisions/README.md; any ADR index edit must coordinate or rebase rather than overwrite it.
• Any workflow implementation must re-audit overlap with open PRs ci(deps): bump actions/setup-python from 6.3.0 to 7.0.0 #358/ci(deps): bump actions/setup-go from 6.5.0 to 7.0.0 #360.
• Branch protection remains human-controlled.