CI: harden flaky latency benchmark before making it required

[gnanirahulnutakki]

Summary

The GitHub-hosted Latency benchmarks (informational) job is nondeterministic and currently lacks enough diagnostics to distinguish socket timeouts, EINTR, and other native read failures.

This is not a v0.2.0 release blocker: the job is explicitly continue-on-error, it is excluded from the required tests aggregate, the exact dev push passed all mandatory checks, and the aggregate tests check is green. It must be hardened before latency is promoted to a required gate.

Exact same-SHA evidence

Both jobs tested head SHA 8af2340965a28894129a7300611f8dc210023776 on 2026-07-23:

• PR job failed: run 29973387753 / job 89099994191
• Push job passed: run 29973383602 / job 89099981314

The PR checkout used synthetic merge commit 51a98894b63f5fdfaba412b8e45595005c08b114, but its tree SHA was a41b5d90ea17bf44e41fa88cb899ac519bebd939, exactly equal to the tree of 8af2340965a28894129a7300611f8dc210023776. Both jobs used:

• Actions runner 2.336.0
• Ubuntu 24.04 image 20260714.240.1
• CPython 3.13.14
• the same resolved Python package versions

They ran on separate workers and regions:

• failed: worker 73fe17d7-dcc9-4ace-bd40-6c0deb4ae076, centralus
• passed: worker b3b12663-af79-4a89-914a-74343651b920, westus

GitHub documents that each standard GitHub-hosted ubuntu-latest job runs in a fresh VM: https://docs.github.com/en/actions/reference/runners/github-hosted-runners

Failure evidence

Failed job:

• subprocess cold path: median 173.36 ms, p95 302.34 ms, p99 504.39 ms
• native daemon-client warmup: return code 11, empty stderr
• wrapper daemon-client measured call: return code 11, empty stderr
• in-process hot path: median 2.84 ms, p95 74.99 ms, p99 171.95 ms
• failed assertion: 74.98812900001894 < 10
• result: 3 failed, 1 passed

Passing same-SHA job:

• subprocess cold path: median 262.66 ms, p95 277.94 ms, p99 291.44 ms
• native daemon-client: median 5.13 ms, p95 5.47 ms, p99 5.93 ms
• wrapper telemetry: median 6.64 ms, p95 7.02 ms, p99 7.21 ms
• in-process hot path: median 4.10 ms, p95 4.49 ms, p99 4.99 ms
• result: 4 passed

In the generated native client, return code 11 means read() returned a negative value. The test configures a 100 ms socket timeout, so a receive timeout under scheduler stalls is the leading explanation, but the client collapses all negative read errors to code 11 and emits no errno; the exact cause cannot be proved from the current log.

Historical flake evidence

In the 20 most recent latency jobs inspected, 5 failed: a 25% observed failure rate.

Four same-SHA push/PR pairs produced one pass and one failure:

• 8af2340965a28894129a7300611f8dc210023776
• 7b88e693f1228afa94584d3f997e92414eb892c8
• a1b561f2b06dc7892c2ab0106cf869da3944ddca
• 23047fb8a21eb729852ae4690a3daf35a7616a87

The event type alternated between the passing and failing member of those pairs, so this is not a consistent push-vs-PR code-path difference.

Diagnosis

High confidence: hosted-runner tail-latency jitter / benchmark flakiness, not a deterministic Ardur regression.

Evidence:

• identical tree and dependencies passed concurrently on another runner;
• failures include large p95/p99 stalls in the in-process benchmark, not only Unix-socket IPC;
• repeated same-SHA mixed outcomes occur historically;
• worker IDs and regions differ, so there is no shared process, socket, or filesystem race between the duplicate jobs.

The simultaneous push and PR workflows create duplicate benchmark samples and expose the flake, but direct cross-job interference is not established because the jobs run in separate VMs.

Current workflow semantics and release disposition

At 8af2340, .github/workflows/tests.yml:

• names the job Latency benchmarks (informational);
• sets job-level continue-on-error: true;
• excludes latency-bench from the blocking tests aggregate's needs list.

main branch protection requires the aggregate tests context, not the raw latency job. Both same-SHA workflow runs concluded success, and the required aggregate tests check passed. The raw informational job still correctly records its own failed check for visibility.

Therefore this issue is parked as a CI-quality follow-up and does not block the current release. Do not make the latency job required until the acceptance criteria below are met.

Acceptance criteria

• Preserve native error detail: report the read failure's numeric and symbolic errno (without logging request payloads, tokens, or sensitive paths).
• Record and upload the raw latency sample distribution plus runner/image/Python metadata, not only median/p95/p99.
• Handle EINTR correctly by retrying the interrupted syscall; define and test the policy for transient receive timeouts without masking persistent socket, protocol, or daemon failures.
• Distinguish functional failures (connect/write/read/protocol) from statistical threshold failures in test output.
• Add deterministic tests for native read-error handling and retry behavior.
• Decide and document the benchmark execution model: a dedicated performance runner, or a statistically defensible multi-run gate with an explicit false-positive budget and platform-specific baselines.
• Decide whether to deduplicate push/PR benchmark execution for the same SHA; document the chosen CI concurrency/event policy.
• Demonstrate that the chosen design is stable across repeated clean runs before adding the latency job to required branch-protection checks.
• Keep genuine daemon/client errors fail-closed and visible; do not solve the flake by broadly suppressing failures or merely raising thresholds.

Decomposition — 2026-07-23

A fresh audit at exact dev dfe4905bccfa3189ab2ee97e0d8f206a711ec9d4 found three independently reviewable concerns. This issue is now the tracker and is linked to three formal subissues:

• fix(claude): preserve native response-read errno and bound EINTR retries #378 — preserve native response-read errno, bound EINTR retries, define timeout policy, and add deterministic native tests.
• test(ci): publish raw Claude hook latency reports with runner metadata #379 — publish versioned raw sample distributions and bounded runner/Python metadata, separating functional from statistical failures.
• ci(perf): choose and validate a stable Claude hook latency gate #380 — choose and validate the execution model, false-positive budget, platform baselines, event policy, and promotion evidence.

Original acceptance-criteria mapping:

• Numeric/symbolic read errno; correct EINTR and timeout policy; deterministic native retry tests → fix(claude): preserve native response-read errno and bound EINTR retries #378
• Raw samples and metadata; functional versus threshold reporting → test(ci): publish raw Claude hook latency reports with runner metadata #379
• Dedicated-runner versus hosted multi-run decision; event/deduplication policy; repeated first-attempt stability proof → ci(perf): choose and validate a stable Claude hook latency gate #380
• Genuine daemon/client failures remain fail closed → all three children

Current disposition:

• CI: harden flaky latency benchmark before making it required #364 remains non-blocking for v0.2.0.
• The raw latency job remains informational and outside the blocking tests aggregate.
• No latency context should be added to branch protection until all children are complete and the final evidence is reviewed by a human.
• fix(claude): preserve native response-read errno and bound EINTR retries #378 is collision-free and independently implementable.
• test(ci): publish raw Claude hook latency reports with runner metadata #379 currently overlaps open workflow PRs ci(deps): bump actions/setup-python from 6.3.0 to 7.0.0 #358/ci(deps): bump actions/setup-go from 6.5.0 to 7.0.0 #360 and must wait, coordinate, or rebase without overwriting them.
• ci(perf): choose and validate a stable Claude hook latency gate #380 depends on fix(claude): preserve native response-read errno and bound EINTR retries #378/test(ci): publish raw Claude hook latency reports with runner metadata #379 evidence. Any controlled runner, paid capacity, credentials, organization administration, or branch-protection mutation remains human-gated and must be parked precisely if required.

[gnanirahulnutakki]

Decomposition checkpoint — 2026-07-23

A fresh live audit at exact dev dfe4905bccfa3189ab2ee97e0d8f206a711ec9d4 found no duplicate child, matching remote branch, linked PR, assignee, project item, or active owner. #364 is now linked to three focused subissues:

• fix(claude): preserve native response-read errno and bound EINTR retries #378 — native errno preservation and bounded EINTR; collision-free and safe to implement.
• test(ci): publish raw Claude hook latency reports with runner metadata #379 — raw versioned latency reports/artifacts; file-overlaps open PRs ci(deps): bump actions/setup-python from 6.3.0 to 7.0.0 #358/ci(deps): bump actions/setup-go from 6.5.0 to 7.0.0 #360 and must coordinate or rebase.
• ci(perf): choose and validate a stable Claude hook latency gate #380 — execution-model and stable-gate decision; depends on fix(claude): preserve native response-read errno and bound EINTR retries #378/test(ci): publish raw Claude hook latency reports with runner metadata #379 evidence and keeps runner/branch-protection decisions human-gated.

The cited POSIX.1-2024, Linux, Apple, and GitHub primary documentation was re-opened during decomposition. POSIX confirms EINTR only when no data transferred and socket-read equivalence to recv(); Linux requires portable handling of EAGAIN|EWOULDBLOCK; current GitHub docs preserve the artifact, hosted/self-hosted runner, event, concurrency, security, and billing boundaries recorded in the children.

This classifies #364 as correctly decomposed, not implemented. It remains an open tracker. The informational latency job stays outside the required tests aggregate, and no branch-protection or main change is authorized.