Recurring truncateSlug e2e flake needs quarantine+bisect, not repeated flake-tracking PRs

[danabaxia]

Signal: commit d8d52cc records the L1 real-claude e2e demo-e2e test 1 (truncateSlug write) failing for a 3rd distinct day (2026-07-04), explicitly escalated 'past flake-tracking'. The current response loop (PRs #789/#799/#798/#788) only records/promotes memory about the flake each time it recurs — it never changes CI behavior, so the same test keeps blocking/alerting on repeat.

Proposal: adopt the quarantine+bisect pattern common in mature OSS CI systems (e.g. Google's flaky-test bisection via targeted re-run ranges, and the quarantine-suite approach used by Buildkite Test Analytics/Shopify): once a test fails on N distinct days, auto-move it into a non-blocking quarantine lane (still runs, still reported, but doesn't gate merge) and simultaneously kick off a bisect job across the commit range since its last green run to localize the root cause. This converts a repeating manual-tracking cost into one bounded diagnostic job.

[github-actions]

Dedupe check

No exact duplicate. Related but distinct:

• Auto-file (and quarantine) flaky-test regressions from the ledger at an occurrence threshold #336 ("Auto-file (and quarantine) flaky-test regressions from the ledger") — closed/done, but scoped to the MIR ledger (nightly tester-suite-run scan of node --test/eval artifacts), reclassifying test:*:red findings to a non-blocking flaky tier. It does not touch integration.yml's L1 real-claude e2e job, which per CLAUDE.md explicitly gates the nightly — a separate lane from the MIR ledger. Note for the record: grepping the tree for everClean/flaky tier code turns up nothing, so Auto-file (and quarantine) flaky-test regressions from the ledger at an occurrence threshold #336's spec appears to have merged as docs-only (PR spec: Auto-file (and quarantine) flaky-test regressions from the ledger at an occurrence threshold (#336) #338) without a corresponding implementation PR — worth flagging to a human separately, but out of scope for this issue.
• L1 real-claude e2e: demo-e2e test 1 (truncateSlug write) failed once on nightly 2026-07-01 #699/L1 real-claude e2e: demo-e2e test 1 (truncateSlug write) failed again on nightly 2026-07-03 — 2nd distinct-day occurrence #788/[triager] Integration nightly - L1 real-claude e2e: truncateSlug delegation flake #798/L1 real-claude e2e: demo-e2e test 1 (truncateSlug write) failed — 3rd distinct day (2026-07-04), escalate past flake-tracking #801 — individual dated incident reports for the same recurring flake, all closed/done (flake-tracking, no CI-behavior change), which is exactly the repeated-cost problem this issue calls out.

This issue is a distinct proposal: convert the L1 nightly e2e gate itself into a quarantine+bisect lane, not the MIR ledger's classification of node --test results.

Scoping questions before a spec

1. Where does the quarantine lane live? L1 currently gates integration.yml pass/fail directly (not routed through the MIR ledger/Auto-file (and quarantine) flaky-test regressions from the ledger at an occurrence threshold #336 machinery at all). Does quarantine mean: (a) a new non-blocking job step/matrix leg in integration.yml that still runs+reports demo-e2e.test.js but no longer fails the workflow, gated by a per-test occurrence count, or (b) something that reuses/extends the MIR ledger's occurrence tracking so there's one flake-classification mechanism instead of two? Leaning (b) to avoid a parallel bookkeeping system, but that couples this to Auto-file (and quarantine) flaky-test regressions from the ledger at an occurrence threshold #336's actual implementation gap above.
2. What triggers bisect, and against what range? demo-e2e.test.js's failure mode here is a real-claude behavioral flake (model non-determinism in the truncateSlug write path), not obviously a code regression — bisecting a commit range may just relocate blame onto whichever commit happened to run when the model behaved differently. Should bisect only fire when a code change touched the delegate/path-guard/write pipeline since last-green (i.e. skip bisect entirely on doc/unrelated-file-only ranges), and be reported as "no regression found — confirmed environmental" rather than always pinning a commit?
3. Threshold — reuse Auto-file (and quarantine) flaky-test regressions from the ledger at an occurrence threshold #336's proposed defaults (occurrences>=2 across >=2 distinct days) or pick new ones specific to the nightly cadence (1/day)?
4. Bisect cost — each bisect step is a real claude -p e2e run (API cost + ~minutes). Should this cap at a small fixed number of candidate commits (e.g. only commits since last green, linear not exponential) and require a human nudge to actually kick it off, rather than auto-running on every recurrence?

Moving to discussing — please weigh in on 1–4, or a human can confirm defaults and I'll draft the spec.

[github-actions]

Triager sweep — 2026-07-04 17:xx UTC

Swept for lingering red checks per the hourly dev-mesh-triage cron. Findings:

No new/unclassified red check found. The only recent hard failure is the Integration (nightly) L1 real-claude e2e (gates) job from the 2026-07-04 09:13 run (test/demo-e2e.test.js:58, truncateSlug must be in B/lib/strings.js) — that occurrence was already run through src/dev-mesh/classify.js and reported in #801:

• Label: real_bug (escalated past flake-tracking)
• Reason: not infra_auth (claude -p smoke passed, suite ran to completion); not out_of_scope (intermittent, not 6+ consecutive nights per bug: Integration nightly L1 e2e failing on main 6+ days — do-mode write + path-guard denial log #150's bar); superficially flake-shaped (relatedToChange: false — diff since last relevant change touches unrelated paths) but crosses the pre-committed 3rd-distinct-day threshold from dev-mesh/curator/memory/workflows/task-first-delegate-prompt.md, so per that precedent it's classified real_bug rather than re-kicked a 3rd time on the same unverified hypothesis.
• Action taken then: handed to backlog/Coder with a 3-step investigation plan (get a stream-json transcript to confirm whether delegate_task actually fired; check agent B's own run log for a reached-but-didn't-write vs never-reached signature; extend MIR occurrence/quarantine tracking to L1).

Re-verifying now: #801 was closed via cross-reference from PR #802/#803, which promoted a process lesson (the self-diagnosing-assertion pattern) — not the investigation itself. None of the 3 investigation steps appear to have actually landed (no B-side run-log analysis or stream-json transcript is attached anywhere in #801/#802/#803, and MIR still has no L1 quarantine tracking). So the real_bug is still open work, just closed administratively.

This is consistent with what this issue is pointing at: the loop is re-documenting the recurrence instead of resolving it. Recommend: route the original 3-step investigation plan from #801 to Coder now (independent of whether quarantine+bisect is ultimately adopted) — a transcript + B-side log check is cheap and would finally distinguish "tool-registration race" from "write-pipeline bug" instead of guessing again on the next distinct-day recurrence.

No other lingering red checks: ci.yml is green on main and all open PRs; PR #772's action_required (non-required-check UNSTABLE) is already tracked/escalated via #778 (needs-human, open) with a reviewed draft fix in #779 — no new action needed there.

[github-actions]

Triager sweep — 2026-07-04 19:3x UTC

Re-ran the hourly dev-mesh-triage classification. No new lingering red check.

• Checked: Integration (nightly) runs (none since 2026-07-04T09:13:11Z, run 28701588786), ci.yml on main (green), all open PRs' status checks (research-fix: needs-human: PR#772 stuck (not-clean:UNSTABLE) — non-required check unmet (#778) #779 all green/mergeable, memory:promote — production-vs-fixture-precision-fix #772 has no checks reported, memory:promote — sibling-metric-noise-band-audit (PR #761 resolution) #769/memory(curator): expire regression-band-override-semantics (valid_to 2026-06-23) #495 all green).
• Label (unchanged): real_bug — same L1 real-claude e2e (gates) / demo-e2e.test.js:58 (truncateSlug must be in B/lib/strings.js) failure from the 09:13 run, already run through src/dev-mesh/classify.js and reported in L1 real-claude e2e: demo-e2e test 1 (truncateSlug write) failed — 3rd distinct day (2026-07-04), escalate past flake-tracking #801 (closed) and re-analyzed in the previous sweep (this issue's 17:23 UTC comment).
• Reason: unchanged from the 17:23 sweep — not infra_auth (claude smoke passed, suite completed), not out_of_scope (intermittent, below the 6-night bar), superficially flake-shaped (unrelated diff) but past the pre-committed 3rd-distinct-day threshold, so real_bug per precedent.
• Action: none new to take here — not re-kicking (no new occurrence to re-kick this cycle), and not duplicating the investigation plan already posted in L1 real-claude e2e: demo-e2e test 1 (truncateSlug write) failed — 3rd distinct day (2026-07-04), escalate past flake-tracking #801/this issue. That plan (get a stream-json transcript, check agent B's own run log, extend MIR occurrence tracking to L1) is still the next concrete step, and still requires a human to move a backlog issue to approved before Coder can claim it — Triager doesn't self-approve.

No action taken beyond this comment; nothing new to re-kick or escalate this cycle.

[github-actions]

Triager sweep — 2026-07-05 00:1x UTC

Hourly dev-mesh-triage cron. No new lingering red check this cycle.

• Checked: Integration (nightly) — no run since 28701588786 (2026-07-04T09:13:11Z; next scheduled nightly run is ~09:xx UTC today, hasn't fired yet). ci.yml on main — still green (last push run 28702632514, success). All open PRs' status checks: research-fix: needs-human: PR#772 stuck (not-clean:UNSTABLE) — non-required check unmet (#778) #779 all green, memory:promote — production-vs-fixture-precision-fix #772 still "no checks reported" (tracked separately via needs-human: PR#772 stuck (not-clean:UNSTABLE) — non-required check unmet #778, already escalated needs-human), memory:promote — sibling-metric-noise-band-audit (PR #761 resolution) #769/memory(curator): expire regression-band-override-semantics (valid_to 2026-06-23) #495 all green.
• Label (unchanged): real_bug — same L1 real-claude e2e (gates) / test/demo-e2e.test.js:58 (truncateSlug must be in B/lib/strings.js) failure from the 2026-07-04 09:13 nightly run. Classified via src/dev-mesh/classify.js precedence: not infra_auth (claude smoke passed, suite ran to completion, no auth/network signature in the log); not out_of_scope (intermittent, below the 6-consecutive-night bar for pre-existing); superficially flake-shaped (relatedToChange: false) but past the pre-committed 3rd-distinct-day escalation threshold, so real_bug per the precedent already applied in L1 real-claude e2e: demo-e2e test 1 (truncateSlug write) failed — 3rd distinct day (2026-07-04), escalate past flake-tracking #801 and the two prior sweeps on this issue.
• Action: none new — not re-kicking (no new occurrence this cycle to re-kick), not re-escalating (already needs-human-adjacent via the open investigation). The outstanding next step is still the 3-step investigation plan from L1 real-claude e2e: demo-e2e test 1 (truncateSlug write) failed — 3rd distinct day (2026-07-04), escalate past flake-tracking #801 (stream-json transcript to confirm delegate_task actually fired; check agent B's own run log for reached-but-didn't-write vs never-reached; extend MIR occurrence/quarantine tracking to L1) — it requires a human to move a backlog issue to approved before Coder can claim it; Triager doesn't self-approve.

Also noted but out of scope for this duty (not a CI-check job log classify.js applies to): #795 [mesh-heartbeat] analyst/inspiration-digest: failing is a daemon job-health signal, already needs-human/blocked/spec:draft, being handled through a separate track.

No action taken beyond this comment.

[github-actions]

Triager sweep — 2026-07-05 04:2x UTC

Hourly dev-mesh-triage cron. No new lingering red check this cycle.

• Checked: Integration (nightly) — no run since 28701588786 (2026-07-04T09:13:11Z; today's nightly hasn't fired yet, ~09:xx UTC). ci.yml on main — green (last push run 28702632514, success). All open PRs' status checks: research-fix: needs-human: PR#772 stuck (not-clean:UNSTABLE) — non-required check unmet (#778) #779 all pass, memory:promote — production-vs-fixture-precision-fix #772 still "no checks reported" (tracked via needs-human: PR#772 stuck (not-clean:UNSTABLE) — non-required check unmet #778, already needs-human), memory:promote — sibling-metric-noise-band-audit (PR #761 resolution) #769/memory(curator): expire regression-band-override-semantics (valid_to 2026-06-23) #495 all pass. Only intervening commit on main since the last nightly is 545318e (memory-file promotion only — no touch to src/delegate.js, src/path-guard.js, hooks/path-guard.js, or test/demo-e2e.test.js), so relatedToChange: false still holds.
• Label (unchanged): real_bug — same L1 real-claude e2e (gates) / test/demo-e2e.test.js:58 (truncateSlug must be in B/lib/strings.js) failure from the 2026-07-04 09:13 nightly run (28701588786). Classified via src/dev-mesh/classify.js precedence: not infra_auth (the claude -p smoke step succeeded — is_error:false, real model turn — and the suite ran to completion, 4/5 tests passing); not out_of_scope (intermittent, below the 6-consecutive-night pre-existing bar); superficially flake-shaped (unrelated diff since last relevant change) but past the pre-committed 3rd-distinct-day escalation threshold, so real_bug per the precedent applied in L1 real-claude e2e: demo-e2e test 1 (truncateSlug write) failed — 3rd distinct day (2026-07-04), escalate past flake-tracking #801 and the three prior sweeps on this issue.
• Action: none new — no new occurrence to re-kick this cycle, and not duplicating the outstanding investigation plan (stream-json transcript to confirm delegate_task actually fired; check agent B's own run log for reached-but-didn't-write vs never-reached; extend MIR occurrence/quarantine tracking to L1 — see L1 real-claude e2e: demo-e2e test 1 (truncateSlug write) failed — 3rd distinct day (2026-07-04), escalate past flake-tracking #801/prior sweeps). That plan still requires a human to move a backlog issue to approved before Coder can claim it; Triager doesn't self-approve.

No action taken beyond this comment.

[github-actions]

Triager sweep — 2026-07-05 (nightly run 28736395631)

New occurrence, but new evidence this time. Filed as #808 (not a duplicate — it's the concrete root-cause finding this issue's investigation plan was waiting on):

• Label: real_bug (same as L1 real-claude e2e: demo-e2e test 1 (truncateSlug write) failed — 3rd distinct day (2026-07-04), escalate past flake-tracking #801's precedent — 4th distinct-day occurrence).
• What changed: the self-diagnosing assertion from PR L1 real-claude e2e: demo-e2e test 1 (truncateSlug write) failed — 3rd distinct day (2026-07-04), escalate past flake-tracking (#801) #802/memory:promote — self-diagnosing-flake-assertion #803 finally captured agent A's actual final reply on a failing run. It shows A explicitly declined to call delegate_task because describe_self for agent-b reports a pure "book catalog" identity that contradicts the test prompt's "library peer's strings library" framing — not a tool-registration race, and not a downstream write-pipeline bug (no B-side log at all).
• Fix plan: update examples/agent-b/AGENT.md to also mention the string-utilities module it already carries (lib/strings.js), so the fixture's self-description stops contradicting the task. One fixture file, no protocol change. Full detail in L1 real-claude e2e: demo-e2e truncateSlug failure — root cause found (agent-b AGENT.md contradicts "strings library" framing), 4th distinct-day occurrence 2026-07-05 #808.
• Given this, I'd suggest the bisect half of this issue's proposal isn't needed for this flake specifically — it has a known, addressable fixture-content cause, not a commit-range regression to localize. The quarantine-lane idea may still be worth keeping for other future L1 flakes.

Not re-kicking. Handing #808 to backlog/Coder — needs a human approved label before it's claimed.

[github-actions]

Spec ready attempt 1 — push blocked (HTTP 403)

Spec authored and committed locally on branch spec/issue-804 (commit 130cc34):
docs/superpowers/specs/2026-07-05-l1-quarantine-lane-design.md

git push origin spec/issue-804 → HTTP 403 (Permission to danabaxia/agent_mesh.git denied to github-actions[bot]). Confirmed via gh api repos/danabaxia/agent_mesh --jq '.permissions': push: false, active token is the GitHub App installation (integration/15368), not a PAT. Same root blocker tracked centrally in #771 (DEV_MESH_PAT repo secret still not configured, confirmed still open as of this run).

What the spec covers: narrows the original quarantine+bisect idea (dedupe-checked 2026-07-04, no exact duplicate) to quarantine-only — a human-reviewed .github/quarantine.json entry moves one named node --test assertion in the L1 nightly (test/demo-e2e.test.js et al.) to a continue-on-error step once it's crossed the repo's existing 3-distinct-day recurrence bar (#801), so it stops blocking the nightly gate while still running and being visibly reported every night. Auto-bisect is dropped: #808 (closed) root-caused this issue's own motivating flake — a fixture (examples/agent-b/AGENT.md) whose self-description contradicted the test's task framing — purely through manual reasoning over the model's failing-run transcript, not commit-range bisection; L1 failures here are per-run model nondeterminism, not a single culpable commit, so an automated bisect job would frequently converge on nothing while burning real-claude runs. Un-quarantine is automatic-only-in-the-tightening-direction (5 consecutive clean nights → auto-PR removing the entry); entering quarantine always requires a human-reviewed PR. L3 adversarial (security, 100%-gated) is explicitly out of scope for quarantine.

Moving this issue to spec:draft + blocked. Will unblock once DEV_MESH_PAT is configured (see #771) — a human can then push spec/issue-804 (already committed locally) or re-run this scoping to re-author it.