ci: add a watchdog for required checks stuck past a threshold

[JSONbored]

Summary

Directly motivated by today's incident: "Superagent Security Scan" (a required, third-party check this repo has zero control over) hung in_progress for 90+ minutes, silently stalling the entire auto-merge pipeline — this repo's gate merges based on GitHub's own mergeable_state, which only goes "clean" once every required check resolves, so one stuck required check blocks ALL open PRs, not just the one it's running on. Finding it required manually pulling check-run data PR-by-PR; nothing surfaced it proactively.

This can't fix a stuck check — nothing on this repo's side can, for a third-party check it doesn't control — it just makes the situation visible fast. .github/workflows/stuck-check-watchdog.yml runs scripts/check-stuck-required-checks.mjs every 15 minutes: it flags any open, non-draft PR where a required check has been running past a threshold (default 20 minutes, comfortably above the ~1-2.5 minutes these checks normally take) via a single PR comment, idempotent — it checks for an existing marker before posting again, so it won't spam the same stuck check repeatedly.

One real design constraint surfaced during this: required-check names are hardcoded (validate, Superagent Security Scan) rather than read live from branch protection. GET .../protection/required_status_checks needs "Administration" repository read permission, which the default GITHUB_TOKEN doesn't get even with an elevated permissions: block — confirmed against GitHub's own docs that this scope isn't in the grantable set for the ephemeral per-run token at all. Hardcoding avoids a dependency that would otherwise 403 on every scheduled run; the script has a comment explaining how to update the list by hand if the required checks on main ever change.

Scope

• The PR title follows type(scope): short summary Conventional Commit format.
• This PR is focused — one new script, one new scheduled workflow.
• This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
• I linked a currently open issue this PR resolves — N/A, maintainer-initiated build-tooling work.

Validation

• git diff --check
• npm run actionlint
• node --check scripts/check-stuck-required-checks.mjs
• Ran the script for real against the live repo (both --dry-run at the default 20-minute threshold, and --dry-run --threshold-minutes=0 to force-check anything currently in-flight): correctly found zero stuck checks now that today's incident has resolved.
• The threshold-minutes=0 dry run also incidentally proved a real distinction the script needs to get right: a currently-open PR had validate-tests shards genuinely queued-but-not-yet-started (no started_at yet, waiting on validate-code) at the exact moment I ran this — the script correctly did NOT flag them, since a job with no started_at is legitimately queued, not stuck. Confirmed this wasn't a silent detection failure by cross-checking the same PR's real state via gh pr checks directly.
• Verified the GITHUB_TOKEN branch-protection-permission constraint against GitHub's own documentation before designing around it, rather than discovering it via a failed production run.
• npm run typecheck / npm run test:coverage — not applicable; this PR touches no src/** file, only a standalone .mjs script and a workflow file.
• npm audit --audit-level=moderate — not applicable; no dependency changes.

If any required check was skipped, explain why: this PR is CI/build-tooling configuration — the skipped commands aren't applicable, and every actually-relevant command (including live verification against the real repo in dry-run mode, and independently cross-checking the one edge case that came up) is listed above.

Safety

• No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
• Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
• The posted PR comment is informational only, states plainly that this is very likely the check's own service issue and not the PR's content, and does not take any destructive or blocking action — it only ever comments, never modifies checks, branch protection, or PR state.

Notes

• Touches .github/workflows/** (a guarded path), so it'll be held for manual owner review rather than auto-merged.
• Lower rigor than the rest of this session's CI work by design: this is a "nice to have" spawned from today's incident, not an audited, hard-invariant-checked correctness fix like the duration-aware sharding PR. No synthetic unit-test harness was built for it — verification here is live dry-runs against the real repo instead, which felt like the more meaningful signal for a script whose entire job is reading real, current GitHub state.
• Not yet done from the same conversation: two background investigations (release-please job starvation pattern, CodeQL JS/TS coverage gap) are still running and will follow as separate PRs once they land.

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.21%. Comparing base (218a5ae) to head (c6d1871).
⚠️ Report is 3 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #7411   +/-   ##
=======================================
  Coverage   91.21%   91.21%           
=======================================
  Files         716      716           
  Lines       72789    72789           
  Branches    20858    20859    +1     
=======================================
  Hits        66392    66392           
  Misses       5355     5355           
  Partials     1042     1042           

Flag	Coverage Δ
shard-1	35.90% <ø> (ø)
shard-2	41.32% <ø> (-0.03%)	⬇️
shard-3	38.95% <ø> (ø)
shard-4	39.92% <ø> (ø)
shard-5	27.14% <ø> (ø)
shard-6	31.24% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.