Skip to content

fix(observability): forward ops-alerts anomalies to Sentry, add a review-burst rule#3766

Merged
JSONbored merged 1 commit into
mainfrom
feat/orb-ops-alerts-sentry-and-review-burst
Jul 6, 2026
Merged

fix(observability): forward ops-alerts anomalies to Sentry, add a review-burst rule#3766
JSONbored merged 1 commit into
mainfrom
feat/orb-ops-alerts-sentry-and-review-burst

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

Found while investigating the CI-stuck repeat-review incident (#3747, #3755, #3760):

  1. Sentry-visibility gap fix. The hourly ops-alerts anomaly scan (gate false-positive spike, slop score inverting, recommendations not panning out) has been logging via console.warn with an ev field since it shipped. forwardStructuredLogToSentry only wraps console.log/console.error and keys the Sentry issue off a field literally named event -- so none of these anomalies has ever reached Sentry, active or not. They only ever reached Workers Logs, which is part of why the 20+-hour token-usage bleed this week went unnoticed until a human queried the database directly. Switched to console.error + an event field, matching the convention selfhost_ai_provider_failed / regate_repair_exhausted / ci_stuck_review_repeat_suppressed already use.
  2. New anomaly rule: "review burst". Flags any PR that published more than 6 review surfaces within a 2-hour rolling window -- the exact signature of the stuck-CI finalize loop / sweep retry storm found this week. Backed by a new findHottestReviewTargetForRepo query over the existing audit_events ledger; github_app.pr_public_surface_published is a genuine INSERT-only event per publish pass (unlike ai_review_cache or review_audit's gate_decision rows, both upserted at a fixed head SHA, so neither can see a same-head repeat). Runs on the existing hourly cron -- a future recurrence of this bleed class now surfaces within an hour instead of requiring manual discovery.

No new DB table/migration, no OpenAPI/UI surface change (the internal /v1/internal/ops/stats endpoint's anomalies: string[] shape is unchanged -- the new rule just contributes one more possible string).

Scope

Validation

  • git diff --check
  • npm run actionlint -- not run (no workflow files touched)
  • npm run typecheck -- clean
  • npm run test:coverage (full/unsharded) -- not run locally; ran all 3 affected files directly instead (test/unit/ops-wire.test.ts, test/unit/db-parsers.test.ts, test/unit/queue.test.ts -- 645/645 passed)
  • npm run test:workers / npm run build:mcp / npm run test:mcp-pack / npm run ui:openapi:check / npm run ui:lint / npm run ui:typecheck / npm run ui:build / npm audit --audit-level=moderate -- not run locally (no wrangler/MCP/UI/dependency surface touched; confirmed no OpenAPI/UI schema references the changed shape); left to CI.
  • New behavior has full regression coverage: findHottestReviewTargetForRepo (hottest-target selection, repo-scoping via LIKE, window-cutoff, no-match cases), the new "review burst" rule (flags at threshold, doesn't flag below it, doesn't flag when absent), and an end-to-end runOpsAlerts/computeOpsStats integration test seeding real repeat-publish events. Updated the 3 existing anomaly tests (console.warn→console.error, evevent) plus queue.test.ts's ops-alerts dispatch test.

Safety

  • No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
  • Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
  • N/A -- no auth/cookie/CORS/GitHub App/Cloudflare/session changes.
  • N/A -- no API/OpenAPI/MCP surface changed (internal-only endpoint shape unchanged).
  • N/A -- no UI changes.
  • N/A -- no docs/changelog changes needed.

Comment thread src/db/repositories.ts
@superagent-security superagent-security Bot added the pr:flagged PR flagged for review by security analysis. label Jul 6, 2026
@codecov

codecov Bot commented Jul 6, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.30%. Comparing base (2302554) to head (cc88c6d).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #3766   +/-   ##
=======================================
  Coverage   93.29%   93.30%           
=======================================
  Files         317      317           
  Lines       32368    32377    +9     
  Branches    11872    11873    +1     
=======================================
+ Hits        30199    30208    +9     
  Misses       1537     1537           
  Partials      632      632           
Files with missing lines Coverage Δ
src/db/repositories.ts 96.54% <100.00%> (+<0.01%) ⬆️
src/review/ops-wire.ts 95.83% <100.00%> (+0.59%) ⬆️
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…iew-burst rule

Two changes to the hourly ops-alerts anomaly scan:

1. Fix a Sentry-visibility gap found while investigating the CI-stuck
   repeat-review incident: every existing anomaly (gate false-positive
   spike, slop score inverting, recommendations not panning out) was
   logged via console.warn with an `ev` field. forwardStructuredLogToSentry
   only wraps console.log/console.error and keys the issue off a field
   literally named `event` -- so none of these anomalies has EVER reached
   Sentry, regardless of whether Sentry is active. They only ever reached
   Workers Logs, which nobody was watching -- part of why the CI-stuck
   bleed went unnoticed for 20+ hours. Switched to console.error + an
   `event` field, matching the convention already used by
   selfhost_ai_provider_failed / regate_repair_exhausted /
   ci_stuck_review_repeat_suppressed.

2. Add a 4th anomaly rule: "review burst" -- flags a PR that published
   more than 6 review surfaces within a 2-hour rolling window, the exact
   signature of a stuck-CI finalize loop or sweep retry storm. Backed by
   a new findHottestReviewTargetForRepo query over the existing
   audit_events ledger (github_app.pr_public_surface_published is a
   genuine INSERT-only event per publish pass, unlike ai_review_cache or
   review_audit's gate_decision rows, which are both upserted at a fixed
   head SHA and so cannot see a same-head repeat). This runs on the
   existing hourly ops-alerts cron, so a future recurrence of this class
   of bleed surfaces within an hour instead of requiring a human to
   notice and query the database directly.
@JSONbored
JSONbored force-pushed the feat/orb-ops-alerts-sentry-and-review-burst branch from 886c092 to cc88c6d Compare July 6, 2026 08:55
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@superagent-security superagent-security Bot removed the pr:flagged PR flagged for review by security analysis. label Jul 6, 2026
@JSONbored JSONbored self-assigned this Jul 6, 2026
@JSONbored
JSONbored merged commit 74131c1 into main Jul 6, 2026
10 checks passed
@JSONbored
JSONbored deleted the feat/orb-ops-alerts-sentry-and-review-burst branch July 6, 2026 09:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant