fix(observability): forward ops-alerts anomalies to Sentry, add a review-burst rule#3766
Merged
Merged
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3766 +/- ##
=======================================
Coverage 93.29% 93.30%
=======================================
Files 317 317
Lines 32368 32377 +9
Branches 11872 11873 +1
=======================================
+ Hits 30199 30208 +9
Misses 1537 1537
Partials 632 632
🚀 New features to boost your workflow:
|
…iew-burst rule Two changes to the hourly ops-alerts anomaly scan: 1. Fix a Sentry-visibility gap found while investigating the CI-stuck repeat-review incident: every existing anomaly (gate false-positive spike, slop score inverting, recommendations not panning out) was logged via console.warn with an `ev` field. forwardStructuredLogToSentry only wraps console.log/console.error and keys the issue off a field literally named `event` -- so none of these anomalies has EVER reached Sentry, regardless of whether Sentry is active. They only ever reached Workers Logs, which nobody was watching -- part of why the CI-stuck bleed went unnoticed for 20+ hours. Switched to console.error + an `event` field, matching the convention already used by selfhost_ai_provider_failed / regate_repair_exhausted / ci_stuck_review_repeat_suppressed. 2. Add a 4th anomaly rule: "review burst" -- flags a PR that published more than 6 review surfaces within a 2-hour rolling window, the exact signature of a stuck-CI finalize loop or sweep retry storm. Backed by a new findHottestReviewTargetForRepo query over the existing audit_events ledger (github_app.pr_public_surface_published is a genuine INSERT-only event per publish pass, unlike ai_review_cache or review_audit's gate_decision rows, which are both upserted at a fixed head SHA and so cannot see a same-head repeat). This runs on the existing hourly ops-alerts cron, so a future recurrence of this class of bleed surfaces within an hour instead of requiring a human to notice and query the database directly.
JSONbored
force-pushed
the
feat/orb-ops-alerts-sentry-and-review-burst
branch
from
July 6, 2026 08:55
886c092 to
cc88c6d
Compare
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
12 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Found while investigating the CI-stuck repeat-review incident (#3747, #3755, #3760):
ops-alertsanomaly scan (gate false-positive spike, slop score inverting, recommendations not panning out) has been logging viaconsole.warnwith anevfield since it shipped.forwardStructuredLogToSentryonly wrapsconsole.log/console.errorand keys the Sentry issue off a field literally namedevent-- so none of these anomalies has ever reached Sentry, active or not. They only ever reached Workers Logs, which is part of why the 20+-hour token-usage bleed this week went unnoticed until a human queried the database directly. Switched toconsole.error+ aneventfield, matching the conventionselfhost_ai_provider_failed/regate_repair_exhausted/ci_stuck_review_repeat_suppressedalready use.findHottestReviewTargetForRepoquery over the existingaudit_eventsledger;github_app.pr_public_surface_publishedis a genuine INSERT-only event per publish pass (unlikeai_review_cacheorreview_audit'sgate_decisionrows, both upserted at a fixed head SHA, so neither can see a same-head repeat). Runs on the existing hourly cron -- a future recurrence of this bleed class now surfaces within an hour instead of requiring manual discovery.No new DB table/migration, no OpenAPI/UI surface change (the internal
/v1/internal/ops/statsendpoint'sanomalies: string[]shape is unchanged -- the new rule just contributes one more possible string).Scope
type(scope): short summaryConventional Commit format.CONTRIBUTING.mdand does not reintroduce GitHub Pages, VitePress,site/, orCNAME.Validation
git diff --checknpm run actionlint-- not run (no workflow files touched)npm run typecheck-- cleannpm run test:coverage(full/unsharded) -- not run locally; ran all 3 affected files directly instead (test/unit/ops-wire.test.ts,test/unit/db-parsers.test.ts,test/unit/queue.test.ts-- 645/645 passed)npm run test:workers/npm run build:mcp/npm run test:mcp-pack/npm run ui:openapi:check/npm run ui:lint/npm run ui:typecheck/npm run ui:build/npm audit --audit-level=moderate-- not run locally (no wrangler/MCP/UI/dependency surface touched; confirmed no OpenAPI/UI schema references the changed shape); left to CI.findHottestReviewTargetForRepo(hottest-target selection, repo-scoping via LIKE, window-cutoff, no-match cases), the new "review burst" rule (flags at threshold, doesn't flag below it, doesn't flag when absent), and an end-to-endrunOpsAlerts/computeOpsStatsintegration test seeding real repeat-publish events. Updated the 3 existing anomaly tests (console.warn→console.error,ev→event) plus queue.test.ts's ops-alerts dispatch test.Safety