Alert when the daily workspace backup sweep does not run - #839
Merged
Merged
Conversation
The sweep's only trace is one log line, so a cron that silently stops producing nothing looked exactly like a healthy quiet day. Confirming a backup happened meant counting objects in R2 by hand. Two provisioned rules in a new backups group: - no backup.completed in 26h (2h grace on the 05:00 UTC cron, which also absorbs Cloudflare re-delivering OTLP batches late) - any backup.failed, grouped by stage and projectId so the notification names the project The absence rule inverts the file-level `or vector(0)` convention: an empty result evaluates to 0 and trips the threshold instead of holding a Normal series, and noDataState is Alerting for the same reason. If the rule cannot confirm a backup ran, it should fire. Claude-Session: https://claude.ai/code/session_01JySapHzgzwPW1yBWHccZYf
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (2)
Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review. 📜 Recent review details🧰 Additional context used📓 Path-based instructions (1)NEVER use emojis anywhere - not in code, comments, documentation, plan files, commit messages, or examples.📄 CodeRabbit inference engine (.github/copilot-instructions.md) Files:
🔇 Additional comments (2)
📝 WalkthroughWalkthroughGrafana provisioning adds alerts for missing workspace backup completions and failed backup events. The observability README documents the alert group, queries, thresholds, grouping, and no-data behavior. ChangesBackup alerting
Priority: ⬇️ Low Estimated code review effort: 2 (Simple) | ~10 minutes Change: Feature Merge Risk: ⚪ Minimal · up to The backup alerts correctly detect missing completed sweeps and per-project backup failures. No merge-blocking risk is identified. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The backup sweep from #827 leaves exactly one trace per day: a single
backup.completedlog line. Nothing watches for its absence, so a cron that silently stops firing is indistinguishable from a healthy quiet day. Verifying that last night's backup actually happened meant listing R2 objects by hand and summing their sizes against the logged byte count.Rules
New
backupsgroup inconfig/grafana/provisioning/alerting/backups.yaml, following the shape ofinvitations.yaml.backup.completedin 26hbackup.failed, grouped bystageandprojectIdThe 26h window gives the 05:00 UTC cron a 2h grace period, which also absorbs Cloudflare re-delivering OTLP batches hours late.
The second rule covers both failure stages the sweep emits:
stage=sweepmeans a project has no snapshot for that day,stage=pre-deletemeans a project was torn down without its finaldeleted/copy, which nothing retries.One inversion worth reviewing
Everywhere else in
alerting/, a trailingor vector(0)exists to keep a Normal series alive so that No Data means Loki is broken rather than "nothing failed". On the absence rule it does the opposite: an empty result evaluates to 0 and trips thelt 1threshold.noDataStateisAlertingfor the same reason. A rule that cannot confirm a backup happened should fire, not stay quiet. Both the file header and the README call this out so it does not read as a copy-paste slip.Verification
All three LogQL expressions were run against production Loki before the file was written: the absence query returns 1 (today's sweep), both failure queries return 0.
Already deployed to the homelab Grafana:
deploy.shto rsync the config, then acorates-grafanarestart, since compose alone will not reload bind-mounted provisioning. Grafana loggedfinished to provision alertingwith no errors and/api/v1/provisioning/alert-ruleslists both UIDs. Merging this only brings the repo in line with what is already running.While checking the sweep's history I confirmed backups themselves are healthy: both runs since #827 landed (Sept 17, 8/8; Sept 18, 9/9), every object exists in
corates-backups-prod, and the sizes sum byte for byte to the logged totals. The two zero-row snapshots are genuinely empty workspaces, cross-checked against the live DOs via sync-admin stats.Not covered here
A project created after the sweep has no backup until the next midnight. Today that was a real project that took a full day of two-reviewer work in its first 19 hours. Fixing it means a second daily sweep or a snapshot on project create, which is a change to the worker rather than to alerting.
https://claude.ai/code/session_01JySapHzgzwPW1yBWHccZYf
Summary by CodeRabbit
New Features
Documentation