Bug 1942867 - Add an alert management system for telemetry alerts. #9015
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gmierz
requested review from
a team,
beatrice-acasandrei and
esanuandra
as code owners
gmierz
force-pushed
the
telemetry-alert-manager-comp
branch
from
44fab00 to
0e26ca1
Compare
|
Here's a sample bug that is filed by this: https://bugzilla.mozilla.org/show_bug.cgi?id=1993145 |
Andrej1198
reviewed
Oct 24, 2025
treeherder/perf/auto_perf_sheriffing/telemetry_alerting/email_manager.py
Outdated
Show resolved
Hide resolved
treeherder/perf/auto_perf_sheriffing/telemetry_alerting/alert_manager.py
Show resolved
Hide resolved
gmierz
force-pushed
the
telemetry-alert-manager-comp
branch
from
94d89a2 to
4f7ecc0
Compare
Andrej1198
approved these changes
Oct 24, 2025
beatrice-acasandrei
approved these changes
Oct 28, 2025
…g telemetry bugs.
gmierz
force-pushed
the
telemetry-alert-manager-comp
branch
from
4f7ecc0 to
6f2e463
Compare
|
Another DB migration has landed before this one, so I had to remake the ones we're doing here. |
misspran
pushed a commit
to misspran/treeherder
that referenced
this pull request
Nov 3, 2025
…ozilla#9015) This patch adds a new alert management system for telemetry alerts. Some generic base and utility classes are added directly to the auto_perf_sheriffing folder. These are not specific to telemetry alerting and could be used in other performance sheriffing automation. The concrete classes for telemetry alert management are found in the treeherder/perf/auto_perf_sheriffing folder. These are then integrated into the telemetry detection code in Sherlock through the TelemetryAlertManager and run from TelemetryAlertManager.manage_alerts. The manage_alerts method is defined generically in the AlertManager class. It starts by updating the DB with any changes made in telemetry bugs in Bugzilla - this is only for their resolutions at the moment. After this, bugs are filed for the alerts that are generated for any probes that specify a bug should be filed (by setting the monitor.alert field to True in their probe definition). Once bugs are filed, modifications are made to these bugs and any existing bugs as needed. This currently only modifies the see_also field to associate all bugs filed for the same detection range together - in other words, all the bugs that are part of the same PerformanceTelemetryAlertSummary. At the end of this "bug handling" phase, emails are produced for any alerts that request it (either bugs are produced or emails, but never both to reduce spamming). Finally, it's possible that either the bug modifications or emails fail. In that case, we have a "house keeping" stage where we do retries of the failed alerts on a daily basis.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This set of patches adds a new alert management system for telemetry alerts. The commits below attempt to split up the system into some logical chunks with newer commits building on previous ones.
Some generic base and utility classes are added directly to the
auto_perf_sheriffingfolder. These are not specific to telemetry alerting and could be used in other performance sheriffing automation.The concrete classes for telemetry alert management are found in the
treeherder/perf/auto_perf_sheriffingfolder. These are then integrated into the telemetry detection code in Sherlock through theTelemetryAlertManagerand run fromTelemetryAlertManager.manage_alerts.The
manage_alertsmethod is defined generically in theAlertManagerclass. It starts by updating the DB with any changes made in telemetry bugs in Bugzilla - this is only for their resolutions at the moment. After this, bugs are filed for the alerts that are generated for any probes that specify a bug should be filed (by setting themonitor.alertfield to True in their probe definition). Once bugs are filed, modifications are made to these bugs and any existing bugs as needed. This currently only modifies thesee_alsofield to associate all bugs filed for the same detection range together - in other words, all the bugs that are part of the samePerformanceTelemetryAlertSummary. At the end of this "bug handling" phase, emails are produced for any alerts that request it (either bugs are produced or emails, but never both to reduce spamming). Finally, it's possible that either the bug modifications or emails fail. In that case, we have a "house keeping" stage where we do retries of the failed alerts on a daily basis.For treeherder-admins, the relevant changes will be in the first commit where I am adding a new env field to capture the
BUG_COMMENTER_API_KEYbeing set locally. This is needed for testing the bug modification aspect of the management system.