Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Configuration hook for generating Prometheus alerts in MSTransferor #12103

Open
amaltaro opened this issue Sep 19, 2024 · 0 comments
Open

Configuration hook for generating Prometheus alerts in MSTransferor #12103

amaltaro opened this issue Sep 19, 2024 · 0 comments

Comments

@amaltaro
Copy link
Contributor

Impact of the new feature
MSTransferor

Is your feature request related to a problem? Please describe.
MSTransferor can generate alerts for misconfigured workflows, which are delivered to our email inboxes. Here is one example [1].
We should be able to configure whether we want these alert/notifications to be created or not.

Describe the solution you'd like
In addition to checking with the Monitoring team who is supposed to be getting such alerts, we need to implement the following features:

  • create a service configuration that can be used to enable/disable these notifications
  • change services_config test environment to have it disabled by default
  • change MSTransferor code to respect the relevant configuration

Describe alternatives you've considered
Keep dealing with these unwanted alerts.

Additional context
[1]

Title example: [FIRING:1] ms-transferor: PU misconfiguration error. Workflow: user_TaskChain_Prod_SiteListsTest_v5_240919_144211_3106 (ms-transferor high wmcore)

Labels
alertname = ms-transferor: PU misconfiguration error. Workflow: user_TaskChain_Prod_SiteListsTest_v5_240919_144211_3106
service = ms-transferor
severity = high
tag = wmcore
Annotations
description = Workflow: user_TaskChain_Prod_SiteListsTest_v5_240919_144211_3106 could not proceed due to some PU misconfiguration,so it will be skipped.
hostname = ms-transferor-7dfc9d6ffc-zvqf8
summary = [MSTransferor] Workflow cannot proceed due to some PU misconfiguration.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: No status
Development

No branches or pull requests

1 participant