Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Not sending mail #4165

Open
danielstankw opened this issue Dec 18, 2024 · 1 comment
Open

Not sending mail #4165

danielstankw opened this issue Dec 18, 2024 · 1 comment

Comments

@danielstankw
Copy link

danielstankw commented Dec 18, 2024 •
edited
Loading

Hi all,
I have been breaking my head for past few days on the following:

I have a prometheusRule and AlermanagerConfig CRDs. And promtheus/ thanos installed using kube-prometheus
When I deploy them in K8s, I can see them in the UI
image

The issue is that only one of them send me email notification, and the other doesn't. Also, even if it does, it doesnt send email consistently.
How can I fix that?

My prometheusRule

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    release: monitoring
  name: daniel-custom-rules-prod
  namespace: monitoring
spec:
  groups:
    - name: "pod-error-rule"
      rules:
        - alert: PodErrorAlert
          for: 3m
          expr: count(kube_pod_container_status_terminated_reason{reason='Error'}) by (namespace, pod)
          labels:
            severity: critical
          annotations:
            summary: Pod has been in error state for more than 10 minutes.
            message: "pod {{ $labels.pod }} in the {{ $labels.namespace }} namespace has been in error state for more than 10m"
    - name: "pod-pending-rule"
      rules:
        - alert: PodPendingFor30mAlert
          for: 2m
          expr: count(kube_pod_status_phase{phase="Pending"} > 0) by (namespace, pod)
          labels:
            severity: critical
          annotations:
            summary: Pod has been pending for more than 30 minutes.
            message: "pod {{ $labels.pod }} in the {{ $labels.namespace }} namespace has been in pending state for more than 30m"

My alertmanagerConfig

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: alert-config
  namespace: monitoring
  labels:
    release: monitoring
spec:
  route:
    groupBy: ['cluster', 'namespace']
    groupWait: 30s
    groupInterval: 10m
    repeatInterval: 10m
    receiver: email
  receivers:
    - name: email
      emailConfigs:
        - to: '[email protected]'
          from: '[email protected]'
          smarthost: 'ZZZZ.XXXX.com:25'
          headers:
            - key: subject
              value: "[{{ .Status | toUpper }}] Alert Notification"
          requireTLS: false
          sendResolved: false
          html: |
            <html>
            <body>
              <h3>Alerts Summary</h3>       
              <ul>
              {{ range .Alerts }}
                <li>
                  <b>Alert:</b> {{ .Annotations.summary }}<br/>
                  <b>Description:</b> {{ .Annotations.message }}<br/>
                  <b>Environment:</b> {{ .Labels.env }}<br/>
                  <b>Cluster:</b> {{ .Labels.cluster }}<br/>
                  <b>Namespace:</b> {{ .Labels.namespace }}<br/>
                  <b>Pod:</b> {{ .Labels.pod }}<br/>
                  <b>Severity:</b> {{ .Labels.severity }}<br/>
                  <b>Status:</b> {{ .Status | toUpper }}<br/>
                  <b>Starts At:</b> {{ .StartsAt }}<br/>
                  {{ if eq .Status "resolved" }}
                  <b>Resolved At:</b> {{ .EndsAt }}<br/>
                  {{ end }}
                  <!-- Grafana Button -->
                  <a href="https://my-grafana.com/d/ddqn51m3mrn5sa/cluster-view?orgId=1&var-cluster={{ .Labels.cluster }}&var-resolution=30s" 
                    style="display:inline-block;padding:10px 20px;color:black;background-color:#d3d3d3;border-radius:5px;text-decoration:none;font-weight:bold;">
                    Click to view in Grafana
                  </a>
                </li>
              {{ end }}
              </ul>
            </body>
            </html>
@grobinson-grafana
Copy link
Contributor

Your Alertmanager configuration states that notifications should be sent every 10 minutes. Is it possible that some of your alerts are firing and then resolving within 10 minutes? This would mean no notification for those alerts, and is intentional.

If not, could you explain with an example of what you expected to happen, and what happened instead?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@danielstankw @grobinson-grafana