[BUG] Alerts in prometheus triggered from Cloud Watch metrics from YACE hangs in state PENDING, never gets to FIRING

[truskare]

Is there an existing issue for this?

• I have searched the existing issues

YACE version

0.43.0

Config file

No response

Current Behavior

Alerts in prometheus using cloud watch metrics from YACE never gets to FIRING state, always stuck in PENDING. I think this is because some cloudwatch metrics is not updated so often in AWS (around every 5 minutes) so prometheus treats the metrics/data as stale when evaluating the alerts.

Anyone else having this issue?

Expected Behavior

Alerts should get to FIRING state.

Steps To Reproduce

No response

Anything else?

No response

[tyagian]

No. I never had such issue.
It can be issue with tool used to fire the alert.
Alert stay in PENDING for a time period before firing. The time period depends on the alertmanager alert rule.

For example:

groups:
  - name: example-alert-rules
    rules:
      - alert: HighMemoryUsage
        expr: node_memory_Active_bytes > 0.9 * node_memory_MemTotal_bytes
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High memory usage detected"

In this case, alertmanager will check the alert condition matching for 5m. After that it will fire the alert.
You can understand based on that, it has nothing to do with YACE exporter.
Role of YACE is just to send metrics. You should run promql query in prometheus to check if Prometheus getting the metrics.

[tyagian]

@truskare If you understand the workflow, we can close this issue.