Sysdig alert on PV firing with unidentified scope

[stewartshea]

An [incorrect] alert is firing on PV utilisation with an unknown scope identified, when the scope should be focused on any prometheus metrics that match a specific namespace label.

Example Alert Event: https://app.sysdigcloud.com/#/event-feed?category=alert&eventId=862301847891861504&last=43200
Configured Trigger:

(avg(kubelet_volume_stats_used_bytes/kubelet_volume_stats_capacity_bytes{namespace='[namespace name]'}) by (persistentvolumeclaim) * 100) >= 50: 0

Reported Scope:

instance = localhost:9100
job = node

Actual Scope: prometheus namespace = [namespace name]

[stewartshea]

@dleard I've dropped a message in the Sysdig chat to see if they can shed some insight.

[stewartshea]

@dleard The issue might have been transient on their back-end. Are you able to re-enable the alerts and let me know if it happens again?

[dleard]

Thanks @stewartshea, that sounds likely. I re-enabled them last week and have had no further issues since. I'll let you know if it happens again.

[stewartshea]

Cool, thanks for the note! I saw the same with one of ours and turned it on as well.