Debugging Matrix notifications /w Shoutrrr

[tilllt]

We are getting notification spammed by netdata, because it's detecting watchtower as unhealthy frequently. This is correct, since docker ps -a reports watchtower as unhealthy.

We tracked down the problem to shoutrrr, configured with Matrix notifications, as described here:

#1911

So, in summary
(Case A) All Containers are healthy: watchtower updates the containers and uses my report template fine to send Matrix notifications about it.

(Case B) One or more containers are unhealthy: watchtower does something with the unhealthy containers, tries to report about it through matrix, spams the matrix API, gets rate limited and it's container turns unhealthy ITSELF. Unlike the other unhealthy containers, that caused the problem with watchtower, netdata starts reporting the unhealthy status of watchtower.

Watchtower is NOT crashed in these moments, the unhealthy status reverts to healthy after approx 1min, causing ANOTHER notification. It's also working as usual updating all containers so it merely seems to be a hickup in the shoutrrr matrix notifications.

I know there are various workarounds for this, like fixing the unhealthy containers that watchtower stumbles over, turning off health notifications for watchtower etc.pp. I am not interested in those right now. I want to debug what exactly watchtower tries to send to matrix, that causes it to stall.

SHOUTRRR_DEBUG: true

does not output what's happening, so I can't figure out if there is a debug mechanism I can use AT ALL. I was thinking to use a different notification mechanism like SMTP to receive whatever watchtower tries to send when getting rate limited but maybe there is a better way?

It would be great if somebody more knowledgeable about watchtower could help me with this because it's really incredibly annoying.