You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Have two MQTT consumers (A & B) listening on different topics. MQTT consumer B has a max_undelivered_messages set to a low number (in my example, it's set to 10)
Have both MQTT consumers sending to TWO outputs, one of which is unresponsive / will fail to write, and one which we'll call "good"
Send constant info over MQTT to both MQTT topics for A & B
Monitor "good" output to make sure data from A & B are flowing
Wait for MQTT B consumer to disconnect + reconnect
Monitor "good" output for data from A & B
Expected behavior
Data from A & B continues to send to both outputs. After B disconnects, I expect it to reconnect and resume sending MQTT B messages to the "good" output.
Actual behavior
After B disconnects, it seems to reconnect but does not send any more data to outputs.
Additional info
This is not a resource issue, as I've confirmed that my docker containers are not even close to running out of memory / CPU. Attached is a diagram showing how I set up my telegraf.conf to reproduce the issue, but we were seeing this issue with independent MQTT messages coming in production.
@annamooseity that is intended because a metric is marked as delivered if it is sent to all outputs successfully. The number of in-flight metrics can be controlled via the max_undelivered_messages parameter...
@srebhan that does make sense! My expectation would be one of the following, but I totally understand if that's not implemented yet. I'd be happy to take a crack at it myself.
A message being sent to at least one output would mark the message as "delivered", and the "half-delivered" messages would remain in the output buffer for the output that is not receiving, but not prevent new MQTT messages from being received.
There is an option when configuring the input to not wait for delivery to the outputs (effectively lowering the QoS of the MQTT connection to 0)
The MQTT consumer logs that max_delivered_messages has been exceeded and the input will stop receiving more messages.
Relevant telegraf.conf
Logs from Telegraf
System info
Official Telegraf Docker container for 1.32 and 1.33, Mosquitto docker container 2.0, Docker 20.10.12
Docker
Steps to reproduce
max_undelivered_messages
set to a low number (in my example, it's set to 10)Expected behavior
Data from A & B continues to send to both outputs. After B disconnects, I expect it to reconnect and resume sending MQTT B messages to the "good" output.
Actual behavior
After B disconnects, it seems to reconnect but does not send any more data to outputs.
Additional info
This is not a resource issue, as I've confirmed that my docker containers are not even close to running out of memory / CPU. Attached is a diagram showing how I set up my telegraf.conf to reproduce the issue, but we were seeing this issue with independent MQTT messages coming in production.
Here is a link to my Github repo with a docker compose, mosquitto.conf, and telegraf.conf that repro the issue reliably. https://github.com/annamooseity/telegraf-mqtt-weirdness
The text was updated successfully, but these errors were encountered: