Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MQTT input sent to two different outputs completely stops reporting if one output unresponsive #16537

Open
annamooseity opened this issue Feb 20, 2025 · 2 comments
Labels
bug unexpected problem or unintended behavior

Comments

@annamooseity
Copy link
Contributor

Relevant telegraf.conf

[agent]
debug = true
quiet = false
interval = "10s"


### --- setup for just mocks ---- ###
[[inputs.mock]]
  metric_name = "fake_alpha"

  [inputs.mock.tags]
    destination_topic = "alpha"

  [[inputs.mock.constant]]
    name = "fake_val"
    value = "1234"


[[inputs.mock]]
  metric_name = "fake_beta"

  [inputs.mock.tags]
    destination_topic = "beta"

  [[inputs.mock.constant]]
    name = "fake_val"
    value = "5678"


[[outputs.mqtt]]
client_id = "fake_data_client"
namepass = ["fake*"]
servers = ["mqtt://mosquitto:1883"]
data_format = "json"
topic = "{{ .Tag \"destination_topic\" }}"

### -- end setup for mock data -- ###

[[inputs.mqtt_consumer]]
servers = ["mqtt://mosquitto:1883"]
data_format = "json"
topics = ["alpha"]
name_override = "alpha_mqtt"

[[inputs.mqtt_consumer]]
servers = ["mqtt://mosquitto:1883"]
data_format = "json"
topics = ["beta"]
name_override = "beta_mqtt"
max_undelivered_messages = 10

[[outputs.file]]
files = ["stdout"]
data_format = "json"
namepass = ["alpha_mqtt", "beta_mqtt"]

[[outputs.http]]
url = "http://notarealsite1234.com"
method = "PUT"
data_format = "json"
metric_batch_size = 1

Logs from Telegraf

2025-02-20T18:27:25Z I! Loading config: /etc/telegraf/telegraf.conf
2025-02-20T18:27:25Z I! Starting Telegraf 1.33.2 brought to you by InfluxData the makers of InfluxDB
2025-02-20T18:27:25Z I! Available plugins: 236 inputs, 9 aggregators, 33 processors, 26 parsers, 63 outputs, 6 secret-stores
2025-02-20T18:27:25Z I! Loaded inputs: mock (2x) mqtt_consumer (2x)
2025-02-20T18:27:25Z I! Loaded aggregators:
2025-02-20T18:27:25Z I! Loaded processors:
2025-02-20T18:27:25Z I! Loaded secretstores:
2025-02-20T18:27:25Z I! Loaded outputs: file http mqtt
2025-02-20T18:27:25Z I! Tags enabled: host=654f1341740f
2025-02-20T18:27:25Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"654f1341740f", Flush Interval:10s
2025-02-20T18:27:25Z W! [agent] The default value of 'skip_processors_after_aggregators' will change to 'true' with Telegraf v1.40.0! If you need the current default behavior, please explicitly set the option to 'false'!
2025-02-20T18:27:25Z D! [agent] Initializing plugins
2025-02-20T18:27:25Z D! [agent] Connecting outputs
2025-02-20T18:27:25Z D! [agent] Attempting connection to [outputs.http]
2025-02-20T18:27:25Z D! [agent] Successfully connected to outputs.http
2025-02-20T18:27:25Z D! [agent] Attempting connection to [outputs.mqtt]
2025-02-20T18:27:25Z D! [agent] Successfully connected to outputs.mqtt
2025-02-20T18:27:25Z D! [agent] Attempting connection to [outputs.file]
2025-02-20T18:27:25Z D! [agent] Successfully connected to outputs.file
2025-02-20T18:27:25Z D! [agent] Starting service inputs
2025-02-20T18:27:25Z I! [inputs.mqtt_consumer] Connected [mqtt://mosquitto:1883]
2025-02-20T18:27:25Z I! [inputs.mqtt_consumer] Connected [mqtt://mosquitto:1883]
2025-02-20T18:27:30Z D! [outputs.http] Buffer fullness: 2 / 10000 metrics
2025-02-20T18:27:30Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:27:30Z D! [outputs.http] Buffer fullness: 2 / 10000 metrics
2025-02-20T18:27:30Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:27:35Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:27:35Z D! [outputs.mqtt] Wrote batch of 2 metrics in 685.2µs
2025-02-20T18:27:35Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:27:35Z D! [outputs.http] Buffer fullness: 4 / 10000 metrics
2025-02-20T18:27:35Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:27:35Z D! [outputs.http] Buffer fullness: 4 / 10000 metrics
2025-02-20T18:27:35Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:27:39Z D! [outputs.http] Buffer fullness: 6 / 10000 metrics
2025-02-20T18:27:39Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:27:39Z D! [outputs.http] Buffer fullness: 6 / 10000 metrics
2025-02-20T18:27:39Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:27:45Z D! [outputs.file] Wrote batch of 2 metrics in 153.9µs
2025-02-20T18:27:45Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:27:45Z D! [outputs.mqtt] Wrote batch of 2 metrics in 563.5µs
2025-02-20T18:27:45Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:27:45Z D! [outputs.http] Buffer fullness: 8 / 10000 metrics
2025-02-20T18:27:45Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:27:45Z D! [outputs.http] Buffer fullness: 8 / 10000 metrics
2025-02-20T18:27:45Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:27:50Z D! [outputs.http] Buffer fullness: 10 / 10000 metrics
2025-02-20T18:27:50Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:27:50Z D! [outputs.http] Buffer fullness: 10 / 10000 metrics
2025-02-20T18:27:50Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:27:55Z D! [outputs.file] Wrote batch of 2 metrics in 144.2µs
2025-02-20T18:27:55Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:27:55Z D! [outputs.mqtt] Wrote batch of 2 metrics in 592.3µs
2025-02-20T18:27:55Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:27:55Z D! [outputs.http] Buffer fullness: 12 / 10000 metrics
2025-02-20T18:27:55Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:27:55Z D! [outputs.http] Buffer fullness: 12 / 10000 metrics
2025-02-20T18:27:55Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:00Z D! [outputs.http] Buffer fullness: 14 / 10000 metrics
2025-02-20T18:28:00Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:00Z D! [outputs.http] Buffer fullness: 14 / 10000 metrics
2025-02-20T18:28:00Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:05Z D! [outputs.file] Wrote batch of 2 metrics in 192.6µs
2025-02-20T18:28:05Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:28:05Z D! [outputs.mqtt] Wrote batch of 2 metrics in 511.4µs
2025-02-20T18:28:05Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:28:05Z D! [outputs.http] Buffer fullness: 16 / 10000 metrics
2025-02-20T18:28:05Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:05Z D! [outputs.http] Buffer fullness: 16 / 10000 metrics
2025-02-20T18:28:05Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:09Z D! [outputs.http] Buffer fullness: 18 / 10000 metrics
2025-02-20T18:28:09Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:09Z D! [outputs.http] Buffer fullness: 18 / 10000 metrics
2025-02-20T18:28:09Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:15Z D! [outputs.file] Wrote batch of 2 metrics in 192µs
2025-02-20T18:28:15Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:28:15Z D! [outputs.mqtt] Wrote batch of 2 metrics in 970.8µs
2025-02-20T18:28:15Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:28:15Z D! [outputs.http] Buffer fullness: 20 / 10000 metrics
2025-02-20T18:28:15Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:15Z D! [outputs.http] Buffer fullness: 20 / 10000 metrics
2025-02-20T18:28:15Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:20Z D! [outputs.http] Buffer fullness: 22 / 10000 metrics
2025-02-20T18:28:20Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:20Z D! [outputs.http] Buffer fullness: 22 / 10000 metrics
2025-02-20T18:28:20Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:25Z D! [outputs.file] Wrote batch of 2 metrics in 115.7µs
2025-02-20T18:28:25Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:28:25Z D! [outputs.mqtt] Wrote batch of 2 metrics in 406.9µs
2025-02-20T18:28:25Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:28:25Z D! [outputs.http] Buffer fullness: 24 / 10000 metrics
2025-02-20T18:28:25Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:25Z D! [outputs.http] Buffer fullness: 24 / 10000 metrics
2025-02-20T18:28:25Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:30Z D! [outputs.http] Buffer fullness: 26 / 10000 metrics
2025-02-20T18:28:30Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:30Z D! [outputs.http] Buffer fullness: 26 / 10000 metrics
2025-02-20T18:28:30Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:35Z D! [outputs.file] Wrote batch of 2 metrics in 125µs
2025-02-20T18:28:35Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:28:35Z D! [outputs.mqtt] Wrote batch of 2 metrics in 551µs
2025-02-20T18:28:35Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:28:35Z D! [outputs.http] Buffer fullness: 28 / 10000 metrics
2025-02-20T18:28:35Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:35Z D! [outputs.http] Buffer fullness: 28 / 10000 metrics
2025-02-20T18:28:35Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:39Z D! [outputs.http] Buffer fullness: 30 / 10000 metrics
2025-02-20T18:28:39Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:39Z D! [outputs.http] Buffer fullness: 30 / 10000 metrics
2025-02-20T18:28:39Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:45Z D! [outputs.file] Wrote batch of 2 metrics in 220.9µs
2025-02-20T18:28:45Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:28:45Z D! [outputs.mqtt] Wrote batch of 2 metrics in 643.6µs
2025-02-20T18:28:45Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:28:45Z D! [outputs.http] Buffer fullness: 32 / 10000 metrics
2025-02-20T18:28:45Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:45Z D! [outputs.http] Buffer fullness: 32 / 10000 metrics
2025-02-20T18:28:45Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:50Z D! [outputs.http] Buffer fullness: 34 / 10000 metrics
2025-02-20T18:28:50Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:50Z D! [outputs.http] Buffer fullness: 34 / 10000 metrics
2025-02-20T18:28:50Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:55Z D! [outputs.file] Wrote batch of 2 metrics in 199.9µs
2025-02-20T18:28:55Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:28:55Z D! [outputs.mqtt] Wrote batch of 2 metrics in 977.5µs
2025-02-20T18:28:55Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:28:55Z D! [outputs.http] Buffer fullness: 36 / 10000 metrics
2025-02-20T18:28:55Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:28:55Z D! [outputs.http] Buffer fullness: 36 / 10000 metrics
{"fields":{"timestamp":1740076050},"name":"beta_mqtt","tags":{"host":"654f1341740f","topic":"beta"},"timestamp":1740076055}
{"fields":{"timestamp":1740076050},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076055}
{"fields":{"timestamp":1740076060},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076065}
{"fields":{"timestamp":1740076060},"name":"beta_mqtt","tags":{"host":"654f1341740f","topic":"beta"},"timestamp":1740076065}
{"fields":{"timestamp":1740076070},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076075}
{"fields":{"timestamp":1740076070},"name":"beta_mqtt","tags":{"host":"654f1341740f","topic":"beta"},"timestamp":1740076075}
{"fields":{"timestamp":1740076080},"name":"beta_mqtt","tags":{"host":"654f1341740f","topic":"beta"},"timestamp":1740076085}
{"fields":{"timestamp":1740076080},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076085}
{"fields":{"timestamp":1740076090},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076095}
{"fields":{"timestamp":1740076090},"name":"beta_mqtt","tags":{"host":"654f1341740f","topic":"beta"},"timestamp":1740076095}
{"fields":{"timestamp":1740076100},"name":"beta_mqtt","tags":{"host":"654f1341740f","topic":"beta"},"timestamp":1740076105}
{"fields":{"timestamp":1740076100},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076105}
{"fields":{"timestamp":1740076110},"name":"beta_mqtt","tags":{"host":"654f1341740f","topic":"beta"},"timestamp":1740076115}
{"fields":{"timestamp":1740076110},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076115}
{"fields":{"timestamp":1740076120},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076125}
{"fields":{"timestamp":1740076120},"name":"beta_mqtt","tags":{"host":"654f1341740f","topic":"beta"},"timestamp":1740076125}
{"fields":{"timestamp":1740076130},"name":"beta_mqtt","tags":{"host":"654f1341740f","topic":"beta"},"timestamp":1740076135}
{"fields":{"timestamp":1740076130},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076135}
{"fields":{"timestamp":1740076140},"name":"beta_mqtt","tags":{"host":"654f1341740f","topic":"beta"},"timestamp":1740076145}
{"fields":{"timestamp":1740076140},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076145}
{"fields":{"timestamp":1740076150},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076155}
{"fields":{"timestamp":1740076160},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076165}
{"fields":{"timestamp":1740076170},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076175}
2025-02-20T18:28:55Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:00Z D! [outputs.http] Buffer fullness: 38 / 10000 metrics
2025-02-20T18:29:00Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:00Z D! [outputs.http] Buffer fullness: 38 / 10000 metrics
2025-02-20T18:29:00Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:05Z D! [outputs.file] Wrote batch of 2 metrics in 153.7µs
2025-02-20T18:29:05Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:29:05Z D! [outputs.mqtt] Wrote batch of 2 metrics in 655.3µs
2025-02-20T18:29:05Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:29:05Z D! [outputs.http] Buffer fullness: 40 / 10000 metrics
2025-02-20T18:29:05Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:05Z D! [outputs.http] Buffer fullness: 40 / 10000 metrics
2025-02-20T18:29:05Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:09Z D! [outputs.http] Buffer fullness: 42 / 10000 metrics
2025-02-20T18:29:09Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:09Z D! [outputs.http] Buffer fullness: 42 / 10000 metrics
2025-02-20T18:29:09Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:15Z D! [outputs.file] Wrote batch of 2 metrics in 212.7µs
2025-02-20T18:29:15Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:29:15Z D! [outputs.mqtt] Wrote batch of 2 metrics in 812.3µs
2025-02-20T18:29:15Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:29:15Z D! [outputs.http] Buffer fullness: 43 / 10000 metrics
2025-02-20T18:29:15Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:15Z D! [outputs.http] Buffer fullness: 43 / 10000 metrics
2025-02-20T18:29:15Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:20Z D! [outputs.http] Buffer fullness: 45 / 10000 metrics
2025-02-20T18:29:20Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:20Z D! [outputs.http] Buffer fullness: 45 / 10000 metrics
2025-02-20T18:29:20Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:25Z D! [outputs.file] Wrote batch of 1 metrics in 100µs
2025-02-20T18:29:25Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:29:25Z D! [outputs.mqtt] Wrote batch of 2 metrics in 518.5µs
2025-02-20T18:29:25Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:29:25Z D! [outputs.http] Buffer fullness: 46 / 10000 metrics
2025-02-20T18:29:25Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:25Z D! [outputs.http] Buffer fullness: 46 / 10000 metrics
2025-02-20T18:29:25Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:30Z D! [outputs.http] Buffer fullness: 48 / 10000 metrics
2025-02-20T18:29:30Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:30Z D! [outputs.http] Buffer fullness: 48 / 10000 metrics
2025-02-20T18:29:30Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:35Z D! [outputs.file] Wrote batch of 1 metrics in 109µs
2025-02-20T18:29:35Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:29:35Z D! [outputs.mqtt] Wrote batch of 2 metrics in 565.1µs
2025-02-20T18:29:35Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:29:35Z D! [outputs.http] Buffer fullness: 49 / 10000 metrics
2025-02-20T18:29:35Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:35Z D! [outputs.http] Buffer fullness: 49 / 10000 metrics
2025-02-20T18:29:35Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:39Z D! [outputs.http] Buffer fullness: 51 / 10000 metrics
2025-02-20T18:29:39Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:39Z D! [outputs.http] Buffer fullness: 51 / 10000 metrics
2025-02-20T18:29:39Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:45Z D! [outputs.file] Wrote batch of 1 metrics in 99.8µs
2025-02-20T18:29:45Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:29:45Z D! [outputs.mqtt] Wrote batch of 2 metrics in 453.7µs
2025-02-20T18:29:45Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:29:45Z D! [outputs.http] Buffer fullness: 52 / 10000 metrics
2025-02-20T18:29:45Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:45Z D! [outputs.http] Buffer fullness: 52 / 10000 metrics
2025-02-20T18:29:45Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:50Z D! [outputs.http] Buffer fullness: 54 / 10000 metrics
2025-02-20T18:29:50Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:50Z D! [outputs.http] Buffer fullness: 54 / 10000 metrics
2025-02-20T18:29:50Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:55Z D! [outputs.file] Wrote batch of 1 metrics in 124.8µs
2025-02-20T18:29:55Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:29:55Z D! [outputs.mqtt] Wrote batch of 2 metrics in 885.5µs
2025-02-20T18:29:55Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:29:55Z D! [outputs.http] Buffer fullness: 55 / 10000 metrics
2025-02-20T18:29:55Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:29:55Z D! [outputs.http] Buffer fullness: 55 / 10000 metrics
2025-02-20T18:29:55Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:00Z D! [outputs.http] Buffer fullness: 57 / 10000 metrics
2025-02-20T18:30:00Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:00Z D! [outputs.http] Buffer fullness: 57 / 10000 metrics
{"fields":{"timestamp":1740076180},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076185}
{"fields":{"timestamp":1740076190},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076195}
{"fields":{"timestamp":1740076200},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076205}
{"fields":{"timestamp":1740076210},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076215}
2025-02-20T18:30:00Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:05Z D! [outputs.file] Wrote batch of 1 metrics in 123.3µs
2025-02-20T18:30:05Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:30:05Z D! [outputs.mqtt] Wrote batch of 2 metrics in 716.5µs
2025-02-20T18:30:05Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:30:05Z D! [outputs.http] Buffer fullness: 58 / 10000 metrics
2025-02-20T18:30:05Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:05Z D! [outputs.http] Buffer fullness: 58 / 10000 metrics
2025-02-20T18:30:05Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:09Z D! [outputs.http] Buffer fullness: 60 / 10000 metrics
2025-02-20T18:30:09Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:09Z D! [outputs.http] Buffer fullness: 60 / 10000 metrics
2025-02-20T18:30:09Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:15Z D! [outputs.file] Wrote batch of 1 metrics in 90.8µs
2025-02-20T18:30:15Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:30:15Z D! [outputs.mqtt] Wrote batch of 2 metrics in 644.3µs
2025-02-20T18:30:15Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:30:15Z D! [outputs.http] Buffer fullness: 61 / 10000 metrics
2025-02-20T18:30:15Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:15Z D! [outputs.http] Buffer fullness: 61 / 10000 metrics
2025-02-20T18:30:15Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:20Z D! [outputs.http] Buffer fullness: 63 / 10000 metrics
2025-02-20T18:30:20Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:20Z D! [outputs.http] Buffer fullness: 63 / 10000 metrics
2025-02-20T18:30:20Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:25Z D! [outputs.file] Wrote batch of 1 metrics in 117.3µs
2025-02-20T18:30:25Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:30:25Z D! [outputs.mqtt] Wrote batch of 2 metrics in 905.4µs
2025-02-20T18:30:25Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:30:25Z D! [outputs.http] Buffer fullness: 64 / 10000 metrics
2025-02-20T18:30:25Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:25Z D! [outputs.http] Buffer fullness: 64 / 10000 metrics
2025-02-20T18:30:25Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:30Z D! [outputs.http] Buffer fullness: 66 / 10000 metrics
2025-02-20T18:30:30Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:30Z D! [outputs.http] Buffer fullness: 66 / 10000 metrics
2025-02-20T18:30:30Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:35Z D! [outputs.file] Wrote batch of 1 metrics in 81.8µs
2025-02-20T18:30:35Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:30:35Z D! [outputs.mqtt] Wrote batch of 2 metrics in 594.8µs
2025-02-20T18:30:35Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:30:35Z D! [outputs.http] Buffer fullness: 67 / 10000 metrics
2025-02-20T18:30:35Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:35Z D! [outputs.http] Buffer fullness: 67 / 10000 metrics
2025-02-20T18:30:35Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:39Z D! [outputs.http] Buffer fullness: 69 / 10000 metrics
2025-02-20T18:30:39Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:39Z D! [outputs.http] Buffer fullness: 69 / 10000 metrics
2025-02-20T18:30:39Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:45Z D! [outputs.file] Wrote batch of 1 metrics in 102.3µs
2025-02-20T18:30:45Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:30:45Z D! [outputs.mqtt] Wrote batch of 2 metrics in 876.4µs
2025-02-20T18:30:45Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:30:45Z D! [outputs.http] Buffer fullness: 70 / 10000 metrics
2025-02-20T18:30:45Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:45Z D! [outputs.http] Buffer fullness: 70 / 10000 metrics
2025-02-20T18:30:45Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:50Z D! [inputs.mqtt_consumer] Connecting [mqtt://mosquitto:1883]
2025-02-20T18:30:50Z I! [inputs.mqtt_consumer] Connected [mqtt://mosquitto:1883]
2025-02-20T18:30:50Z D! [outputs.http] Buffer fullness: 72 / 10000 metrics
2025-02-20T18:30:50Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:50Z D! [outputs.http] Buffer fullness: 72 / 10000 metrics
2025-02-20T18:30:50Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:55Z D! [outputs.file] Wrote batch of 1 metrics in 233.5µs
2025-02-20T18:30:55Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:30:55Z D! [outputs.mqtt] Wrote batch of 2 metrics in 756.1µs
2025-02-20T18:30:55Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:30:55Z D! [outputs.http] Buffer fullness: 73 / 10000 metrics
2025-02-20T18:30:55Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:30:55Z D! [outputs.http] Buffer fullness: 73 / 10000 metrics
2025-02-20T18:30:55Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:31:00Z D! [outputs.http] Buffer fullness: 75 / 10000 metrics
{"fields":{"timestamp":1740076220},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076225}
{"fields":{"timestamp":1740076230},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076235}
{"fields":{"timestamp":1740076240},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076245}
{"fields":{"timestamp":1740076250},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076255}
{"fields":{"timestamp":1740076260},"name":"alpha_mqtt","tags":{"host":"654f1341740f","topic":"alpha"},"timestamp":1740076265}
2025-02-20T18:31:00Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:31:00Z D! [outputs.http] Buffer fullness: 75 / 10000 metrics
2025-02-20T18:31:00Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:31:05Z D! [outputs.file] Wrote batch of 1 metrics in 127.9µs
2025-02-20T18:31:05Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:31:05Z D! [outputs.mqtt] Wrote batch of 2 metrics in 692µs
2025-02-20T18:31:05Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:31:05Z D! [outputs.http] Buffer fullness: 76 / 10000 metrics
2025-02-20T18:31:05Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:31:05Z D! [outputs.http] Buffer fullness: 76 / 10000 metrics
2025-02-20T18:31:05Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:31:05Z D! [agent] Stopping service inputs
2025-02-20T18:31:05Z D! [inputs.mqtt_consumer] Disconnecting [mqtt://mosquitto:1883]
2025-02-20T18:31:05Z D! [inputs.mqtt_consumer] Disconnected [mqtt://mosquitto:1883]
2025-02-20T18:31:05Z D! [inputs.mqtt_consumer] Disconnecting [mqtt://mosquitto:1883]
2025-02-20T18:31:06Z D! [inputs.mqtt_consumer] Disconnected [mqtt://mosquitto:1883]
2025-02-20T18:31:06Z D! [agent] Input channel closed
2025-02-20T18:31:06Z I! [agent] Hang on, flushing any cached metrics before shutdown
2025-02-20T18:31:06Z D! [outputs.file] Wrote batch of 1 metrics in 76.3µs
2025-02-20T18:31:06Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:31:06Z D! [outputs.mqtt] Buffer fullness: 0 / 10000 metrics
2025-02-20T18:31:06Z D! [outputs.http] Buffer fullness: 76 / 10000 metrics
2025-02-20T18:31:06Z E! [agent] Error writing to outputs.http: Put "http://notarealsite1234.com": dial tcp: lookup notarealsite1234.com on 127.0.0.11:53: no such host
2025-02-20T18:31:06Z I! [agent] Stopping running outputs
2025-02-20T18:31:06Z D! [agent] Stopped Successfully

System info

Official Telegraf Docker container for 1.32 and 1.33, Mosquitto docker container 2.0, Docker 20.10.12

Docker

version: '3.6'

services:

  telegraf:
    container_name: telegraf
    image: telegraf:1.33
    volumes:
    - ./telegraf.conf:/etc/telegraf/telegraf.conf
    restart: always
    depends_on:
      - mosquitto
    command: ["telegraf", "--config", "/etc/telegraf/telegraf.conf", "--debug"]

  mosquitto:
    container_name: mosquitto
    image: eclipse-mosquitto:2.0
    volumes:
    - ./mosquitto.conf:/mosquitto/config/mosquitto.conf
    ports:
      - 1883:1883


Steps to reproduce

  1. Have two MQTT consumers (A & B) listening on different topics. MQTT consumer B has a max_undelivered_messages set to a low number (in my example, it's set to 10)
  2. Have both MQTT consumers sending to TWO outputs, one of which is unresponsive / will fail to write, and one which we'll call "good"
  3. Send constant info over MQTT to both MQTT topics for A & B
  4. Monitor "good" output to make sure data from A & B are flowing
  5. Wait for MQTT B consumer to disconnect + reconnect
  6. Monitor "good" output for data from A & B

Expected behavior

Data from A & B continues to send to both outputs. After B disconnects, I expect it to reconnect and resume sending MQTT B messages to the "good" output.

Actual behavior

After B disconnects, it seems to reconnect but does not send any more data to outputs.

Additional info

This is not a resource issue, as I've confirmed that my docker containers are not even close to running out of memory / CPU. Attached is a diagram showing how I set up my telegraf.conf to reproduce the issue, but we were seeing this issue with independent MQTT messages coming in production.

Image

Here is a link to my Github repo with a docker compose, mosquitto.conf, and telegraf.conf that repro the issue reliably. https://github.com/annamooseity/telegraf-mqtt-weirdness

@annamooseity annamooseity added the bug unexpected problem or unintended behavior label Feb 20, 2025
@srebhan
Copy link
Member

srebhan commented Feb 20, 2025

@annamooseity that is intended because a metric is marked as delivered if it is sent to all outputs successfully. The number of in-flight metrics can be controlled via the max_undelivered_messages parameter...

What is your expectation?

@annamooseity
Copy link
Contributor Author

@srebhan that does make sense! My expectation would be one of the following, but I totally understand if that's not implemented yet. I'd be happy to take a crack at it myself.

  1. A message being sent to at least one output would mark the message as "delivered", and the "half-delivered" messages would remain in the output buffer for the output that is not receiving, but not prevent new MQTT messages from being received.
  2. There is an option when configuring the input to not wait for delivery to the outputs (effectively lowering the QoS of the MQTT connection to 0)
  3. The MQTT consumer logs that max_delivered_messages has been exceeded and the input will stop receiving more messages.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug unexpected problem or unintended behavior
Projects
None yet
Development

No branches or pull requests

2 participants