[Bots.Experts][Cymru Whois] Pipeline failed

[ludoComp9]

Hello,

From my IntelMQ instance running on Debian 12 (amd64) with IntelMQ 3.2.1, Cymru Whois bot expert failed with following error messages:

cymru-whois-expert: Loading destination pipeline and queues {'_default': ['File-Output-3-queue']}.
cymru-whois-expert: Connected to destination queues.
cymru-whois-expert: Pipeline failed.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/intelmq/lib/bot.py", line 348, in start
    self.process()
  File "/usr/lib/python3/dist-packages/intelmq/bots/experts/cymru_whois/expert.py", line 60, in process
    self.acknowledge_message()
  File "/usr/lib/python3/dist-packages/intelmq/lib/bot.py", line 751, in acknowledge_message
    self.__source_pipeline.acknowledge()
  File "/usr/lib/python3/dist-packages/intelmq/lib/pipeline.py", line 165, in acknowledge
    self._acknowledge()
  File "/usr/lib/python3/dist-packages/intelmq/lib/pipeline.py", line 303, in _acknowledge
    raise exceptions.PipelineError("Could not pop message from internal queue "
intelmq.lib.exceptions.PipelineError: pipeline failed - 'Could not pop message from internal queue for acknowledgement. Return value was None.'
cymru-whois-expert: Bot will continue in 15 seconds.
cymru-whois-expert: Loading source pipeline and queue 'cymru-whois-expert-queue'.
cymru-whois-expert: Connected to source queue.
cymru-whois-expert: Loading destination pipeline and queues {'_default': ['File-Output-3-queue']}.
cymru-whois-expert: Connected to destination queues.
cymru-whois-expert: Got no result from Cymru for IP address '90.84.41.204'.
cymru-whois-expert: Processed 500 messages since last logging.
[...]

Note:
dnspython module version is 2.3.0

Execution of this expert bot is really really slow.
Same configuration worked fine before IntelMQ 3.2.1.

Any idea ?

Regards,

[sebix]

Did the bot work normally after the hiccup?

[ludoComp9]

It worked fine with IntelMQ3.1

[kamil-certat]

Have you changed anything in the pipeline (redis etc.) configuration? How are other bots (experts, parsers, or outputs) working?

[sebix]

Can you check the process list if there is no other cymru expert running?

[ludoComp9]

Single one Cymru expert process is running:

intelmq     1603       1 83 13:25 ?        00:02:28 /usr/bin/python3 /usr/bin/intelmq.bots.parsers.shadowserver.parser Shadowserver-Parser
intelmq     1622       1 63 13:26 ?        00:01:44 /usr/bin/python3 /usr/bin/intelmq.bots.experts.deduplicator.expert deduplicator-expert
intelmq     1627       1 50 13:26 ?        00:01:21 /usr/bin/python3 /usr/bin/intelmq.bots.experts.taxonomy.expert taxonomy-expert
intelmq     1632       1 50 13:26 ?        00:01:18 /usr/bin/python3 /usr/bin/intelmq.bots.experts.url.expert URL-Expert
intelmq     1639       1 31 13:26 ?        00:00:48 /usr/bin/python3 /usr/bin/intelmq.bots.experts.gethostbyname.expert Gethostbyname-Expert-2
intelmq     1648       1 34 13:26 ?        00:00:51 /usr/bin/python3 /usr/bin/intelmq.bots.experts.asn_lookup.expert ASNLookup-Expert
intelmq     1651       1  0 13:26 ?        00:00:00 /usr/bin/python3 /usr/bin/intelmq.bots.experts.cymru_whois.expert cymru-whois-expert

[ludoComp9]

[sebix]

I see no obvious reason so far, and it is hard to guess what the issue was. Thanks to the existing safeguard architecture of IntelMQ, we see no indications for a critical issue. The concept is, better raise a warning too much, than one too less.

[kamil-certat]

I would suggest looking into redis logs, and maybe verifying the bot before Cymru. Have you tried to re-create the Whois bot?