Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shadowserver parser may fail to autodetect report type after reload #2360

Closed
kamil-certat opened this issue May 8, 2023 · 2 comments
Closed
Assignees
Labels
bug Indicates an unexpected problem or unintended behavior component: bots

Comments

@kamil-certat
Copy link
Contributor

Handling feed detection

During the init method the parser try to determine the feed type detection type [0]:

def init(self):
if self.feedname is not None:

If the self.feedname isn't set, then the mode is detect [1]:

else:
self._mode = 'detect'

Later, during processing a report, the feed name is detected and eventually set [2]:

if not retval:
raise ValueError('Could not get a config for {!r}, check the documentation.'
''.format(self.report_name))
self.feedname, self._sparser_config = retval

Reloading bot

Then, during the reload (not restart) happens, the bot is re-initialized without creating a new object:

intelmq/intelmq/lib/bot.py

Lines 271 to 285 in 7674949

def __handle_sighup(self):
"""
Handle SIGHUP.
"""
if not self.__sighup.is_set():
return False
self.logger.info('Handling SIGHUP, initializing again now.')
self.__disconnect_pipelines()
try:
self.shutdown() # disconnects, stops threads etc
except Exception:
self.logger.exception('Error during shutdown of bot.')
self.logger.handlers = [] # remove all existing handlers
self.__sighup.clear()
self.__init__(self.__bot_id_full, sighup_event=self.__sighup)

This mechanism is used during log rotation by our standard Debian configuration:

sudo -u intelmq /usr/local/bin/intelmqctl --quiet reload

and of course re-load the configuration, refreshing all parameters set in the config file [3]:

intelmq/intelmq/lib/bot.py

Lines 786 to 788 in 7674949

for option, value in params.get('parameters', {}).items():
setattr(self, option, value)
self.__log_configuration_parameter("runtime", option, value)

Failing edge case

Requirements

  • Assuming you want parser to detect the feed, so you didn't set any value in parametrs.feedname in the parser config.
  • You have also installed our Debian package or manually configured log rotation with reloading.
  • Your IntelMQ instance is working long enough for log rotation to occur.

Case steps

  1. After starting bot, the mode is correctly detect [1], and then handling every report a correct feed is detected and set [2].
  2. The log rotation occurs, and the parser is reloaded.
  3. The configuration is refreshed, overriding all fields set in the config file [3].
  4. Bot init method is called once more and tries to detect mode, but self.feedname is still set with the last value before reload, and the fixed mode is detected [0].
  5. On every next report handled, the last previously detected feed is used

Workarounds

Set the feedname as null in the config. However, due to the IntelMQ Manager issue with nulls certtools/intelmq-manager#294 this may be changed to an empty string and case the fail again.

Fixes

  • The quickest - clean the feedname in bot's shutdown method. If the config has a value, it will be set.

However, this issue can happen to any bot relying on default values and modifying their config. Probably it should be later solved on the IntelMQ library level.

@elsif2
Copy link
Collaborator

elsif2 commented May 8, 2023

Fix added to shadowserver-dynamic-config.

@kamil-certat
Copy link
Contributor Author

Could you cherrypick it to the main develop branch? I would like to push releasing patch soon.

@sebix sebix added bug Indicates an unexpected problem or unintended behavior component: bots labels May 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Indicates an unexpected problem or unintended behavior component: bots
Projects
None yet
Development

No branches or pull requests

3 participants