Monitor: Loading referenced configuration files causes 500 fails if file does not exist

[ghost]

The Monitor page of the Manager allows to load files, which are referenced by bots' parameters. If such a file doe not exist, the API throws an exception as this stacktrace from an Apache server error log shows:

[Tue Aug 31 17:53:28.161856 2021] [wsgi:error] [pid 28511] [client 127.0.0.1:34008] mod_wsgi (pid=28511): Exception occurred processing WSGI script '/usr/local/lib/python3.8/site-packages/intelmq_api/intelmq-api.wsgi'., referer: http://localhost/intelmq-manager/monitor.html?bot_id=bitsight-output
[Tue Aug 31 17:53:28.166546 2021] [wsgi:error] [pid 28511] [client 127.0.0.1:34008] Traceback (most recent call last):, referer: http://localhost/intelmq-manager/monitor.html?bot_id=bitsight-output
[Tue Aug 31 17:53:28.166822 2021] [wsgi:error] [pid 28511] [client 127.0.0.1:34008]   File "/usr/local/lib/python3.8/site-packages/intelmq_api/intelmq-api.wsgi", line 12, in application, referer: http://localhost/intelmq-manager/monitor.html?bot_id=bitsight-output
[Tue Aug 31 17:53:28.166846 2021] [wsgi:error] [pid 28511] [client 127.0.0.1:34008]     return __hug_wsgi__(environ, start_response), referer: http://localhost/intelmq-manager/monitor.html?bot_id=bitsight-output
[Tue Aug 31 17:53:28.166868 2021] [wsgi:error] [pid 28511] [client 127.0.0.1:34008]   File "/usr/lib/python3.8/site-packages/hug/api.py", line 502, in api_auto_instantiate, referer: http://localhost/intelmq-manager/monitor.html?bot_id=bitsight-output
[Tue Aug 31 17:53:28.166876 2021] [wsgi:error] [pid 28511] [client 127.0.0.1:34008]     return module.__hug_wsgi__(*args, **kwargs), referer: http://localhost/intelmq-manager/monitor.html?bot_id=bitsight-output
[Tue Aug 31 17:53:28.166903 2021] [wsgi:error] [pid 28511] [client 127.0.0.1:34008]   File "/usr/lib/python3.8/site-packages/falcon/api.py", line 269, in __call__, referer: http://localhost/intelmq-manager/monitor.html?bot_id=bitsight-output
[Tue Aug 31 17:53:28.166923 2021] [wsgi:error] [pid 28511] [client 127.0.0.1:34008]     responder(req, resp, **params), referer: http://localhost/intelmq-manager/monitor.html?bot_id=bitsight-output
[Tue Aug 31 17:53:28.166942 2021] [wsgi:error] [pid 28511] [client 127.0.0.1:34008]   File "/usr/lib/python3.8/site-packages/hug/api.py", line 355, in version_router, referer: http://localhost/intelmq-manager/monitor.html?bot_id=bitsight-output
[Tue Aug 31 17:53:28.166958 2021] [wsgi:error] [pid 28511] [client 127.0.0.1:34008]     versions.get(request_version or False, versions.get(None, not_found))(, referer: http://localhost/intelmq-manager/monitor.html?bot_id=bitsight-output
[Tue Aug 31 17:53:28.167036 2021] [wsgi:error] [pid 28511] [client 127.0.0.1:34008]   File "/usr/lib/python3.8/site-packages/hug/interface.py", line 947, in __call__, referer: http://localhost/intelmq-manager/monitor.html?bot_id=bitsight-output
[Tue Aug 31 17:53:28.167050 2021] [wsgi:error] [pid 28511] [client 127.0.0.1:34008]     raise exception, referer: http://localhost/intelmq-manager/monitor.html?bot_id=bitsight-output
[Tue Aug 31 17:53:28.167073 2021] [wsgi:error] [pid 28511] [client 127.0.0.1:34008]   File "/usr/lib/python3.8/site-packages/hug/interface.py", line 918, in __call__, referer: http://localhost/intelmq-manager/monitor.html?bot_id=bitsight-output
[Tue Aug 31 17:53:28.167085 2021] [wsgi:error] [pid 28511] [client 127.0.0.1:34008]     self.call_function(input_parameters), context, request, response, **kwargs, referer: http://localhost/intelmq-manager/monitor.html?bot_id=bitsight-output
[Tue Aug 31 17:53:28.167114 2021] [wsgi:error] [pid 28511] [client 127.0.0.1:34008]   File "/usr/lib/python3.8/site-packages/hug/interface.py", line 840, in call_function, referer: http://localhost/intelmq-manager/monitor.html?bot_id=bitsight-output
[Tue Aug 31 17:53:28.167126 2021] [wsgi:error] [pid 28511] [client 127.0.0.1:34008]     return self.interface(**parameters), referer: http://localhost/intelmq-manager/monitor.html?bot_id=bitsight-output
[Tue Aug 31 17:53:28.167145 2021] [wsgi:error] [pid 28511] [client 127.0.0.1:34008]   File "/usr/lib/python3.8/site-packages/hug/interface.py", line 129, in __call__, referer: http://localhost/intelmq-manager/monitor.html?bot_id=bitsight-output
[Tue Aug 31 17:53:28.167157 2021] [wsgi:error] [pid 28511] [client 127.0.0.1:34008]     return __hug_internal_self._function(*args, **kwargs), referer: http://localhost/intelmq-manager/monitor.html?bot_id=bitsight-output
[Tue Aug 31 17:53:28.167175 2021] [wsgi:error] [pid 28511] [client 127.0.0.1:34008]   File "/usr/local/lib/python3.8/site-packages/intelmq_api/api.py", line 177, in config, referer: http://localhost/intelmq-manager/monitor.html?bot_id=bitsight-output
[Tue Aug 31 17:53:28.167184 2021] [wsgi:error] [pid 28511] [client 127.0.0.1:34008]     result = file_access.load_file_or_directory(file, fetch), referer: http://localhost/intelmq-manager/monitor.html?bot_id=bitsight-output
[Tue Aug 31 17:53:28.167216 2021] [wsgi:error] [pid 28511] [client 127.0.0.1:34008]   File "/usr/local/lib/python3.8/site-packages/intelmq_api/files.py", line 75, in load_file_or_directory, referer: http://localhost/intelmq-manager/monitor.html?bot_id=bitsight-output
[Tue Aug 31 17:53:28.167224 2021] [wsgi:error] [pid 28511] [client 127.0.0.1:34008]     stat = path.stat(), referer: http://localhost/intelmq-manager/monitor.html?bot_id=bitsight-output
[Tue Aug 31 17:53:28.167238 2021] [wsgi:error] [pid 28511] [client 127.0.0.1:34008]   File "/usr/lib64/python3.8/pathlib.py", line 1198, in stat, referer: http://localhost/intelmq-manager/monitor.html?bot_id=bitsight-output
[Tue Aug 31 17:53:28.167253 2021] [wsgi:error] [pid 28511] [client 127.0.0.1:34008]     return self._accessor.stat(self), referer: http://localhost/intelmq-manager/monitor.html?bot_id=bitsight-output
[Tue Aug 31 17:53:28.167290 2021] [wsgi:error] [pid 28511] [client 127.0.0.1:34008] FileNotFoundError: [Errno 2] No such file or directory: '/opt/intelmq/var/lib/bots/file-output/bitsight.txt', referer: http://localhost/intelmq-manager/monitor.html?bot_id=bitsight-output

The exception is thrown here:

intelmq-api/intelmq_api/files.py

Line 75 in 9f8513b

stat = path.stat()

If the file does not exist, probably nothing should be returned.