FIX/ENH: HttpMixin refactored and various fixes

**General**
Removed 'requests' MissingDependencyError, because requests is a core lib from intelmq
Removed HTTP variables from Bot class in favor of HttpMixin
Removed trying to import requests in pipeline, its a core lib from intelmq
Added additional configuration variables to HttpMixin ( from Bot class )

**Bots**
GitHub API is now using HttpMixin
MS Azure Collector is now using HttpMixin
DO-Portal Expert is now using HttpMixin
GeoHash using MissingDependencyError instead of ValueError (consistency)
HttpContentExpert is now using HttpMixin
HttpStatusExpert is now using HttpMixin
NationalCERTContactCertATExpert is now using HttpMixin
RDAPExpert is now using HttpMixin
RIPEExpert is now using HttpMixin
SplunkSavedSearchExpert is now using HttpMixin
TuencyExpert is now using HttpMixin
RestAPIOutput is now using HttpMixin

**Bot tests**
GitHub API Collector is now using requests_mock instead of MagicMock (consistency)
RestAPI Output is now using correct headers

Fixes #2150
Fixes #2137

Signed-off-by: Sebastian Waldbauer <[email protected]>

CHANGELOG.md

@@ -29,6 +29,12 @@ CHANGELOG
   The `LogLevel` and `ReturnType` Enums were added to `intelmq.lib.datatypes`.
 - `intelmq.lib.bot`:
   - Enhance behaviour if an unconfigured bot is started (PR#2054 by Sebastian Wagner).
+  - Remove `http_*` variables and moved them into HTTPMixin (PR#2151 by Sebastian Waldbauer, fixes #2150).
+  - Remove `set_request_parameters` in favor of HTTPMixin (PR#2151 by Sebastian Waldbauer).
+- `intelmq.lib.mixins.http`:
+  - Added missing variables types and simplified code (PR#2151 by Sebastian Waldbauer).
+- `intelmq.lib.pipeline`:
+  - Removed trying to import requests, its a core library specified in setup.py (PR#2151 by Sebastian Waldbauer).
 
 ### Development
 
@@ -39,6 +45,10 @@ CHANGELOG
 
 #### Collectors
 - `intelmq.bots.collectors.mail._lib`: Add support for unverified SSL/STARTTLS connections (PR#2055 by Sebastian Wagner).
+- `intelmq.bots.collectors.github_api`: Removed requests dependency in favor of HttpMixin (PR#2151 by Sebastian Waldbauer).
+- `intelmq.bots.collectors.collector_azure`: Added HttpMixin (PR#2151 by Sebastian Waldbauer).
+- `intelmq.bots.collectors.shodan.collector_stream`: Removed `set_request_paramters()` in favor of HttpMixin (PR#2151 by Sebastian Waldbauer).
+
 
 #### Parsers
 - `intelmq.bots.parsers.alienvault.parser_otx`: Save CVE data in `extra.cve` instead of `extra.CVE` due to the field name restriction on lower-case characters  (PR#2059 by Sebastian Wagner).
@@ -75,12 +85,21 @@ CHANGELOG
 - `intelmq.bots.experts.truncate_by_delimiter.expert`: Cut string if its length is higher than a maximum length (PR#1967 by Marius Karotkis).
 - `intelmq.bots.experts.remove_affix`: Remove prefix or postfix strings from a field (PR#1965 by Marius Karotkis).
 - `intelmq.bots.experts.asn_lookup.expert`: Fixes update-database script on the last few days of a month (PR#2121 by Filip Pokorný, fixes #2088).
+- `intelmq.bots.experts.do_portal`: Removed requests dependency in favor of HTTPMixin (PR#2151 by Sebastian Waldbauer).
+- `intelmq.bots.experts.http.*`: Using HTTPMixin instead of `create_request_session` (PR#2151 by Sebastian Waldbauer).
+- `intelmq.bots.experts.national_cert_contact_certat`: Using HttpMixin instead of `requests` library (PR#2151 by Sebastian Waldbauer).
+- `intelmq.bots.experts.rdap`: Removed requests dependency & `create_request_session` in favor of HTTPMixin (PR#2151 by Sebastian Waldbauer).
+- `intelmq.bots.experts.ripe`: Simplified code & uses HTTPMixin (PR#2151 by Sebastian Waldbauer).
+- `intelmq.bots.experts.splunk_saved_search`: Simplified & uses HTTPMixin (PR#2151 by Sebastian Waldbauer).
+- `intelmq.bots.experts.tuency`: Removed `create_request_session` in favor of HTTPMixin (PR#2151 by Sebastian Waldbauer).
 
 #### Outputs
 - Removed `intelmq.bots.outputs.postgresql`: this bot was marked as deprecated in 2019 announced to be removed in version 3 of IntelMQ (PR#2045 by Birger Schacht).
 - Added `intelmq.bots.outputs.rpz_file.output` to create RPZ files (PR#1962 by Marius Karotkis).
 - Added `intelmq.bots.outputs.bro_file.output` to create Bro intel formatted files (PR#1963 by Marius Karotkis).
 - `intelmq.bots.outputs.templated_smtp.output`: Add new function `from_json()` (which just calls `json.loads()` in the standard Python environment), meaning the Templated SMTP output bot can take strings containing JSON documents and do the formatting itself (PR#2120 by Karl-Johan Karlsson).
+- `intelmq.bots.outputs.elasticsearch`: Uses HttpMixin (PR#2151 by Sebastian Waldbauer).
+- `intelmq.bots.outputs.restapi`: Using HttpMixin instead of importing `requests` (PR#2151 by Sebastian Waldbauer).
 
 ### Documentation
 - Feeds: Add documentation for newly supported dataplane feeds, see above (PR#2102 by Mikk Margus Möll).
@@ -95,6 +114,7 @@ CHANGELOG
 - Also test on Python 3.10 (PR#2140 by Sebastian Wagner).
 - Switch from nosetests to pytest, as the former does not support Python 3.10 (PR#2140 by Sebastian Wagner).
 - CodeQL Github Actions `exponential backtracking on strings` fixed. (PR#2148 by Sebastian Waldbauer, fixes #2138)
+- Replaced `MagicMock` & `patch` with `requests_mock` (PR#2151 by Sebastian Waldbauer).
 
 ### Tools
 

intelmq/bots/collectors/github_api/_collector_github_api.py

@@ -7,13 +7,10 @@
 GITHUB API Collector bot
 """
 import base64
+from requests import exceptions
 
 from intelmq.lib.bot import CollectorBot
-
-try:
-    import requests
-except ImportError:
-    requests = None
+from intelmq.lib.mixins import HttpMixin
 
 static_params = {
     'headers': {
@@ -22,14 +19,11 @@
 }
 
 
-class GithubAPICollectorBot(CollectorBot):
+class GithubAPICollectorBot(CollectorBot, HttpMixin):
     basic_auth_username = None
     basic_auth_password = None
 
     def init(self):
-        if requests is None:
-            raise ValueError('Could not import requests. Please install it.')
-
         self.__user_headers = static_params['headers']
         if self.basic_auth_username is not None and self.basic_auth_password is not None:
             self.__user_headers.update(self.__produce_auth_header(self.basic_auth_username, self.basic_auth_password))
@@ -47,13 +41,13 @@ def process_request(self):
 
     def github_api(self, api_path: str, **kwargs) -> dict:
         try:
-            response = requests.get(f"{api_path}", params=kwargs, headers=self.__user_headers)
+            response = self.http_get(api_path, headers=self.__user_headers, params=kwargs)
             if response.status_code == 401:
                 # bad credentials
                 raise ValueError(response.json()['message'])
             else:
                 return response.json()
-        except requests.RequestException:
+        except exceptions.RequestException:
             raise ValueError(f"Unknown repository {api_path!r}.")
 
     @staticmethod

intelmq/bots/collectors/github_api/collector_github_contents_api.py

@@ -14,17 +14,14 @@
     'regex': file regex (DEFAULT = '*.json')
 """
 import re
+from requests import exceptions
 
 from intelmq.lib.exceptions import InvalidArgument
 from intelmq.bots.collectors.github_api._collector_github_api import GithubAPICollectorBot
+from intelmq.lib.mixins import HttpMixin
 
-try:
-    import requests
-except ImportError:
-    requests = None
 
-
-class GithubContentsAPICollectorBot(GithubAPICollectorBot):
+class GithubContentsAPICollectorBot(GithubAPICollectorBot, HttpMixin):
     "Collect files from a GitHub repository via the API. Optionally with GitHub credentials."
     regex: str = None  # TODO: could be re
     repository: str = None
@@ -62,7 +59,7 @@ def process_request(self):
                 if item['extra'] != {}:
                     report.add('extra.file_metadata', item['extra'])
                 self.send_message(report)
-        except requests.RequestException as e:
+        except exceptions.RequestException as e:
             raise ConnectionError(e)
 
     def __recurse_repository_files(self, base_api_url: str, extracted_github_files: list = None) -> list:
@@ -75,7 +72,7 @@ def __recurse_repository_files(self, base_api_url: str, extracted_github_files:
             elif github_file['type'] == 'file' and bool(re.search(self.regex, github_file['name'])):
                 extracted_github_file_data = {
                     'download_url': github_file['download_url'],
-                    'content': requests.get(github_file['download_url']).content,
+                    'content': self.http_get(github_file['download_url']).content,
                     'extra': {}
                 }
                 for field_name in self.__extra_fields:

intelmq/bots/collectors/mail/collector_mail_url.py

@@ -8,6 +8,7 @@
 """
 import io
 import re
+from requests import exceptions
 
 from intelmq.lib.mixins import HttpMixin
 from intelmq.lib.splitreports import generate_reports
@@ -50,7 +51,7 @@ def process_message(self, uid, message):
                 self.logger.info("Downloading report from %r.", url)
                 try:
                     resp = self.http_get(url)
-                except requests.exceptions.Timeout:
+                except exceptions.Timeout:
                     self.logger.error("Request timed out %i times in a row." %
                                       self.http_timeout_max_tries)
                     erroneous = True

intelmq/bots/collectors/microsoft/collector_azure.py

@@ -11,7 +11,7 @@
 
 from intelmq.lib.bot import CollectorBot
 from intelmq.lib.exceptions import MissingDependencyError
-from intelmq.lib.mixins import CacheMixin
+from intelmq.lib.mixins import CacheMixin, HttpMixin
 
 try:
     from azure.storage.blob import ContainerClient
@@ -23,7 +23,7 @@
     create_configuration = None  # noqa
 
 
-class MicrosoftAzureCollectorBot(CollectorBot, CacheMixin):
+class MicrosoftAzureCollectorBot(CollectorBot, CacheMixin, HttpMixin):
     "Fetch data blobs from a Microsoft Azure container"
     connection_string: str = "<insert your connection string here>"
     container_name: str = "<insert the container name>"

intelmq/bots/collectors/shodan/collector_stream.py

@@ -19,6 +19,7 @@
 from typing import List
 
 from intelmq.lib.bot import CollectorBot
+from intelmq.lib.mixins import HttpMixin
 
 try:
     import shodan
@@ -27,7 +28,7 @@
     shodan = None
 
 
-class ShodanStreamCollectorBot(CollectorBot):
+class ShodanStreamCollectorBot(CollectorBot, HttpMixin):
     "Collect the Shodan stream from the Shodan API"
     api_key: str = "<INSERT your API key>"
     countries: List[str] = []
@@ -36,7 +37,7 @@ def init(self):
         if shodan is None:
             raise ValueError("Library 'shodan' is needed but not installed.")
 
-        self.set_request_parameters()
+        self.setup()
         if tuple(int(v) for v in pkg_resources.get_distribution("shodan").version.split('.')) <= (1, 8, 1):
             if self.proxy:
                 raise ValueError('Proxies are given but shodan-python > 1.8.1 is needed for proxy support.')

intelmq/bots/experts/do_portal/expert.py

@@ -8,40 +8,26 @@
 a "502 Bad Gateway" status code is treated the same as a timeout,
 i.e. will be retried instead of a fail.
 """
-try:
-    import requests
-except ImportError:
-    requests = None
-
+from intelmq.lib.mixins import HttpMixin
 import intelmq.lib.utils as utils
 from intelmq.lib.bot import ExpertBot
 
 
-class DoPortalExpertBot(ExpertBot):
+class DoPortalExpertBot(ExpertBot, HttpMixin):
     """Retrieve abuse contact information for the source IP address from a do-portal instance"""
     mode: str = "append"
     portal_api_key: str = None
     portal_url: str = None
 
     def init(self):
-        if requests is None:
-            raise ValueError("Library 'requests' could not be loaded. Please install it.")
-
-        self.set_request_parameters()
-
         self.url = self.portal_url + '/api/1.0/ripe/contact?cidr=%s'
         self.http_header.update({
             "Content-Type": "application/json",
             "Accept": "application/json",
             "API-Authorization": self.portal_api_key
         })
 
-        self.session = utils.create_request_session(self)
-        retries = requests.urllib3.Retry.from_int(self.http_timeout_max_tries)
-        retries.status_forcelist = [502]
-        adapter = requests.adapters.HTTPAdapter(max_retries=retries)
-        self.session.mount('http://', adapter)
-        self.session.mount('https://', adapter)
+        self.session = self.http_session()
 
     def process(self):
         event = self.receive_message()

intelmq/bots/experts/geohash/expert.py

@@ -9,6 +9,7 @@
 https://github.com/joyanujoy/geolib
 '''
 from intelmq.lib.bot import ExpertBot
+from intelmq.lib.exceptions import MissingDependencyError
 
 try:
     from geolib import geohash
@@ -23,7 +24,7 @@ class GeohashExpertBot(ExpertBot):
 
     def init(self):
         if not geohash:
-            raise ValueError("Library 'geolib' is required, please install it.")
+            raise MissingDependencyError("geolib")
 
     def process(self):
         event = self.receive_message()

intelmq/bots/experts/http/expert_content.py

@@ -7,10 +7,10 @@
 from typing import List
 
 from intelmq.lib.bot import ExpertBot
-from intelmq.lib.utils import create_request_session
+from intelmq.lib.mixins import HttpMixin
 
 
-class HttpContentExpertBot(ExpertBot):
+class HttpContentExpertBot(ExpertBot, HttpMixin):
     """
     Test if a given string is part of the content for a given URL
 
@@ -29,8 +29,7 @@ class HttpContentExpertBot(ExpertBot):
     __session = None
 
     def init(self):
-        self.set_request_parameters()
-        self.__session = create_request_session(self)
+        self.__session = self.http_session()
 
     def process(self):
         event = self.receive_message()

intelmq/bots/experts/http/expert_status.py

@@ -8,9 +8,10 @@
 
 from intelmq.lib.bot import ExpertBot
 from intelmq.lib.utils import create_request_session
+from intelmq.lib.mixins import HttpMixin
 
 
-class HttpStatusExpertBot(ExpertBot):
+class HttpStatusExpertBot(ExpertBot, HttpMixin):
     """
     Fetch the HTTP Status for a given URL
 
@@ -31,8 +32,7 @@ def process(self):
         event = self.receive_message()
 
         if self.field in event:
-            self.set_request_parameters()
-            session = create_request_session(self)
+            session = self.http_session()
 
             try:
                 response = session.get(event[self.field])

intelmq/bots/experts/national_cert_contact_certat/expert.py

@@ -20,30 +20,22 @@
 """
 
 from intelmq.lib.bot import ExpertBot
+from intelmq.lib.mixins import HttpMixin
 from intelmq.lib.utils import create_request_session
 from intelmq.lib.exceptions import MissingDependencyError
 
-try:
-    import requests
-except ImportError:
-    requests = None
-
 
 URL = 'https://contacts.cert.at/cgi-bin/abuse-nationalcert.pl'
 
 
-class NationalCERTContactCertATExpertBot(ExpertBot):
+class NationalCERTContactCertATExpertBot(ExpertBot, HttpMixin):
     """Add country and abuse contact information from the CERT.at national CERT Contact Database. Set filter to true if you want to filter out events for Austria. Set overwrite_cc to true if you want to overwrite an existing country code value"""
     filter: bool = False
     http_verify_cert: bool = True
     overwrite_cc: bool = False
 
     def init(self):
-        if requests is None:
-            raise MissingDependencyError("requests")
-
-        self.set_request_parameters()
-        self.session = create_request_session(self)
+        self.session = self.http_session()
 
     def process(self):
         event = self.receive_message()

intelmq/bots/experts/rdap/expert.py

@@ -3,18 +3,13 @@
 # SPDX-License-Identifier: AGPL-3.0-or-later
 
 # -*- coding: utf-8 -*-
+import requests
 from intelmq.lib.bot import ExpertBot
 from intelmq.lib.utils import create_request_session
-from intelmq.lib.exceptions import MissingDependencyError
-from intelmq.lib.mixins import CacheMixin
+from intelmq.lib.mixins import CacheMixin, HttpMixin
 
-try:
-    import requests
-except ImportError:
-    requests = None
 
-
-class RDAPExpertBot(ExpertBot, CacheMixin):
+class RDAPExpertBot(ExpertBot, CacheMixin, HttpMixin):
     """ Get RDAP data"""
     rdap_order: list = ['abuse', 'technical', 'administrative', 'registrant', 'registrar']
     rdap_bootstrapped_servers: dict = {}
@@ -30,11 +25,7 @@ class RDAPExpertBot(ExpertBot, CacheMixin):
     __session: requests.Session
 
     def init(self):
-        if requests is None:
-            raise MissingDependencyError("requests")
-
-        self.set_request_parameters()
-        self.__session = create_request_session(self)
+        self.__session = self.http_session()
 
         # get overall rdap data from iana
         resp = self.__session.get('https://data.iana.org/rdap/dns.json')
@@ -73,7 +64,7 @@ def process(self):
             if result:
                 event.add('source.abuse_contact', result, overwrite=self.overwrite)
             else:
-                self.__session = create_request_session(self)
+                self.__session = self.http_session()
                 domain_parts = url.split('.')
                 domain_suffix = None
                 while domain_suffix is None: