Merge branch 'certtools:develop' into develop

aaronkaplan · Feb 16, 2024 · a4f6fc7 · a4f6fc7
2 parents 97df165 + ac6aa4e
commit a4f6fc7
Show file tree

Hide file tree

Showing 44 changed files with 185 additions and 2,153 deletions.
diff --git a/.github/workflows/build-docs.yml b/.github/workflows/build-docs.yml
@@ -45,6 +45,10 @@ jobs:
           git config user.name intelmq-bot
           git config user.email intelmq-bot
 
+      - name: "Fix images path"
+        run: |
+          sed -i 's|docs/static/images/|static/images/|' docs/index.md
+
       - name: "Build docs without publishing"
         if: github.event_name == 'pull_request'
         run: |

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -34,6 +34,8 @@
   (PR#2408 and PR#2414 by Jan Kaliszewski).
 - `intelmq.lib.upgrades`: Replace deprecated instances of `url2fqdn` experts by the new `url` expert in runtime configuration (PR#2432 by Sebastian Wagner).
 - `intelmq.lib.bot`: Ensure closing log files on reloading (PR#2435 by Kamil Mankowski).
+- AMQP Pipeline: fix SSL context to pointing to create a client-side connection that verifies the server (PR by Kamil Mankowski).
+- Only load the config once when starting intelmqctl (which makes IntelMQ API calls take less time) (PR#2444 by DigitalTrustCenter).
 
 ### Development
 - Makefile: Add codespell and test commands (PR#2425 by Sebastian Wagner).
@@ -61,8 +63,20 @@
   - Remove undocumented and unused attributes of `StompCollectorBot` instances:
     `ssl_ca_cert`, `ssl_cl_cert`, `ssl_cl_cert_key`.
   - Minor fixes/improvements and some refactoring (see also above: *Core*...).
+- `intelmq.bots.collectors.amqp`: fix SSL context to pointing to create a client-side connection that verifies the server (PR by Kamil Mankowski).
+- `intelmq.bots.collectors.shadowserver.collector_reports_api`:
+  - The 'json' option is no longer supported as the 'csv' option provides better performance (PR#2372 by elsif2).
+- `intelmq.bots.collectors.alienvault_otx.collector` (PR#2449 by qux-bbb):
+  - Fix modified_pulses_only is always False.
+
 
 #### Parsers
+- `intelmq.bots.parsers.netlab_360.parser`: Removed as the feed is discontinued. (#2442 by Filip Pokorný)
+- `intelmq.bots.parsers.webinspektor.parser`: Removed as the feed is discontinued. (#2442 by Filip Pokorný)
+- `intelmq.bots.parsers.sucuri.parser`: Removed as the feed is discontinued. (#2442 by Filip Pokorný)
+- `intelmq.bots.parsers.shadowserver._config`:
+  - Switch to dynamic configuration to decouple report schema changes from IntelMQ releases by regularly downloading them from the Shadowserver server (PR#2372 by elsif2).
+- `intelmq.bots.parsers.cymru`: Save current line. (PR by Kamil Mankowski)
 
 #### Experts
 - `intelmq.bots.experts.jinja` (PR#2417 by Mikk Margus Möll):
@@ -90,13 +104,16 @@
   - Try to reconnect on `NotConnectedException`.
 - `intelmq.bots.outputs.smtp_batch.output` (PR #2439 by Edvard Rejthar):
   - Fix ability to send with the default `bcc`
+- `intelmq.bots.outputs.amqp`: fix SSL context to pointing to create a client-side connection that verifies the server (PR by Kamil Mankowski).
 
 ### Documentation
 - Add a readthedocs configuration file to fix the build fail (PR#2403 by Sebastian Wagner).
 - Add a guide of developing extensions packages (PR#2413 by Kamil Mankowski)
 - Update/fix/improve the stuff related to the STOMP bots and integration with the *n6*'s
   Stream API (PR#2408 and PR#2414 by Jan Kaliszewski).
 - Complete documentation overhaul. Change to markdown format. Uses the mkdocs-material (PR#2419 by Filip Pokorný).
+- Adds warning banner if not browsing the latest version of the docs (PR#2445 by Filip Pokorný).
+- Fix logo path in index.md when building the docs (PR#2445 by Filip Pokorný).
 
 ### Packaging
 - Add `pendulum` to suggested packages, as it is required for the sieve bot (PR#2424 by Sebastian Wagner).
@@ -156,13 +173,11 @@
   - added support for `Subject NOT LIKE` queries,
   - added support for multiple values in ticket subject queries.
 - `intelmq.bots.collectors.rsync`: Support for optional private key, relative time parsing for the source path, extra rsync parameters and strict host key checking (PR#2241 by Mateo Durante).
-- `intelmq.bots.collectors.shadowserver.collector_reports_api`:
-  - The 'json' option is no longer supported as the 'csv' option provides better performance.
 
 #### Parsers
 - `intelmq.bots.parsers.shadowserver._config`:
   - Reset detected `feedname` at shutdown to re-detect the feedname on reloads (PR#2361 by @elsif2, fixes #2360).
-  - Switch to dynamic configuration to decouple report schema changes from IntelMQ releases. 
+  - Switch to dynamic configuration to decouple report schema changes from IntelMQ releases.
   - Added 'IPv6-Vulnerable-Exchange' alias and 'Accessible-WS-Discovery-Service' report. (PR#2338)
   - Removed unused `p0f_genre` and `p0f_detail` from the 'DNS-Open-Resolvers' report. (PR#2338)
   - Added 'Accessible-SIP' report. (PR#2348)

diff --git a/debian/cron.d/intelmq-update-database b/debian/cron.d/intelmq-update-database
@@ -3,14 +3,16 @@
 #
 # m h  dom mon dow      command
 # Update database for tor_nodes bot, update frequency is unknown:
-11  0 * * *     intelmq   intelmq.bots.experts.tor_nodes.expert --update-database
+11  0 * * *         intelmq     intelmq.bots.experts.tor_nodes.expert --update-database
 # Update database for maxmind_geoip bot, updated every Tuesday:
-30 23 * * 2,5     intelmq   intelmq.bots.experts.maxmind_geoip.expert --update-database
+30 23 * * 2,5       intelmq     intelmq.bots.experts.maxmind_geoip.expert --update-database
 # Update database for asn_lookup bot, updated every two hours:
-5 */2 * * *     intelmq   intelmq.bots.experts.asn_lookup.expert --update-database
+5 */2 * * *         intelmq     intelmq.bots.experts.asn_lookup.expert --update-database
 # Update database for domain_suffix bot, updated daily:
-45 1 * * *     intelmq   intelmq.bots.experts.domain_suffix.expert --update-database
+45 1 * * *          intelmq     intelmq.bots.experts.domain_suffix.expert --update-database
 # Update database for recordedfuture_iprisk bot, update frequency is unknown:
-27 1 * * *      intelmq  intelmq.bots.experts.recordedfuture_iprisk.expert --update-database
+27 1 * * *          intelmq     intelmq.bots.experts.recordedfuture_iprisk.expert --update-database
 # Update database for domain_valid bot, updated daily:
-50 1 * * *     intelmq   intelmq.bots.experts.domain_valid.expert --update-database
+50 1 * * *          intelmq     intelmq.bots.experts.domain_valid.expert --update-database
+# Update schema for shadowserver parser bot:
+02 1 * * *          intelmq     intelmq.bots.parsers.shadowserver.parser --update-schema
diff --git a/docs/_overrides/main.html b/docs/_overrides/main.html
@@ -0,0 +1,11 @@
+{#SPDX-FileCopyrightText: 2023 Filip Pokorný#}
+{#SPDX-License-Identifier: AGPL-3.0-or-later#}
+
+{% extends "base.html" %}
+
+{% block outdated %}
+  You're not viewing the latest version.
+  <a href="{{ '../' ~ base_url }}" style="color: black">
+    <strong>Click here to go to latest.</strong>
+  </a>
+{% endblock %}
diff --git a/docs/index.md b/docs/index.md
@@ -7,7 +7,7 @@
 [![CII Badge](https://bestpractices.coreinfrastructure.org/projects/4186/badge)](https://bestpractices.coreinfrastructure.org/projects/4186/)
 -->
 
-![IntelMQ](/docs/static/images/Logo_Intel_MQ.svg)
+![IntelMQ](docs/static/images/Logo_Intel_MQ.svg)
 
 # Introduction
 

diff --git a/docs/user/bots.md b/docs/user/bots.md
@@ -930,8 +930,7 @@ oldest files available!).
 
 The resulting reports contain the following special field:
 
-- `extra.file_name`: The name of the downloaded file, with fixed filename extension. The API returns file names with the
-  extension `.csv`, although the files are JSON, not CSV. Therefore, for clarity and better error detection in the parser, the file name in `extra.file_name` uses `.json` as extension.
+- `extra.file_name`: The name of the downloaded file, with fixed filename extension. 
 
 **Module:** `intelmq.bots.collectors.shadowserver.collector_reports_api`
 
@@ -951,7 +950,7 @@ The resulting reports contain the following special field:
 
 **`types`**
 
-(optional, string/array of strings) An array of strings (or a list of comma-separated values) with the names of report types you want to process. If you leave this empty, all the available reports will be downloaded and processed (i.e. 'scan', 'drones', 'intel', 'sandbox_connection', 'sinkhole_combined'). The possible report types are equivalent to the file names given in the section Supported Reports of the [Shadowserver parser](#intelmq.bots.parsers.shadowserver.parser).
+(optional, string/array of strings) An array of strings (or a list of comma-separated values) with the names of report types you want to process. If you leave this empty, all the available reports will be downloaded and processed (i.e. 'scan', 'drones', 'intel', 'sandbox_connection', 'sinkhole_combined'). The possible report types are equivalent to the file names defined the the schema.  Please see the Supported Reports of the [Shadowserver parser](#intelmq.bots.parsers.shadowserver.parser) for details.
 
 **Sample configuration**
 
@@ -2128,23 +2127,23 @@ correct mapping of the columns:
 
 1. **Automatic report type detection**
 
-   Since IntelMQ version 2.1 the parser can detect the feed based on metadata provided by the collector.
-
-   When processing a report, this bot takes `extra.file_name` from the report and looks in `config.py` how the report
-   should be parsed. If this lookup is not possible, and the `feedname` is not given as parameter, the feed cannot be
-   parsed.
-
-   The field `extra.file_name` has the following structure: `%Y-%m-%d-${report_name}[-suffix].csv` where the optional
-   suffix can be something like `country-geo`. For example, some possible filenames
-   are `2019-01-01-scan_http-country-geo.csv` or `2019-01-01-scan_tftp.csv`. The important part is the `report_name`,
-   between the date and the suffix. Since version 2.1.2 the date in the filename is optional, so filenames
-   like `scan_tftp.csv` are also detected.
+    Since IntelMQ version 2.1 the parser can detect the feed based on metadata provided by the collector.
+    
+    When processing a report, this bot takes `extra.file_name` from the report and looks in `config.py` how the report
+    should be parsed. If this lookup is not possible, and the `feedname` is not given as parameter, the feed cannot be
+    parsed.
+    
+    The field `extra.file_name` has the following structure: `%Y-%m-%d-${report_name}[-suffix].csv` where the optional
+    suffix can be something like `country-geo`. For example, some possible filenames
+    are `2019-01-01-scan_http-country-geo.csv` or `2019-01-01-scan_tftp.csv`. The important part is the `report_name`,
+    between the date and the suffix. Since version 2.1.2 the date in the filename is optional, so filenames
+    like `scan_tftp.csv` are also detected.
 
 2. **Fixed report type**
 
-   If the method above is not possible and for upgraded instances, the report type can be set with the `feedname`
-   parameter. Report type is derived from the subject of Shadowserver e-mails. A list of possible values of
-   the `feedname` parameter can be found in the table below in the column "Report Type".
+    If the method above is not possible and for upgraded instances, the report type can be set with the `feedname`
+    parameter. Report type is derived from the subject of Shadowserver e-mails. A list of possible values of
+    the `feedname` parameter can be found in the table below in the column "Report Type".
 
 **Module:**
 
@@ -2154,7 +2153,9 @@ correct mapping of the columns:
 
 **`feedname`**
 
-(optional, string) Name of the Shadowserver report, see list below for possible values.
+(optional, string) Name of the Shadowserver report. The value for each report type can be found in the schema `feed_name` field.
+
+For example using `curl -s https://interchange.shadowserver.org/intelmq/v1/schema | jq .[].feed_name`.
 
 **`overwrite`**
 
@@ -2200,6 +2201,7 @@ The schema revision history is maintained at https://github.com/The-Shadowserver
       auto_update: true
     run_mode: continuous
 ```
+
 ---
 
 ### Shodan <div id="intelmq.bots.parsers.shodan.parser" />

diff --git a/docs/user/feeds.md b/docs/user/feeds.md
@@ -1899,101 +1899,6 @@ module: intelmq.bots.parsers.microsoft.parser_ctip
 ---
 
 
-## Netlab 360
-
-### DGA
-
-This feed lists DGA family, Domain, Start and end of valid time(UTC) of a number of DGA families.
-
-**Public:** yes
-
-**Revision:** 2018-01-20
-
-**Documentation:** <http://data.netlab.360.com/dga>
-
-
-**Collector configuration**
-
-```yaml
-module: intelmq.bots.collectors.http.collector_http
-parameters:
-  http_url: http://data.netlab.360.com/feeds/dga/dga.txt
-  name: DGA
-  provider: Netlab 360
-  rate_limit: 3600
-```
-
-**Parser configuration**
-
-```yaml
-module: intelmq.bots.parsers.netlab_360.parser
-```
-
----
-
-
-### Hajime Scanner
-
-This feed lists IP address for know Hajime bots network. These IPs data are obtained by joining the DHT network and interacting with the Hajime node
-
-**Public:** yes
-
-**Revision:** 2019-08-01
-
-**Documentation:** <https://data.netlab.360.com/hajime/>
-
-
-**Collector configuration**
-
-```yaml
-module: intelmq.bots.collectors.http.collector_http
-parameters:
-  http_url: https://data.netlab.360.com/feeds/hajime-scanner/bot.list
-  name: Hajime Scanner
-  provider: Netlab 360
-  rate_limit: 3600
-```
-
-**Parser configuration**
-
-```yaml
-module: intelmq.bots.parsers.netlab_360.parser
-```
-
----
-
-
-### Magnitude EK
-
-This feed lists FQDN and possibly the URL used by Magnitude Exploit Kit. Information also includes the IP address used for the domain and last time seen.
-
-**Public:** yes
-
-**Revision:** 2018-01-20
-
-**Documentation:** <http://data.netlab.360.com/ek>
-
-
-**Collector configuration**
-
-```yaml
-module: intelmq.bots.collectors.http.collector_http
-parameters:
-  http_url: http://data.netlab.360.com/feeds/ek/magnitude.txt
-  name: Magnitude EK
-  provider: Netlab 360
-  rate_limit: 3600
-```
-
-**Parser configuration**
-
-```yaml
-module: intelmq.bots.parsers.netlab_360.parser
-```
-
----
-
-
 ## OpenPhish
 
 ### Premium Feed
@@ -2495,41 +2400,6 @@ module: intelmq.bots.parsers.github_feed
 ---
 
 
-## Sucuri
-
-### Hidden IFrames
-
-Latest hidden iframes identified on compromised web sites.
-
-**Public:** yes
-
-**Revision:** 2018-01-28
-
-**Documentation:** <http://labs.sucuri.net/?malware>
-
-**Additional Information:** Please note that the parser only extracts the hidden iframes  and the conditional redirects, not the encoded javascript.
-
-
-**Collector configuration**
-
-```yaml
-module: intelmq.bots.collectors.http.collector_http
-parameters:
-  http_url: http://labs.sucuri.net/?malware
-  name: Hidden IFrames
-  provider: Sucuri
-  rate_limit: 86400
-```
-
-**Parser configuration**
-
-```yaml
-module: intelmq.bots.parsers.sucuri.parser
-```
-
----
-
-
 ## Surbl
 
 ### Malicious Domains
@@ -2917,37 +2787,6 @@ parameters:
 ---
 
 
-## WebInspektor
-
-### Unsafe sites
-
-Latest detected unsafe sites.
-
-**Public:** yes
-
-**Revision:** 2018-03-09
-
-
-**Collector configuration**
-
-```yaml
-module: intelmq.bots.collectors.http.collector_http
-parameters:
-  http_url: https://app.webinspector.com/public/recent_detections/
-  name: Unsafe sites
-  provider: WebInspektor
-  rate_limit: 60
-```
-
-**Parser configuration**
-
-```yaml
-module: intelmq.bots.parsers.webinspektor.parser
-```
-
----
-
-
 ## ZoneH
 
 ### Defacements