From f7b67fdd47287646fbee2fd56a1bc1a2d86fa0dd Mon Sep 17 00:00:00 2001 From: intelmq-bot Date: Fri, 3 Jan 2025 11:12:44 +0000 Subject: [PATCH] Deployed 0678d0f to develop with MkDocs 1.6.1 and mike 2.1.3 --- develop/dev/release/index.html | 2 +- develop/search/search_index.json | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/develop/dev/release/index.html b/develop/dev/release/index.html index ec6dfb169..cc0d972b6 100644 --- a/develop/dev/release/index.html +++ b/develop/dev/release/index.html @@ -11,7 +11,7 @@ body[data-md-color-scheme="slate"] .gslide-desc { color: var(--md-default-fg-color);}
Skip to content

Release procedure

General assumption: You are working on branch maintenance, the next version is a bug fix release. For feature releases it is slightly different.

Check before

  • Make sure the current state is really final ;) You can test most of the steps described here locally before doing it real.
  • Check the upgrade functions in intelmq/lib/upgrades.py.
  • Close the milestone on GitHub and move any open issues to the next one.
  • docs/admin/installation/linux-packages.md: Update supported operating systems.

Documentation

These apply to all projects:

  • CHANGELOG.MD and NEWS.MD: Update the latest header, fix the order, remove empty sections and (re)group the entries if necessary.
  • debian/changelog: Insert a new section for the new version with the tool dch or update the version of the existing last item if yet unreleased. Don't forget the revision after the version number!

IntelMQ

  • intelmq/version.py: Update the version.

Eventually adapt the default log levels if necessary. Should be INFO for stable releases.

IntelMQ API

  • intelmq_api/version.py: Update the version.

IntelMQ Manager

  • intelmq_manager/version.py: Update the version.
  • intelmq_manager/static/js/about.js: Update the version.

Commit, push, review and merge

Commit your changes in a separate branch, the final commit message should start with REL:. Push and create a pull request to the develop branch. Someone else should review the changes. Eventually fix them, make sure the REL: is the last commit, you can also push that one at last, after the reviews.

Why a separate branch? Because if problems show up, you can still force-push to that one, keeping the release commit the latest one.

Tag and release

Tag the commit with git tag -s version HEAD, merge it into develop, push the branches and the tag. The tag is just a.b.c, not prefixed with v (that was necessary only with SVN a long time ago...).

Go to https://github.com/certtools/intelmq/tags and enter the release notes (from the CHANGELOG) for the new tag, then it's considered a release by GitHub.

Tarballs and PyPI

  • Build the source and binary (wheel) distribution:
rm -r build/
+--> 
Release procedure
 
General assumption: You are working on branch maintenance, the next version is a bug fix release. For feature releases it is slightly different.
 
Check before
 
     
  • Make sure the current state is really final ;) You can test most of the steps described here locally before doing it real.
    •  
  • Check the upgrade functions in intelmq/lib/upgrades.py.
    •  
  • Close the milestone on GitHub and move any open issues to the next one.
    •  
  • docs/admin/installation/linux-packages.md: Update supported operating systems.
    •  
 
Documentation
 
These apply to all projects:
 
     
  • CHANGELOG.MD and NEWS.MD: Update the latest header, fix the order, remove empty sections and (re)group the entries if necessary.
    •  
  • debian/changelog: Insert a new section for the new version with the tool dch or update the version of the existing last item if yet unreleased. Don't forget the revision after the version number!
    •  
 
IntelMQ
 
     
  • intelmq/version.py: Update the version.
    •  
 
IntelMQ API
 
     
  • intelmq_api/version.py: Update the version.
    •  
 
IntelMQ Manager
 
     
  • intelmq_manager/version.py: Update the version.
    •  
  • intelmq_manager/static/js/about.js: Update the version.
    •  
 
Commit, push, review and merge
 
Commit your changes in a separate branch, the final commit message should start with REL:. Push and create a pull request to the develop branch. Someone else should review the changes. Eventually fix them, make sure the REL: is the last commit, you can also push that one at last, after the reviews.
 
Why a separate branch? Because if problems show up, you can still force-push to that one, keeping the release commit the latest one.
 
Tag and release
 
Tag the commit with git tag -s version HEAD, merge it into develop, push the branches and the tag. The tag is just a.b.c, not prefixed with v (that was necessary only with SVN a long time ago...).
 
Go to https://github.com/certtools/intelmq/tags and enter the release notes (from the CHANGELOG) for the new tag, then it's considered a release by GitHub.
 
Tarballs and PyPI
 
     
  • Build the source and binary (wheel) distribution:
    •  
 
rm -r build/
 python3 setup.py sdist bdist_wheel
 
 
     
  • Upload the files including signatures to PyPI with e.g. twine: twine upload -u __token__ -p $APITOKEN dist/intelmq... (or set the API Token in .pypirc).
    •  
 
Documentation
 
Since using mkdocs (see https://docs.intelmq.org) nothing needs to be done anymore.
 
Packages
 
We are currently using the public Open Build Service instance of openSUSE: http://build.opensuse.org/project/show/home:sebix:intelmq
 
First, test all the steps first with the unstable-repository and check that at least installations succeed.
 
     
  • Create the tarballs with the script create-archives.sh.
    •  
  • Update the dsc and spec files for new filenames and versions.
    •  
  • Update the .changes file
    •  
  • Build locally for all distributions.
    •  
  • Commit.
    •  
 
Docker Image
 
Releasing a new Docker image is very easy.
 
     
  • Clone IntelMQ Docker Repository with git clone https://github.com/certat/intelmq-docker.git --recursive as this repository contains submodules
    •  
  • If the intelmq-docker repository is not updated yet, use git pull --recurse-submodules to pull the latest changes from their respective repository.
    •  
  • Run ./build.sh, check your console if the build was successful.
    •  
  • Run ./test.sh - It will run nosetests3 with the exotic flag. All errors/warnings will be displayed.
    •  
  • Change the build_version in publish.sh to the new version you want to release.
    •  
  • Change the namespace variable in publish.sh.
    •  
  • If no error/warning was shown, you can release with ./publish.sh.
    •  
  • Update the DockerHub ReadMe and add the latest version.
    •  
  • Commit and push the updates to the intelmq-docker repository
    •  
 
Announcements
 
Announce the new version at the mailinglists intelmq-users, intelmq-dev. For bigger releases, probably also at IHAP, Twitter, etc. Ask your favorite social media consultant.
 
Prepare new version
 
Increase the version in intelmq/version.py and declare it as alpha version. Add the new version in intelmq/lib/upgrades.py. Add a new entry in debian/changelog with dch -v [version] -c debian/changelog.
 
Add new entries to CHANGELOG.md and NEWS.md.
 
IntelMQ
 
For CHANGELOG.md:
 
### Configuration
 
diff --git a/develop/search/search_index.json b/develop/search/search_index.json
index 82cb55a02..692b7997b 100644
--- a/develop/search/search_index.json
+++ b/develop/search/search_index.json
@@ -1 +1 @@
-{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Introduction","text":""},{"location":"#introduction","title":"Introduction","text":"
IntelMQ is a solution for IT security teams (CERTs & CSIRTs, SOCs abuse departments, etc.) for collecting and processing security feeds (such as log files) using a message queuing protocol. It's a community driven initiative called IHAP1 (Incident Handling Automation Project) which was conceptually designed by European CERTs/CSIRTs during several InfoSec events. Its main goal is to give to incident responders an easy way to collect & process threat intelligence thus improving the incident handling processes of CERTs.
 
IntelMQ is frequently used for:
 
     
  • automated incident handling
    •  
  • situational awareness
    •  
  • automated notifications
    •  
  • as data collector for other tools
    •  
  • and more!
    •  
 
The design was influenced by AbuseHelper however it was re-written from scratch and aims at:
 
     
  • Reducing the complexity of system administration
    •  
  • Reducing the complexity of writing new bots for new data feeds
    •  
  • Reducing the probability of events lost in all process with persistence functionality (even system crash)
    •  
  • Use and improve the existing Data Harmonization Ontology
    •  
  • Use JSON format for all messages
    •  
  • Provide easy way to store data into databases and log collectors such as PostgreSQL, Elasticsearch and Splunk
    •  
  • Provide easy way to create your own black-lists
    •  
  • Provide easy communication with other systems via HTTP RESTful API
    •  
 
It follows the following basic meta-guidelines:
 
     
  • Don't break simplicity - KISS
    •  
  • Keep it open source - forever
    •  
  • Strive for perfection while keeping a deadline
    •  
  • Reduce complexity/avoid feature bloat
    •  
  • Embrace unit testing
    •  
  • Code readability: test with inexperienced programmers
    •  
  • Communicate clearly
    •  
"},{"location":"#contribute","title":"Contribute","text":"
     
  • Subscribe to the IntelMQ Developers mailing list and engage in discussions
    •  
  • Report any errors and suggest improvements via issues
    •  
  • Read the Developer Guide and open a pull request
    •  
 
     
  1.  
    Incident Handling Automation Project, mailing list: ihap@lists.trusted-introducer.org\u00a0\u21a9
     
    2.  
"},{"location":"changelog/","title":"Changelog","text":""},{"location":"changelog/#changelog","title":"CHANGELOG","text":""},{"location":"changelog/#332-unreleased","title":"3.3.2 (unreleased)","text":""},{"location":"changelog/#configuration","title":"Configuration","text":""},{"location":"changelog/#core","title":"Core","text":"
     
  • Python 3.8 or newer is required (PR#2541 by Sebastian Wagner).
    •  
"},{"location":"changelog/#development","title":"Development","text":""},{"location":"changelog/#data-format","title":"Data Format","text":""},{"location":"changelog/#bots","title":"Bots","text":""},{"location":"changelog/#collectors","title":"Collectors","text":"
     
  • intelmq.bots.collectors.shadowserver.collector_reports_api.py:
    •  
  • Fixed behaviour if parameter types value is empty string, behave the same way as not set, not like no type.
    •  
"},{"location":"changelog/#parsers","title":"Parsers","text":"
     
  • intelmq.bots.parsers.shadowserver._config:
    •  
  • fix error message formatting if schema file is absent (PR#2528 by Sebastian Wagner).
    •  
  • intelmq.bots.parsers.shadowserver.parser:
    •  
  • Fix to avoid schema download if not configured #2530.
    •  
"},{"location":"changelog/#experts","title":"Experts","text":"
     
  • intelmq.bots.experts.securitytxt:
    •  
  • Added new bot (PR#2538 by Frank Westers and Sebastian Wagner)
    •  
"},{"location":"changelog/#outputs","title":"Outputs","text":"
     
  • intelmq.bots.outputs.cif3.output:
    •  
  • The requirement can only be installed on Python version < 3.12.
    •  
  • Add a check on the Python version and exit if incompatible.
    •  
  • Add a deprecation warning (PR#2544 by Sebastian Wagner)
    •  
"},{"location":"changelog/#documentation","title":"Documentation","text":""},{"location":"changelog/#packaging","title":"Packaging","text":""},{"location":"changelog/#tests","title":"Tests","text":"
     
  • Install build dependencies for pymssql on Python 3.8 as there are no wheels available for this Python version (PR#2542 by Sebastian Wagner).
    •  
  • Install psql explicitly for workflow support on other platforms such as act (PR#2542 by Sebastian Wagner).
    •  
  • Create intelmq user & group if running privileged to allow dropping privileges (PR#2542 by Sebastian Wagner).
    •  
  • intelmq.tests.lib.test_pipeline.TestAmqp.test_acknowledge: Also skip on Python 3.11 besides on 3.8 when running on CI (PR#2542 by Sebastian Wagner).
    •  
"},{"location":"changelog/#tools","title":"Tools","text":""},{"location":"changelog/#contrib","title":"Contrib","text":""},{"location":"changelog/#known-issues","title":"Known issues","text":""},{"location":"changelog/#331-2024-09-03","title":"3.3.1 (2024-09-03)","text":""},{"location":"changelog/#core_1","title":"Core","text":"
     
  • intelmq.lib.utils.drop_privileges: When IntelMQ is called as root and dropping the privileges to user intelmq, also set the non-primary groups associated with the intelmq user. Makes the behaviour of running intelmqctl as root closer to the behaviour of sudo -u intelmq ... (PR#2507 by Mikk Margus M\u00f6ll).
    •  
  • intelmq.lib.utils.unzip: Ignore directories themselves when extracting data to prevent the extraction of empty data for a directory entries (PR#2512 by Kamil Mankowski).
    •  
"},{"location":"changelog/#bots_1","title":"Bots","text":""},{"location":"changelog/#collectors_1","title":"Collectors","text":"
     
  • intelmq.bots.collectors.shadowserver.collector_reports_api.py:
    •  
  • Added support for the types parameter to be either a string or a list (PR#2495 by elsif2).
    •  
  • Refactored to utilize the type field returned by the API to match the requested types instead of a sub-string match on the filename.
    •  
  • Fixed timezone issue for collecting reports (PR#2506 by elsif2).
    •  
  • Fixed behaviour if parameter reports value is empty string, behave the same way as not set, not like no report (PR#2523 by Sebastian Wagner).
    •  
  • intelmq.bots.collectors.shodan.collector_stream (PR#2492 by Mikk Margus M\u00f6ll):
    •  
  • Add alert parameter to Shodan stream collector to allow fetching streams by configured alert ID
    •  
  • intelmq.bots.collectors.mail._lib: Remove deprecated parameter attach_unzip from default parameters (PR#2511 by Sebastian Wagner).
    •  
"},{"location":"changelog/#parsers_1","title":"Parsers","text":"
     
  • intelmq.bots.parsers.shadowserver._config:
    •  
  • Fetch schema before first run (PR#2482 by elsif2, fixes #2480).
    •  
  • intelmq.bots.parsers.dataplane.parser: Use | as field delimiter, fix parsing of AS names including | (PR#2488 by DigitalTrustCenter).
    •  
  • all parsers: add copy_collector_provided_fields parameter allowing copying additional fields from the report, e.g. extra.file_name.   (PR#2513 by Kamil Mankowski).
    •  
"},{"location":"changelog/#experts_1","title":"Experts","text":"
     
  • intelmq.bots.experts.sieve.expert:
    •  
  • For :contains, =~ and !~, convert the value to string before matching avoiding an exception. If the value is a dict, convert the value to JSON (PR#2500 by Sebastian Wagner).
    •  
  • Add support for variables in Sieve scripts (PR#2514 by Mikk Margus M\u00f6ll, fixes #2486).
    •  
  • intelmq.bots.experts.filter.expert:
    •  
  • Treat value false for parameter filter_regex as false (PR#2499 by Sebastian Wagner).
    •  
"},{"location":"changelog/#outputs_1","title":"Outputs","text":"
     
  • intelmq.bots.outputs.misp.output_feed: Handle failures if saved current event wasn't saved or is incorrect (PR by Kamil Mankowski).
    •  
  • intelmq.bots.outputs.smtp_batch.output: Documentation on multiple recipients added (PR#2501 by Edvard Rejthar).
    •  
"},{"location":"changelog/#documentation_1","title":"Documentation","text":"
     
  • Bots: Clarify some section of Mail collectors and the Generic CSV Parser (PR#2510 by Sebastian Wagner).
    •  
"},{"location":"changelog/#known-issues_1","title":"Known Issues","text":"
This is short list of the most important known issues. The full list can be retrieved from GitHub. - intelmq.parsers.html_table may not process invalid URLs in patched Python version due to changes in urllib (#2382). - Breaking changes in 'rt' 3.0 library (#2367). - Type error with SQL output bot's prepare_values returning list instead of tuple (#2255). - intelmq_psql_initdb does not work for SQLite (#2202). - intelmqsetup: should install a default state file (#2175). - Misp Expert - Crash if misp event already exist (#2170). - Spamhaus CERT parser uses wrong field (#2165). - Custom headers ignored in HTTPCollectorBot (#2150). - intelmqctl log: parsing syslog does not work (#2097). - Bash completion scripts depend on old JSON-based configuration files (#2094). - Bots started with IntelMQ-API/Manager stop when the webserver is restarted (#952). - Corrupt dump files when interrupted during writing (#870).
"},{"location":"changelog/#330-2024-03-01","title":"3.3.0 (2024-03-01)","text":""},{"location":"changelog/#configuration_1","title":"Configuration","text":"
     
  • Add new optional configuration parameters for intelmq.bots.collectors.stomp.collector   and intelmq.bots.outputs.stomp.output (PR#2408 by Jan Kaliszewski):
    •  
  • auth_by_ssl_client_certificate (Boolean, default: true; if false then     ssl_client_certificate and ssl_client_certificate_key will be ignored);
    •  
  • username (STOMP authentication login, default: \"guest\"; to be used only     if auth_by_ssl_client_certificate is false);
    •  
  • password (STOMP authentication passcode, default: \"guest\"; to be used only     if auth_by_ssl_client_certificate is false).
    •  
  • Add the possibility to set the ssl_ca_certificate configuration parameter for   intelmq.bots.collectors.stomp.collector and/or intelmq.bots.outputs.stomp.output   to an empty string - which means that the SSL machinery used for STOMP communication   will attempt to load the system\u2019s default CA certificates (PR#2414 by Jan Kaliszewski).
    •  
"},{"location":"changelog/#core_2","title":"Core","text":"
     
  • intelmq.lib.message: For invalid message keys, add a hint on the failure to the exception: not allowed by configuration or not matching regular expression (PR#2398 by Sebastian Wagner).
    •  
  • intelmq.lib.exceptions.InvalidKey: Add optional parameter additional_text (PR#2398 by Sebastian Wagner).
    •  
  • Change the way we discover bots to allow easy extending based on the entry point name. (PR#2413 by Kamil Mankowski)
    •  
  • intelmq.lib.mixins: Add a new class, StompMixin (defined in a new submodule: stomp),   which provides certain common STOMP-bot-specific operations, factored out from   intelmq.bots.collectors.stomp.collector and intelmq.bots.outputs.stomp.output   (PR#2408 and PR#2414 by Jan Kaliszewski).
    •  
  • intelmq.lib.upgrades: Replace deprecated instances of url2fqdn experts by the new url expert in runtime configuration (PR#2432 by Sebastian Wagner).
    •  
  • intelmq.lib.bot: Ensure closing log files on reloading (PR#2435 by Kamil Mankowski).
    •  
  • AMQP Pipeline: fix SSL context to pointing to create a client-side connection that verifies the server (PR by Kamil Mankowski).
    •  
  • Only load the config once when starting intelmqctl (which makes IntelMQ API calls take less time) (PR#2444 by DigitalTrustCenter).
    •  
"},{"location":"changelog/#development_1","title":"Development","text":"
     
  • Makefile: Add codespell and test commands (PR#2425 by Sebastian Wagner).
    •  
"},{"location":"changelog/#data-format_1","title":"Data Format","text":""},{"location":"changelog/#bots_2","title":"Bots","text":""},{"location":"changelog/#collectors_2","title":"Collectors","text":"
     
  • intelmq.bots.collectors.stomp.collector (PR#2408 and PR#2414 by Jan Kaliszewski):
    •  
  • Drop support for versions of stomp.py older than 4.1.12.
    •  
  • Update the code to support new versions of stomp.py, including the latest (8.1.0);     fixes #2342.
    •  
  • Add support for authentication based on STOMP login and passcode, introducing three     new configuration parameters (see above: Configuration).
    •  
  • Add support for loading the system\u2019s default CA certificates, as an alternative to     specifying the CA certificate(s) file path explicitly (see above: Configuration).
    •  
  • Fix (by carefully targeted monkey patching) certain security problems caused by     SSL-related weaknesses that some versions of stomp.py suffer from.
    •  
  • Fix the reconnection behavior: do not attempt to reconnect after shutdown. Also,     never attempt to reconnect if the version of stomp.py is older than 4.1.21 (it     did not work properly anyway).
    •  
  • Add coercion of the port config parameter to int.
    •  
  • Add implementation of the check hook (verifying, in particular, accessibility     of necessary file(s)).
    •  
  • Remove undocumented and unused attributes of StompCollectorBot instances:     ssl_ca_cert, ssl_cl_cert, ssl_cl_cert_key.
    •  
  • Minor fixes/improvements and some refactoring (see also above: Core...).
    •  
  • intelmq.bots.collectors.amqp: fix SSL context to pointing to create a client-side connection that verifies the server (PR by Kamil Mankowski).
    •  
  • intelmq.bots.collectors.shadowserver.collector_reports_api:
    •  
  • The 'json' option is no longer supported as the 'csv' option provides better performance (PR#2372 by elsif2).
    •  
  • intelmq.bots.collectors.alienvault_otx.collector (PR#2449 by qux-bbb):
    •  
  • Fix modified_pulses_only is always False.
    •  
"},{"location":"changelog/#parsers_2","title":"Parsers","text":"
     
  • intelmq.bots.parsers.netlab_360.parser: Removed as the feed is discontinued. (#2442 by Filip Pokorn\u00fd)
    •  
  • intelmq.bots.parsers.webinspektor.parser: Removed as the feed is discontinued. (#2442 by Filip Pokorn\u00fd)
    •  
  • intelmq.bots.parsers.sucuri.parser: Removed as the feed is discontinued. (#2442 by Filip Pokorn\u00fd)
    •  
  • intelmq.bots.parsers.shadowserver._config:
    •  
  • Switch to dynamic configuration to decouple report schema changes from IntelMQ releases by regularly downloading them from the Shadowserver server (PR#2372 by elsif2).
    •  
  • intelmq.bots.parsers.cymru: Save current line. (PR by Kamil Mankowski)
    •  
"},{"location":"changelog/#experts_2","title":"Experts","text":"
     
  • intelmq.bots.experts.jinja (PR#2417 by Mikk Margus M\u00f6ll):
    •  
  • Add optional socket_perms and socket_group parameters to change     file permissions on socket file, if it is in use.
    •  
  • intelmq.bots.experts.ripe (PR#2461 by Mikk Margus M\u00f6ll):
    •  
  • Handle \"No abuse contact found for\" messages for non-ASN resources
    •  
"},{"location":"changelog/#outputs_2","title":"Outputs","text":"
     
  • intelmq.bots.outputs.stomp.output (PR#2408 and PR#2414 by Jan Kaliszewski):
    •  
  • Drop support for versions of stomp.py older than 4.1.12.
    •  
  • Update the code to support new versions of stomp.py, including the latest (8.1.0).
    •  
  • Add support for authentication based on STOMP login and passcode, introducing three     new configuration parameters (see above: Configuration).
    •  
  • Add support for loading the system\u2019s default CA certificates, as an alternative to     specifying the CA certificate(s) file path explicitly (see above: Configuration).
    •  
  • Fix (by carefully targeted monkey patching) certain security problems caused by     SSL-related weaknesses that some versions of stomp.py suffer from.
    •  
  • Fix AttributeError caused by attempts to get unset attributes of StompOutputBot     (ssl_ca_cert et consortes).
    •  
  • Add coercion of the port config parameter to int.
    •  
  • Add implementation of the check hook (verifying, in particular, accessibility     of necessary file(s)).
    •  
  • Add stomp.py version check (raise MissingDependencyError if not >=4.1.12).
    •  
  • Minor fixes/improvements and some refactoring (see also above: Core...).
    •  
  • intelmq.bots.outputs.stomp.output (PR#2423 by Kamil Mankowski):
    •  
  • Try to reconnect on NotConnectedException.
    •  
  • intelmq.bots.outputs.smtp_batch.output (PR #2439 by Edvard Rejthar):
    •  
  • Fix ability to send with the default bcc
    •  
  • intelmq.bots.outputs.amqp: fix SSL context to pointing to create a client-side connection that verifies the server (PR by Kamil Mankowski).
    •  
"},{"location":"changelog/#documentation_2","title":"Documentation","text":"
     
  • Add a readthedocs configuration file to fix the build fail (PR#2403 by Sebastian Wagner).
    •  
  • Add a guide of developing extensions packages (PR#2413 by Kamil Mankowski)
    •  
  • Update/fix/improve the stuff related to the STOMP bots and integration with the n6's   Stream API (PR#2408 and PR#2414 by Jan Kaliszewski).
    •  
  • Complete documentation overhaul. Change to markdown format. Uses the mkdocs-material (PR#2419 by Filip Pokorn\u00fd).
    •  
  • Adds warning banner if not browsing the latest version of the docs (PR#2445 by Filip Pokorn\u00fd).
    •  
  • Fix logo path in index.md when building the docs (PR#2445 by Filip Pokorn\u00fd).
    •  
"},{"location":"changelog/#packaging_1","title":"Packaging","text":"
     
  • Add pendulum to suggested packages, as it is required for the sieve bot (PR#2424 by Sebastian Wagner).
    •  
  • debian/control: in Suggests field, replace python3-stomp.py (>= 4.1.9) with   python3-stomp (>= 4.1.12), i.e., fix the package name by removing the .py   suffix and bump the minimum version to 4.1.12 (PR#2414 by Jan Kaliszewski).
    •  
"},{"location":"changelog/#tests_1","title":"Tests","text":""},{"location":"changelog/#tools_1","title":"Tools","text":"
     
  • intelmq_psql_initdb:
    •  
  • got support for providing custom harmonization file, generating view for storing raw fields separately, and adding IF NOT EXISTS/OR REPLACE clauses (PR#2404 by Kamil Mankowski).
    •  
  • got support for generating JSONB fields for PostgreSQL schema (PR#2436 by Kamil Mankowski).
    •  
"},{"location":"changelog/#321-2023-08-28","title":"3.2.1 (2023-08-28)","text":""},{"location":"changelog/#core_3","title":"Core","text":"
     
  • Fixed issue preventing bots from stopping after reloading (PR by Kamil Mankowski).
    •  
"},{"location":"changelog/#bots_3","title":"Bots","text":""},{"location":"changelog/#experts_3","title":"Experts","text":"
     
  • intelmq.bots.experts.reverse_dns.expert:
    •  
  • Fix the cache key to not cache results for /24 (IPv4) and /128 (IPv6) networks but for single IP-Adresses (PR#2395 by Sebastian Wagner, fixes #2394).
    •  
"},{"location":"changelog/#320-2023-07-18","title":"3.2.0 (2023-07-18)","text":""},{"location":"changelog/#core_4","title":"Core","text":"
     
  • intelmq.lib.utils:
    •  
  • resolve_dns: Deprecate dnspython versions pre-2.0.0 and disable search domains (PR#2352)
    •  
  • Fixed not resetting destination path statistics in the stats cache after restarting bot (Fixes #2331)
    •  
  • Force flushing statistics if bot will sleep longer than flushing delay (Fixes #2336)
    •  
  • intelmq.lib.upgrages: Fix a bug in the upgrade function for version 3.1.0 which caused an exception if a generic csv parser instance had no parameter type (PR#2319 by Filip Pokorn\u00fd).
    •  
  • intelmq.lib.datatypes: Adds TimeFormat class to be used for the time_format bot parameter (PR#2329 by Filip Pokorn\u00fd).
    •  
  • intelmq.lib.exceptions: Fixes a bug in InvalidArgument exception (PR#2329 by Filip Pokorn\u00fd).
    •  
  • intelmq.lib.harmonization:
    •  
  • Changes signature and names of DateTime conversion functions for consistency, backwards compatible (PR#2329 by Filip Pokorn\u00fd).
    •  
  • Ensure rejecting URLs with leading whitespaces after changes in CPython (fixes #2377)
    •  
  • intelmq.lib.bot.Bot: Allow setting the parameters via parameter on bot initialization.
    •  
"},{"location":"changelog/#development_2","title":"Development","text":"
     
  • CI: pin the Codespell version to omit troubles caused by its new releases (PR #2379).
    •  
  • CI: Updated the versions of the github actions in the CI workflows. (PR#2392 by Sebastian Kufner)
    •  
"},{"location":"changelog/#bots_4","title":"Bots","text":""},{"location":"changelog/#collectors_3","title":"Collectors","text":"
     
  • intelmq.bots.collector.rt:
    •  
  • restrict python-rt to be below version 3.0 due to introduced breaking changes,
    •  
  • added support for Subject NOT LIKE queries,
    •  
  • added support for multiple values in ticket subject queries.
    •  
  • intelmq.bots.collectors.rsync: Support for optional private key, relative time parsing for the source path, extra rsync parameters and strict host key checking (PR#2241 by Mateo Durante).
    •  
"},{"location":"changelog/#parsers_3","title":"Parsers","text":"
     
  • intelmq.bots.parsers.shadowserver._config:
    •  
  • Reset detected feedname at shutdown to re-detect the feedname on reloads (PR#2361 by @elsif2, fixes #2360).
    •  
  • Switch to dynamic configuration to decouple report schema changes from IntelMQ releases.
    •  
  • Added 'IPv6-Vulnerable-Exchange' alias and 'Accessible-WS-Discovery-Service' report. (PR#2338)
    •  
  • Removed unused p0f_genre and p0f_detail from the 'DNS-Open-Resolvers' report. (PR#2338)
    •  
  • Added 'Accessible-SIP' report. (PR#2348)
    •  
  • Added 'IPv6-Open-HTTP-Proxy' and 'IPv6-Accessible-HTTP-Proxy' aliases. (PR#2348)
    •  
  • Removed  duplicate mappings from the 'Spam-URL' report. (PR#2348)
    •  
  • intelmq.bots.parsers.generic.parser_csv: Changes time_format parameter to use new TimeFormat class (PR#2329 by Filip Pokorn\u00fd).
    •  
  • intelmq.bots.parsers.html_table.parser: Changes time_format parameter to use new TimeFormat class (PR#2329 by Filip Pokorn\u00fd).
    •  
  • intelmq.bots.parsers.turris.parser.py Updated to the latest data format (issue #2167). (PR#2373 by Filip Pokorn\u00fd).
    •  
"},{"location":"changelog/#experts_4","title":"Experts","text":"
     
  • intelmq.bots.experts.sieve:
    •  
  • Allow empty lists in sieve rule files (PR#2341 by Mikk Margus M\u00f6ll).
    •  
  • intelmq.bots.experts.cymru_whois:
    •  
  • Ignore AS names with unexpected unicode characters (PR#2352, fixes #2132)
    •  
  • Avoid extraneous search domain-based queries on NXDOMAIN result (PR#2352)
    •  
  • intelmq.bots.experts.sieve:
    •  
  • Added :before and :after keywords (PR#2374)
    •  
"},{"location":"changelog/#outputs_3","title":"Outputs","text":"
     
  • intelmq.bots.outputs.cif3.output: Added (PR#2244 by Michael Davis).
    •  
  • intelmq.bots.outputs.sql.output: New parameter fail_on_errors (PR#2362 by Sebastian Wagner).
    •  
  • intelmq.bots.outputs.smtp_batch.output: Added a bot to gathering the events and sending them by e-mails at a stroke as CSV files (PR#2253 by Edvard Rejthar)
    •  
"},{"location":"changelog/#documentation_3","title":"Documentation","text":"
     
  • API: update API installation to be aligned with the rewritten API, and clarify some missing steps.
    •  
"},{"location":"changelog/#tests_2","title":"Tests","text":"
     
  • New decorator skip_installation and environment variable INTELMQ_TEST_INSTALLATION to skip tests requiring an IntelMQ installation on the test host by default (PR#2370 by Sebastian Wagner, fixes #2369)
    •  
"},{"location":"changelog/#tools_2","title":"Tools","text":"
     
  • intelmqsetup:
    •  
  • SECURITY: fixed a low-risk bug causing the tool to change owner of / if run with the INTELMQ_PATHS_NO_OPT environment variable set. This affects only the PIP package as the DEB/RPM packages don't contain this tool. (PR#2355 by Kamil Ma\u0144kowski, fixes #2354)
    •  
  • contrib.eventdb.separate-raws-table.sql: Added the missing commas to complete the sql syntax. (PR#2386, fixes #2125 by Sebastian Kufner)
    •  
  • intelmq_psql_initdb:
    •  
  • Added parameter -o to set the output file destination. (by Sebastian Kufner)
    •  
  • intelmqctl:
    •  
  • Increased the performance through removing unnecessary reads. (by Sebastian Kufner)
    •  
"},{"location":"changelog/#known-issues_2","title":"Known Issues","text":"
This is short list of the most important known issues. The full list can be retrieved from GitHub. - intelmq.parsers.html_table may not process invalid URLs in patched Python version due to changes in urllib (#2382). - Breaking changes in 'rt' library (#2367). - Stomp collector failed (#2342). - Type error with SQL output bot's prepare_values returning list instead of tuple (#2255). - intelmq_psql_initdb does not work for SQLite (#2202). - intelmqsetup: should install a default state file (#2175). - Misp Expert - Crash if misp event already exist (#2170). - Turris greylist has been updated (#2167). - Spamhaus CERT parser uses wrong field (#2165). - Custom headers ignored in HTTPCollectorBot (#2150). - intelmqctl log: parsing syslog does not work (#2097). - Bash completion scripts depend on old JSON-based configuration files (#2094). - Bot configuration examples use JSON instead of YAML (#2066). - Bots started with IntelMQ-API/Manager stop when the webserver is restarted (#952). - Corrupt dump files when interrupted during writing (#870).
"},{"location":"changelog/#310-2023-02-10","title":"3.1.0 (2023-02-10)","text":"
     
  • Upgraded syntax to Python 3.6 (mostly Format-Strings) using pyuprade (PR#2136 by Sebastian Wagner).
    •  
"},{"location":"changelog/#core_5","title":"Core","text":"
     
  • intelmq.lib.upgrades:
    •  
  • Refactor upgrade functions global configuration handling removing the old-style defaults configuration (PR#2058 by Sebastian Wagner).
    •  
  • Pass version history as parameter to upgrade functions (PR#2058 by Sebastian Wagner).
    •  
  • intelmq.lib.message:
    •  
  • Fix and pre-compile the regular expression for harmonization key names and also check keys in the extra. namespace (PR#2059 by Sebastian Wagner, fixes #1807).
    •  
  • intelmq.lib.bot.SQLBot was replaced by an SQLMixin in intelmq.lib.mixins.SQLMixin. The Generic DB Lookup Expert bot and the SQLOutput bot were updated accordingly.
    •  
  • Added support for MSSQL (PR#2171 by Karl-Johan Karlsson).
    •  
  • Added optional reconnect delay parameter (PR#2171 by Karl-Johan Karlsson).
    •  
  • Added an ExpertBot class - it should be used by all expert bots as a parent class
    •  
  • Introduced a module for IntelMQ related datatypes intelmq.lib.datatypes which for now only contains an Enum listing the four bot types
    •  
  • Added a bottype attribute to CollectorBot, ParserBot, ExpertBot, OutputBot
    •  
  • Introduces a module for IntelMQ processmanagers. The processmanagers were up until now part of the intelmqct script.   They now reside in intelmq.lib.processmanager which also contains an interface definition the processmanager implementations must adhere to.   Both the processmanagers and the intelmqctl script were cleaned up a bit.   The LogLevel and ReturnType Enums were added to intelmq.lib.datatypes.
    •  
  • intelmq.lib.bot:
    •  
  • Enhance behaviour if an unconfigured bot is started (PR#2054 by Sebastian Wagner).
    •  
  • Fix line recovery and message dumping of the ParserBot (PR#2192 by Sebastian Wagner).
       
    • Previously the dumped message was always the last message of a report if the report contained multiple lines leading to data-loss.
      •  
     
    •  
  • Fix crashing at start in multithreaded bots (PR#2236 by DigitalTrustCenter).
    •  
  • Added default_fields parameter to ParserBot (PR#2293 by Filip Pokorn\u00fd)
    •  
  • intelmq.lib.pipeline:
    •  
  • Changed BRPOPLPUSH to BLMOVE, because BRPOPLPUSH has been marked as deprecated by redis in favor of BLMOVE (PR#2149 and PR#2240 by Sebastian Waldbauer and Sebastian Wagner, fixes #1827, #2233).
    •  
  • intelmq.lib.utils:
    •  
  • Added wrapper resolve_dns for querying DNS, with the support for recommended methods from dnspython package in versions 1 and 2.
    •  
  • Moved line filtering inside RewindableFileHandle for easier handling and limiting number of temporary objects.
    •  
  • intelmq.lib.harmonization:
    •  
  • Fixed DateTime handling of naive time strings (previously assumed local timezone, now assumes UTC) (PR#2279 by Filip Pokorn\u00fd, fixes #2278)
    •  
  • Removes tzone argument from DateTime.from_timestamp and DateTime.from_epoch_millis
    •  
  • DateTime.from_timstamp now also allows string argument
    •  
  • Removes pytz global dependency
    •  
  • Removed support for Python 3.6, including removing conditional dependencies and updating syntax to use features from newest versions. (fixes #2272)
    •  
"},{"location":"changelog/#development_3","title":"Development","text":"
     
  • Removed Python 3.6 from CI.
    •  
  • Enabled tests against Python 3.11.
    •  
"},{"location":"changelog/#bots_5","title":"Bots","text":"
     
  • Set the parent class of all bots to the correct bot class
    •  
"},{"location":"changelog/#collectors_4","title":"Collectors","text":"
     
  • intelmq.bots.collectors.mail._lib:
    •  
  • Add support for unverified SSL/STARTTLS connections (PR#2055 by Sebastian Wagner).
    •  
  • Fix exception handling for aborted IMAP connections (PR#2187 by Sebastian Wagner).
    •  
  • intelmq.bots.collectors.blueliv: Fix Blueliv collector requirements (PR#2161 by Gethvi).
    •  
  • intelmq.bots.collectors.github_api._collector_github_api: Added personal access token support (PR#2145 by Sebastian Waldbauer, fixes #1549).
    •  
  • intelmq.bots.collectors.file.collector_file: Added file lock support, no more race conditions (PR#2147 by Sebastian Waldbauer, fixes #2128)
    •  
  • intelmq.bots.collectors.shadowserver.collector_reports_api.py: Added file_format option to download reports in CSV format for better performance (PR#2246 by elsif2)
    •  
"},{"location":"changelog/#parsers_4","title":"Parsers","text":"
     
  • intelmq.bots.parsers.alienvault.parser_otx: Save CVE data in extra.cve instead of extra.CVE due to the field name restriction on lower-case characters  (PR#2059 by Sebastian Wagner).
    •  
  • intelmq.bots.parsers.anubisnetworks.parser: Changed field name format from extra.communication.http.x_forwarded_for_#1 to extra.communication.http.x_forwarded_for_1 due to the field name restriction on alphanumeric characters (PR#2059 by Sebastian Wagner).
    •  
  • intelmq.bots.parsers.dataplane.parser:
    •  
  • Add support for additional feeds (PR#2102 by Mikk Margus M\u00f6ll).
       
    • DNS Recursion Desired
      •  
    • DNS Recursion Desired ANY
      •  
    • DNS Version
      •  
    • Protocol 41
      •  
    • SMTP Greet
      •  
    • SMTP Data
      •  
    • Telnet Login
      •  
    • VNC/RFB Login
      •  
     
    •  
  • Fix event object creation (PR#2298 by DigitalTrustCenter).
    •  
  • Removed intelmq.bots.parsers.malc0de: this bot was marked as deprecated and removed from feed due to offline status (PR#2184 by Tamas Gutsohn, fixes #2178).
    •  
  • intelmq.bots.parsers.microsoft.parser_ctip:
    •  
  • New parameter overwrite (PR#2112 by Sebastian Wagner, fixes #2022).
    •  
  • Fix handling of field Payload.domain if it contains the same IP address as Payload.serverIp (PR#2144 by Mikk Margus M\u00f6ll and Sebastian Wagner).
    •  
  • Handle Payload field with non-base64-encoded JSON content and numbered dictionaries (PR#2193 by Sebastian Wagner)
    •  
  • intelmq.bots.parsers.shodan.parser (PR#2117 by Mikk Margus M\u00f6ll):
    •  
  • Instead of keeping track of extra.ftp.<something>.parameters, FTP parameters are collected together into extra.ftp.features as a list of said features, reducing field count.
    •  
  • Shodan field rsync.modules is collected.
    •  
  • Conversion functions can raise NoValueException with a string argument to signify that the conversion would not succeed, such as in the case of a single IP address being given in hostnames, which would then be passed into source.reverse_dns and fail to validate as a FQDN.
    •  
  • Variable _common_keys is moved out of the class.
    •  
  • _dict_dict_to_obj_list is introduced, for converting a string-to-dict mapping into a list of dicts with the previous key as an attribute of the dict; this can be useful for preventing issues where, when feeding the data into aggregating tools, you'd end up with many more fields than necessary, e.g vulns.CVE-2010-0001.cvss, CVE-2010-0002.cvss etc.
    •  
  • _get_first to get the first item from a list, with NoValueException raised on empty lists.
    •  
  • _get_first_hostname to handle the first valid FQDN from a list of hostnames for hostnames in the Shodan banner, if there is one, and gives NoValueException otherwise.
    •  
  • ssl.cert.serial and ssl.dhparams.generator, which may return both integers and strings, are converted to strings.
    •  
  • Changes to method apply_mapping, such as reducing needless loop iterations, removing a big try-except, and adding the NoValueException handling described above.
    •  
  • Stops falsy values (False, 0) besides None from being filtered out.
    •  
  • intelmq.bots.parsers.shadowserver._config:
    •  
  • Added support for Accessible AMQP, Device Identification Report (IPv4 and IPv6) (PR#2134 by Mateo Durante).
    •  
  • Added file name mapping for SSL-POODLE-Vulnerable-Servers IPv6 (file name scan6_ssl_poodle) (PR#2134 by Mateo Durante).
    •  
  • Added Malware-URL, Sandbox-Connection, Sandbox-DNS, Accessible-AMQP, Open-AnonymouIs-MQTT, Accessible-QUIC, Accessible-SSH, SYNful-Knock, and Special (PR#2227 by elsif2)
    •  
  • Removed legacy reports Amplification-DDoS-Victim, CAIDA-IP-Spoofer, Darknet, Drone, Drone-Brute-Force, IPv6-Sinkhole-HTTP-Drone, Microsoft-Sinkhole, and Sinkhole-HTTP-Drone (PR#2227 by elsif2).
    •  
  • Users storing events in a database should be aware that field names and types have been updated (PR#2227 by elsif2).
    •  
  • Corrected \"Accessible-AMQP\" message_length type (int) and added \"STUN\" support (PR#2235 by elsif2).
    •  
  • Added amplification factor to UDP scan reports (PR#2238 by elsif2).
    •  
  • Added version and build_date to \"Vulnerable-HTTP\" report (PR#2238 by elsif2).
    •  
  • The following field types have been standardized across all Shadowserver reports (PR#2246 by elsif2):       destination.fqdn (validate_fqdn)       destination.url (convert_http_host_and_url)       extra.browser_trusted (convert_bool)       extra.duration (convert_int)       extra.end_time (convert_date_utc)       extra.freak_vulnerable (convert_bool)       extra.ok (convert_bool)       extra.password (validate_to_none)       extra.ssl_poodle (convert_bool)       extra.status (convert_int)       extra.uptime (convert_int)       extra.version (convert_to_none)       source.network (validate_network)
    •  
  • The following report field names have changed to better represent their values:       scan_rsync:extra.password renamed to extra.has_password       scan_elasticsearch:status renamed to http_code
    •  
  • Added Accessible-HTTP-proxy and Open-HTTP-proxy (PR#2246 by elsif2).
    •  
  • Added http_agent to the Honeypot-DDoS report and added the DDoS-Participant report (PR#2303 by elsif2)
    •  
  • Added Accessible-SLP, IPv6 Accesssible-SLP, IPv6-DNS-Open-Resolvers, and IPv6-Open-LDAP-TCP reports (PR#2311 by elsif2)
    •  
  •  
    Standardized response_length to response_size in Accessible-ICS and Open-MSSQL (PR#2311 by elsif2)
     
    •  
  •  
    intelmq.bots.parsers.cymru.parser_cap_program: The parser mapped the hostname into source.fqdn which is not allowed by the IntelMQ Data Format. Added a check (PR#2215 by Sebastian Waldbauer, fixes #2169)
     
    •  
  • intelmq.bots.parsers.generic.parser_csv:
    •  
  • Use RewindableFileHandle to use the original current line for line recovery (PR#2192 by Sebastian Wagner).
    •  
  • Recovering CSV lines preserves the original line ending (PR#2280 by Kamil Mankowski, fixes #1597)
    •  
  • intelmq.bots.parsers.autoshun.parser: Removed, as the feed is discontinued (PR#2214 by Sebastian Waldbauer, fixes #2162).
    •  
  • intelmq.bots.parsers.openphish.parser_commercial: Refactored complete code (PR#2160 by Filip Pokorn\u00fd).
    •  
  • Fixes wrong mapping of host field to source.fqdn when the content was an IP address.
    •  
  • Adds newly added fields in the feed.
    •  
  • intelmq.bots.parsers.phishtank.parser: Refactored code (PR#2270 by Filip Pokorn\u00fd)
    •  
  • Changes feed URL to JSON format (contains more information). The URL needs to by manually updated in the configuration!
    •  
  • Adds fields from the JSON feed.
    •  
  • intelmq.bots.parsers.dshield.parser_domain: Has been removed, due to the feed is discontinued. (PR#2276 by Sebastian Waldbauer)
    •  
  • intelmq.bots.parsers.abusech.parser_ip: Removed (PR#2268 by Filip Pokorn\u00fd).
    •  
  • intelmq.bots.parsers.abusech.parser_domain: Removed (PR#2268 by Filip Pokorn\u00fd).
    •  
  • intelmq.bots.parsers.abusech.parser_feodotracker: Added new parser bot (PR#2268 by Filip Pokorn\u00fd)
    •  
  • Changes feed URL to JSON format (contains more information).
    •  
  • Adds fields from the JSON feed.
    •  
  • intelmq.bots.parsers.generic.parser_csv: Parameter type is deprecated, default_fields should be used. (PR#2293 by Filip Pokorn\u00fd)
    •  
  • intelmq.bots.parsers.generic.parser_csv: Parameter skip_header now allows also integer as a fixed number of lines to skip. (PR#2313 by Filip Pokorn\u00fd)
    •  
  • intelmq.bots.parsers.taichung.parser: Removed (PR#2266 by Filip Pokorn\u00fd)
    •  
"},{"location":"changelog/#experts_5","title":"Experts","text":"
     
  • intelmq.bots.experts.domain_valid: New bot for checking domain's validity (PR#1966 by Marius Karotkis).
    •  
  • intelmq.bots.experts.truncate_by_delimiter.expert: Cut string if its length is higher than a maximum length (PR#1967 by Marius Karotkis).
    •  
  • intelmq.bots.experts.remove_affix: Remove prefix or postfix strings from a field (PR#1965 by Marius Karotkis).
    •  
  • intelmq.bots.experts.asn_lookup.expert: Fixes update-database script on the last few days of a month (PR#2121 by Filip Pokorn\u00fd, fixes #2088).
    •  
  • intelmq.bots.experts.threshold.expert: Correctly use the standard parameter redis_cache_ttl instead of the previously used parameter timeout (PR#2155 by Karl-Johan Karlsson).
    •  
  • intelmq.bots.experts.jinja2.expert: Lift restriction on requirement jinja2 < 3 (PR#2158 by Sebastian Wagner).
    •  
  • intelmq.bots.experts.asn_lookup.expert, intelmq.bots.experts.domain_suffix.expert, intelmq.bots.experts.maxmind_geoip.expert, intelmq.bots.experts.recordedfuture_iprisk.expert, intelmq.bots.experts.tor_nodes.expert: New parameter autoupdate_cached_database to disable automatic updates (downloads) of cached databases (PR#2180 by Sebastian Wagner).
    •  
  • intelmq.bots.experts.url.expert: New bot for extracting additional information from source.url and/or destination.url (PR#2315 by Filip Pokorn\u00fd).
    •  
"},{"location":"changelog/#outputs_4","title":"Outputs","text":"
     
  • Removed intelmq.bots.outputs.postgresql: this bot was marked as deprecated in 2019 announced to be removed in version 3 of IntelMQ (PR#2045 by Birger Schacht).
    •  
  • Added intelmq.bots.outputs.rpz_file.output to create RPZ files (PR#1962 by Marius Karotkis).
    •  
  • Added intelmq.bots.outputs.bro_file.output to create Bro intel formatted files (PR#1963 by Marius Karotkis).
    •  
  • intelmq.bots.outputs.templated_smtp.output:
    •  
  • Add new function from_json() (which just calls json.loads() in the standard Python environment), meaning the Templated SMTP output bot can take strings containing JSON documents and do the formatting itself (PR#2120 by Karl-Johan Karlsson).
    •  
  • Lift restriction on requirement jinja2 < 3 (PR#2158 by Sebastian Wagner).
    •  
  • intelmq.bots.outputs.sql:
    •  
  • For PostgreSQL, escape Nullbytes in text to prevent \"unsupported Unicode escape sequence\" issues (PR#2223 by Sebastian Wagner, fixes #2203).
    •  
"},{"location":"changelog/#documentation_4","title":"Documentation","text":"
     
  • Feeds: Add documentation for newly supported dataplane feeds, see above (PR#2102 by Mikk Margus M\u00f6ll).
    •  
  • Installation: Restructured the whole document to make it clearer and straight-forward (PR#2113 by Sebastian Wagner).
    •  
  • Add workaround for https://github.com/sphinx-doc/sphinx/issues/10701 (PR#2225 by Sebastian Wagner, kudos @yarikoptic, fixes #2224).
    •  
  • Fix wrong operator for list-contains-value operation in sieve expert documentation (PR#2256 by Filip Pokorn\u00fd).
    •  
  • Added documentation on default_fields parameter (PR#2293 by Filip Pokorn\u00fd).
    •  
  • Updated documentation on skip_header parameter (PR#2313 by Filip Pokorn\u00fd).
    •  
  • Viriback Unsafe Sites feed replaced with Viriback C2 Tracker. (PR#2266 by Filip Pokorn\u00fd)
    •  
  • Netlab 360 Mirai Scanner feed removed as it is discontinued. (PR#2266 by Filip Pokorn\u00fd)
    •  
  • Benkow Malware Panels Tracker feed changed parser configuration. (PR#2266 by Filip Pokorn\u00fd)
    •  
  • Taichung feed removed as it is discontinued. (PR#2266 by Filip Pokorn\u00fd)
    •  
  • Added new URL Expert bot. (PR#2315 by Filip Pokorn\u00fd)
    •  
"},{"location":"changelog/#packaging_2","title":"Packaging","text":"
     
  • Remove deleted intelmq.bots.experts.sieve.validator from executables in setup.py (PR#2256 by Filip Pokorn\u00fd).
    •  
  • Run the geoip database cron-job twice a week (PR#2285 by Filip Pokorn\u00fd).
    •  
"},{"location":"changelog/#tests_3","title":"Tests","text":"
     
  • Add GitHub Action to run regexploit on all Python, JSON and YAML files (PR#2059 by Sebastian Wagner).
    •  
  • intelmq.lib.test:
    •  
  • Decorator skip_ci also detects dpkg-buildpackage environments by checking the environment variable DEB_BUILD_ARCH (PR#2123 by Sebastian Wagner).
    •  
  • Fixing regex to catchall after python version and process ID, add tests for it (PR#2216 by Sebastian Waldbauer and Sebastian Wagner, fixes #2185)
    •  
  • Also test on Python 3.10 (PR#2140 by Sebastian Wagner).
    •  
  • Switch from nosetests to pytest, as the former does not support Python 3.10 (PR#2140 by Sebastian Wagner).
    •  
  • CodeQL Github Actions exponential backtracking on strings fixed. (PR#2148 by Sebastian Waldbauer, fixes #2138)
    •  
  • Reverse DNS expert tests: remove outdated failing test test_invalid_ptr (PR#2208 by Sebastian Wagner, fixes #2206).
    •  
  • Add test dependency requests_mock to the development extra requirements in setup.py (PR#2210 by Sebastian Wagner).
    •  
  • Threshold Expert tests: Use environment variable INTELMQ_PIPELINE_HOST as redis host, analogous to other tests (PR#2209 by Sebastian Wagner, fixes #2207).
    •  
  • Remove codecov action as it failed regularly (PR#2237 by Sebastian Wagner, fixes #2229).
    •  
  • intelmq.lib.test.BotTestCase: Adds skip_checks variable to not fail on non-empty messages from calling check function (PR#2315 by Filip Pokorn\u00fd).
    •  
"},{"location":"changelog/#tools_3","title":"Tools","text":"
     
  • intelmqctl:
    •  
  • fix process manager initialization if run non-interactively, as intelmqdump does it (PR#2189 by Sebastian Wagner, fixes 2188).
    •  
  • check: handle SyntaxError in bot modules and report it without breaking execution (fixes #2177)
    •  
  • Privilege drop before logfile creation (PR#2277 by Sebastian Waldbauer, fixes 2176)
    •  
  • intelmqsetup: Revised installation of manager by building the static files at setup, not build time, making it behave more meaningful. Requires intelmq-manager >= 3.1.0 (PR#2198 by Sebastian Wagner, fixes #2197).
    •  
  • intelmqdump: Respected global and per-bot custom settings of logging_path (fix #1605).
    •  
"},{"location":"changelog/#contrib_1","title":"Contrib","text":"
     
  • logrotate: Move compress and ownership rules to the IntelMQ-blocks to prevent that they apply to other files (PR#2111 by Sebastian Wagner, fixes #2110).
    •  
"},{"location":"changelog/#known-issues_3","title":"Known issues","text":"
This is short list of the most important known issues. The full list can be retrieved from GitHub. - intelmq_psql_initdb does not work for SQLite (#2202). - intelmqsetup: should install a default state file (#2175). - Misp Expert - Crash if misp event already exist (#2170). - Turris greylist has been updated (#2167). - Spamhaus CERT parser uses wrong field (#2165). - Custom headers ignored in HTTPCollectorBot (#2150). - Missing commas in SQL query for separate Events table (#2125). - intelmqctl log: parsing syslog does not work (#2097). - Bash completion scripts depend on old JSON-based configuration files (#2094). - Bot configuration examples use JSON instead of YAML (#2066). - Bots started with IntelMQ-API/Manager stop when the webserver is restarted (#952). - Corrupt dump files when interrupted during writing (#870).
"},{"location":"changelog/#302-2021-09-10","title":"3.0.2 (2021-09-10)","text":""},{"location":"changelog/#core_6","title":"Core","text":"
     
  • intelmq.lib.bot.CollectorBot: Fixed an issue with within the new_report function, which re-loads the harmonization file after a new incoming dataset, which leads to CPU drain and decreased performance (PR#2106 by Sebastian Waldbauer, fixes #2098).
    •  
  • intelmq.lib.bot.Bot: Make private members __is_multithreadable and __collector_empty_process protected members _is_multithreadable and _collector_empty_process to make them easily modifiable by Bot classes (PR#2109 by Sebastian Wagner, fixes #2108).   Also affected and adapted bots by this change are:
    •  
  • intelmq.bots.collectors.api.collector_api
    •  
  • intelmq.bots.collectors.stomp.collector
    •  
  • intelmq.bots.experts.splunk_saved_search.expert
    •  
  • intelmq.bots.experts.threshold.expert
    •  
  • intelmq.bots.outputs.file.output
    •  
  • intelmq.bots.outputs.misp.output_api
    •  
  • intelmq.bots.outputs.misp.output_feed
    •  
  • intelmq.bots.outputs.tcp.output
    •  
  • intelmq.bots.outputs.udp.output
    •  
  • intelmq.lib.cache: Do not create the Cache class if the host is null, allows deactivating the bot statistics (PR#2104 by Sebastian Waldbauer, fixes #2103).
    •  
"},{"location":"changelog/#bots_6","title":"Bots","text":""},{"location":"changelog/#experts_6","title":"Experts","text":"
     
  • intelmq.bots.experts.domain_suffix.expert: Only print skipped database update message if verbose mode is active (PR#2107 by Sebastian Wagner, fixes #2016).
    •  
"},{"location":"changelog/#documentation_5","title":"Documentation","text":"
     
  • Add configuration upgrade steps for 3.0 to NEWS (PR#2101 by Sebastian Wagner).
    •  
"},{"location":"changelog/#known-issues_4","title":"Known issues","text":"
See open bug reports for a more detailed list. - ParserBot: erroneous raw line recovery in error handling (#1850).
"},{"location":"changelog/#301-2021-09-02","title":"3.0.1 (2021-09-02)","text":""},{"location":"changelog/#configuration_2","title":"Configuration","text":""},{"location":"changelog/#core_7","title":"Core","text":"
     
  • intelmq.lib.bot_debugger: Fix accessing the bot's destination queues (PR#2027 by Mikk Margus M\u00f6ll).
    •  
  • intelmq.lib.pipeline: Fix handling of load_balance parameter (PR#2027 by Mikk Margus M\u00f6ll).
    •  
  • intelmq.lib.bot: Fix handling of parameter destination_queues if value is an empty dictionary (PR#2051 by Sebastian Wagner, fixes #2034).
    •  
"},{"location":"changelog/#bots_7","title":"Bots","text":""},{"location":"changelog/#collectors_5","title":"Collectors","text":"
     
  • intelmq.bots.collectors.shodan.collector_stream: Fix access to parameters, the bot wrongly used self.parameters (PR#2020 by Mikk Margus M\u00f6ll).
    •  
  • intelmq.bots.collectors.mail.collector_mail_attach: Add attachment file name as extra.file_name also if the attachment is not compressed (PR#2021 by Alex Kaplan).
    •  
  • intelmq.bots.collectors.http.collector_http_stream: Fix access to parameters, the bot wrongly used self.parameters (by Sebastian Wagner).
    •  
"},{"location":"changelog/#parsers_5","title":"Parsers","text":"
     
  • intelmq.bots.parsers.microsoft.parser_ctip: Map Payload.domain to destination.fqdn instead of extra.payload.domain as it matches to destination.ip from DestinationIp (PR#2023 by Sebastian Wagner).
    •  
  • Removed intelmq.bots.parsers.malwaredomains because the upstream data source (malwaredomains.com) does not exist anymore (PR#2026 by Birger Schacht, fixes #2024).
    •  
  • intelmq.bots.parsers.shadowserver._config:
    •  
  • Add support for feed \"Vulnerable SMTP Server\" (PR#2037 by Mikk Margus M\u00f6ll).
    •  
  • Fix differentiation between feeds \"Accessible HTTP\" and \"Vulnerable HTTP\" (PR#2037 by Mikk Margus M\u00f6ll, fixes #1984).
    •  
  • Add support for the new feeds Microsoft Sinkhole Events Report, Microsoft Sinkhole HTTP Events Report (PR#2036 by Birger Schacht).
    •  
  • Complement feed mappings and documentation for feeds with IPv4 and IPv6 variants (PR#2046 by Mikk Margus M\u00f6ll and Sebastian Wagner).
    •  
  • Feed names with and without the optional IPv4/IPv6 postfix can be used now consistently.
    •  
  • Add support for feed \"Honeypot HTTP Scan\" (PR#2047 by Mikk Margus M\u00f6ll).
    •  
  • Update filename mapping for changed filename of feed \"Accessible-MSRDPUDP\" (PR#2060 by abr4xc).
    •  
"},{"location":"changelog/#experts_7","title":"Experts","text":"
     
  • intelmq.bots.experts.gethostbyname.expert: Handle numeric values for the gaierrors_to_ignore parameter (PR#2073 by Sebastian Wagner, fixes #2072).
    •  
  • intelmq.bots.experts.filter.expert: Fix handling of empty-string parameters not_after and not_before (PR#2075 by Sebastian Wagner, fixes #2074).
    •  
"},{"location":"changelog/#outputs_5","title":"Outputs","text":"
     
  • intelmq.bots.outputs.mcafee.output_esm_ip: Fix access to parameters, the bot wrongly used self.parameters (by Sebastian Wagner).
    •  
  • intelmq.bots.outputs.misp.output_api: Fix access to parameters, the bot wrongly used self.parameters (by Sebastian Wagner).
    •  
  • intelmq.bots.outputs.smtp.output: Add Content-Disposition-header to the attachment, fixing the display in Mail Clients as actual attachment (PR#2052 by Sebastian Wagner, fixes #2018).
    •  
"},{"location":"changelog/#documentation_6","title":"Documentation","text":"
     
  • Various formatting fixes (by Sebastian Wagner).
    •  
  • Removed the malwaredomains feed from the feeds list because the upstream data source (malwaredomains.com) does not exist anymore (PR#2026 by Birger Schacht, fixes #2024).
    •  
  • Update Docker installation instructions (PR#2035 by Sebastian Waldbauer).
    •  
"},{"location":"changelog/#packaging_3","title":"Packaging","text":"
     
  • intelmq-update-database crontab: Add missing recordedfuture_iprisk update call (by Sebastian Wagner).
    •  
"},{"location":"changelog/#tests_4","title":"Tests","text":"
     
  • Replace calls to deprecated/undocumented logging.warn with logging.warning (by Sebastian Wagner, fixes #2013).
    •  
  • intelmq.tests.bots.experts.rdap.test_expert: Declare cache use, fixes build failures (by Sebastian Wagner, fixes #2014).
    •  
  • intelmq.tests.bots.collectors.mail.test_collector_attach: Test text attachment (by Sebastian Wagner).
    •  
"},{"location":"changelog/#tools_4","title":"Tools","text":"
     
  • intelmqctl:
    •  
  • Also honour parameters from environment variables (PR#2068 by Sebastian Wagner, fixes #2063).
    •  
  • Fix management actions (start/stop/status/reload/restart) for groups (PR#2086 by Sebastian Wagner, fixes #2085).
    •  
  • Do not use hardcoded logging path in /opt/intelmq, use the internal default instead (PR#2092 by Sebastian Wagner, fixes #2091).
    •  
"},{"location":"changelog/#known-issues_5","title":"Known issues","text":"
See open bug reports for a more detailed list. - ParserBot: erroneous raw line recovery in error handling (#1850).
"},{"location":"changelog/#300-2021-07-02","title":"3.0.0 (2021-07-02)","text":""},{"location":"changelog/#configuration_3","title":"Configuration","text":"
     
  • The BOTS file is no longer used and has been removed (by Sebastian Wagner).
    •  
  • The defaults.conf file is no longer used and has been removed (PR#1814 by Birger Schacht).
    •  
  • The pipeline.conf file is no longer used and has been removed (PR#1849 by Birger Schacht).
    •  
  • The runtime.conf was renamed to runtime.yaml and is now in YAML format (PR#1812 by Birger Schacht).
    •  
"},{"location":"changelog/#core_8","title":"Core","text":"
     
  • intelmq.lib.harmonization:
    •  
  • New class ClassificationTaxonomy with fixed list of taxonomies and sanitiation (by Sebastian Wagner).
    •  
  • intelmq.lib.bot:
    •  
  • Handle InvalidValue exceptions upon message retrieval by dumping the message instead of repeating endlessly (#1765, PR#1766 by Filip Pokorn\u00fd).
    •  
  • Rewrite of the parameter loading and handling, getting rid of the parameters member (PR#1729 by Birger Schacht).
    •  
  • The pipeline is now initialized before the call of init to allow bots accessing data directly on startup/initialization for cleanup or maintenance tasks (PR#1982 by Sebastian Wagner).
    •  
  • intelmq.lib.exceptions:
    •  
  • InvalidValue: Add optional parameter object (PR#1766 by Filip Pokorn\u00fd).
    •  
  • intelmq.lib.utils:
    •  
  • New function list_all_bots to list all available/installed bots as replacement for the BOTS file (#368, #552, #644, #757, #1069, #1750, PR#1751 by Sebastian Waldbauer).
    •  
  • New function get_bots_settings to return the effective bot parameters, with global parameters applied (PR#1928 by Sebastian Wagner, #1927).
    •  
  • Removed deprecated function create_request_session_from_bot (PR#1997 by Sebastian Wagner, #1404).
    •  
  • parse_relative: Add support for parsing minutes and seconds (PR#1857 by Sebastian Wagner).
    •  
  • intelmq.lib.bot_debugger:
    •  
  • Set bot's logging_level directly in __init__ before the bot's initialization by changing the default value (by Sebastian Wagner).
    •  
  • Rewrite load_configuration_patch by adapting it to the parameter and configuration rewrite (by Sebastian Wagner).
    •  
  • Do not rely on the runtime configuration's group setting of bots to determine the required message type of messages given on the command line (PR#1949 by Sebastian Wagner).
    •  
"},{"location":"changelog/#development_4","title":"Development","text":"
     
  • rewrite_config_files.py: Removed obsolete BOTS-file-related rewriting functionality (by Sebastian Wagner, #1543).
    •  
  • A GitHub Action that checks for reuse compliance of all the license and copyright headers was added (PR#1976 by Birger Schacht).
    •  
  • PyYAML is no longer a required dependency for development environments, all calls to it have been replaced by ruamel.yaml (by Sebastian Wagner).
    •  
"},{"location":"changelog/#data-format_2","title":"Data Format","text":"
The IntelMQ Data Harmonization (\"DHO\") is renamed to IntelMQ Data Format (\"IDF\"). Internal files remain and work the same as before (PR#1818 by Sebastian Waldbauer, fixes 1810). Update allowed classification fields to version 1.3 (2021-05-18) (by Sebastian Wagner, fixes #1409, #1476). - The taxonomy abusive content has been renamed to abusive-content. - The taxonomy information content security has been renamed to information-content-security.   - The validation of type unauthorised-information-access has been fixed, a bug prevented the use of it.   - The validation of type unauthorised-information-modification has been fixed, a bug prevented the use of it.   - The type leak has been renamed to data-leak.   - The type dropzone has been removed. Taxonomy other with type other and identifier dropzone can be used instead. Ongoing discussion in the RSIT WG. - The taxonomy intrusion attempts has been renamed to intrusion-attempts. - For the taxonomy intrusions (PR#1993 by Sebastian Wagner, addresses #1409):   - The type compromised has been renamed to system-compromise.   - The type unauthorized-command has been merged into system-compromise.   - The type unauthorized-login has been merged into system-compromise.   - The type backdoor has been merged into system-compromise (PR#1995 by Sebastian Wagner, addresses #1409).   - The type defacement has been merged into taxonomy information-content-security, type unauthorised-information-modification (PR#1994 by Sebastian Wagner, addresses #1409). - The taxonomy information gathering has been rename to information-gathering. - The taxonomy malicious code has been renamed to malicious-code.   - The type c2server has been renamed to c2-server.   - The type malware has been integrated into infected-system and malware-distribution, respectively (PR#1917 by Sebastian Wagner addresses #1409).   - The type ransomware has been integrated into infected-system.   - The type dga domain has been moved to the taxonomy other renamed dga-domain (PR#1992 by Sebastian Wagner fixes #1613). - For the taxonomy 'availability', the type misconfiguration is new. - For the taxonomy 'other', the type unknown has been renamed to undetermined. - For the taxonomy 'vulnerable':   - The type vulnerable client has been renamed to vulnerable-system.   - The type vulnerable service has been renamed to vulnerable-system.
"},{"location":"changelog/#bots_8","title":"Bots","text":"
     
  • The parameters handling of numerous bots has been refactored (PR#1751, PR#1729, by Birger Schacht, Sebastian Wagner, Sebastian Waldbauer).
    •  
"},{"location":"changelog/#collectors_6","title":"Collectors","text":"
     
  • Remove intelmq.bots.collectors.xmpp: one of the dependencies of the bot was deprecated and according to a short survey on the IntelMQ   users mailinglist, the bot is not used by anyone. (https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html, PR#1761 by Birger Schacht, closes #1614).
    •  
  • intelmq.bots.collectors.mail._lib: Added parameter mail_starttls for STARTTLS in all mail collector bots (PR#1831 by Marius Karotkis, fixes #1128).
    •  
  • Added intelmq.bots.collectors.fireeye: A bot that collects indicators from Fireeye MAS appliances (PR#1745 by Christopher Schappelwein).
    •  
  • intelmq.bots.collectors.api.collector_api (PR#1987 by Mikk Margus M\u00f6ll, fixes #1986):
    •  
  • Added UNIX socket capability.
    •  
  • Correctly close the IOLoop in the shutdown method to fix reload.
    •  
  • intelmq.bots.collectors.rt.collector_rt (PR#1997 by Sebastian Wagner, #1404):
    •  
  • compatibility with the deprecated parameter unzip_attachment (removed in 2.1.0) was removed.
    •  
"},{"location":"changelog/#parsers_6","title":"Parsers","text":"
     
  • Added intelmq.bots.parsers.fireeye: A bot that parses hashes and URLs from Fireeye MAS indicators (PR#1745 by Christopher Schappelwein).
    •  
  • intelmq.bots.parsers.shadowserver._config:
    •  
  • Improved the feed-mapping and all conversion functions (PR#1971 by Mikk Margus M\u00f6ll).
    •  
  • intelmq.bots.parsers.generic.parser_csv:
    •  
  • Fix handling of empty string values for parameter time_format (by Sebastian Wagner).
    •  
"},{"location":"changelog/#experts_8","title":"Experts","text":"
     
  • intelmq.bots.experts.domain_suffix.expert:
    •  
  • Added --update-database option to update domain suffix database (by Sebastian Wagner).
    •  
  • Fix check method: load database with UTF-8 encoding explicitly (by Sebastian Wagner).
    •  
  • Added intelmq.bots.experts.http.expert_status: A bot that fetches the HTTP Status for a given URI and adds it to the message (PR#1789 by Birger Schacht, fixes #1047 partly).
    •  
  • Added intelmq.bots.experts.http.expert_content: A bot that fetches an HTTP resource and checks if it contains a specific string (PR#1811 by Birger Schacht).
    •  
  • Added intelmq.bots.experts.lookyloo.expert: A bot that sends requests to a lookyloo instance & adds screenshot_url to the event (PR#1844 by Sebastian Waldbauer, fixes #1048).
    •  
  • Added intelmq.bots.experts.rdap.expert: A bot that checks the rdap protocol for an abuse contact for a given domain (PR#1881 by Sebastian Waldbauer and Sebastian Wagner).
    •  
  • intelmq.bots.experts.sieve.expert:
    •  
  • Add operators for comparing lists and sets (PR#1895 by Mikk Margus M\u00f6ll):
       
    • :equals
      •  
    • :overlaps
      •  
    • :supersetof
      •  
    • :subsetof
      •  
    • :equals
      •  
     
    •  
  • Add support for comparing boolean values (PR#1895 by Mikk Margus M\u00f6ll).
    •  
  • Add support for rule negation with ! (PR#1895, PR#1923 by Mikk Margus M\u00f6ll).
    •  
  • Add support for values types float, int, bool and string for all lists items (PR#1895 by Mikk Margus M\u00f6ll).
    •  
  • Add actions for lists (PR#1895 by Mikk Margus M\u00f6ll).
       
    • append
      •  
    • append! (forced/overwriting)
      •  
     
    •  
  • Rewrite the rule-processing and operator-handling code to make it more comprehensible and extensible (PR#1895, PR#1923 by Mikk Margus M\u00f6ll).
    •  
  • Nested if statements, plus mixed actions and actions in the same scope (PR #1923 by Mikk Margus M\u00f6ll).
    •  
  • The attribute manipulation actions add, add! and update support non-string (bool/int/float) values (PR #1923 by Mikk Margus M\u00f6ll).
    •  
  • Drop the :notcontains operator, as it made is redundant by generic negation: ! foo :contains 'x' instead of foo :notcontains 'x' (PR#1957 by Mikk Margus M\u00f6ll).
    •  
  • Split string and numeric matches into single- and multivalued variants, with the relevant new operators :in, :containsany and :regexin for string lists, and :in for numeric value lists (PR#1957 by Mikk Margus M\u00f6ll).
       
    • Removed the == operator for lists, with the previous meaning of :in. Have a look at the NEWS.md for more information.
      •  
     
    •  
  • Added intelmq.bots.experts.uwhoisd: A bot that fetches the whois entry from a uwhois-instance (PR#1918 by Rapha\u00ebl Vinot).
    •  
  • Removed deprecated intelmq.bots.experts.ripencc_abuse_contact.expert. It was replaced by intelmq.bots.experts.ripe.expert and marked as deprecated in 2.0.0.beta1 (PR#1997 by Sebastian Wagner, #1404).
    •  
  • intelmq.bots.experts.modify.expert:
    •  
  • Removed compatibility with deprecated configuration format before 1.0.0.dev7 (PR#1997 by Sebastian Wagner, #1404).
    •  
  • Added intelmq.bots.experts.aggregate: A bot that aggregate events based upon given fields & a timespan (PR#1959 by Sebastian Waldbauer).
    •  
  • Added intelmq.bots.experts.tuency: A bot that queries the IntelMQ API of a tuency instance (PR#1857 by Sebastian Wagner, fixes #1856).
    •  
"},{"location":"changelog/#outputs_6","title":"Outputs","text":"
     
  • Remove intelmq.bots.outputs.xmpp: one of the dependencies of the bot was deprecated and according to a short survey on the IntelMQ   users mailinglist, the bot is not used by anyone. (https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html, PR#1761 by Birger Schacht, closes #1614)
    •  
  • intelmq.bots.outputs.smtp: Add more debug logging (PR#1949 by Sebastian Wagner).
    •  
  • Added new bot intelmq.bots.outputs.templated_smtp (PR#1901 by Karl-Johan Karlsson).
    •  
"},{"location":"changelog/#documentation_7","title":"Documentation","text":"
     
  • Updated user and developer documentation to reflect the removal of the BOTS file (PR#1780 by Birger Schacht).
    •  
  • Bots documentation:
    •  
  • Added anchors to all bot sections derived from the module names for easier linking (PR#1943 by Sebastian Wagner fixes part of certtools/intelmq-api#4).
    •  
  • License and copyright information was added to all the bots (PR#1976 by Birger Schacht).
    •  
  • Added documentation on the EventDB (PR#1955 by Birger Schacht, PR#1985 by Sebastian Wagner).
    •  
  • Added TimescaleDB for time-series documentation (PR#1990 by Sebastian Waldbauer).
    •  
  • Improved n6 interoperability documentation by adding more graphs and illustrations (PR#1991 by Sebastian Wagner).
    •  
  • Feed documentation generation: fix and simplify formatting of parameters of types lists, non-string values have been ill-treated (by Sebastian Wagner).
    •  
  • Added documentation on abuse-contact look-ups (PR#2021 by Sebastian Waldbauer and Sebastian Wagner).
    •  
"},{"location":"changelog/#packaging_4","title":"Packaging","text":"
     
  • Docker images tagged with certat/intelmq-full:develop are built and published on every push to the develop branch (PR#1753 by Sebastian Waldbauer).
    •  
  • Adapt packaging to IntelMQ 3.0 changes: ruamel.yaml dependency, changed configuration, updated database-update scripts (by Birger Schacht and Sebastian Wagner).
    •  
"},{"location":"changelog/#tests_5","title":"Tests","text":"
     
  • intelmq.tests.lib.test_bot:
    •  
  • Add test case for a raised InvalidValue exception upon message retrieval (#1765, PR#1766 by Filip Pokorn\u00fd and Sebastian Wagner).
    •  
  • intelmq.lib.test:
    •  
  • Compare content of the output field as dictionaries, not as string in assertMessageEqual (PR#1975 by Karl-Johan Karlsson).
    •  
  • Support multiple calls to run_bot from test cases (PR#1989 by Sebastian Wagner).
       
    • Split prepare_source_queue out of prepare_bot.
      •  
    • Added new optional parameter stop_bot to run_bot.
      •  
     
    •  
"},{"location":"changelog/#tools_5","title":"Tools","text":"
     
  • intelmqdump (PR#1997 by Sebastian Wagner, #1404):
    •  
  • The command e for deleting single entries by given IDs has been merged into the command d (\"delete\"), which can now delete either entries by ID or the whole file.
    •  
  • The command v for editing entries has been renamed to e (\"edit\").
    •  
"},{"location":"changelog/#contrib_2","title":"Contrib","text":"
     
  • eventdb:
    •  
  • Added separate-raws-table.sql (PR#1985 by Sebastian Wagner).
    •  
  • cron-jobs: Removed the deprecated update scripts (PR#1997 by Sebastian Wagner, #1404):
    •  
  • update-asn-data
    •  
  • update-geoip-data
    •  
  • update-tor-nodes
    •  
  • update-rfiprisk-data   in favor of the built-in update-mechanisms (see the bots' documentation). A crontab file for calling all new update command can be found in contrib/cron-jobs/intelmq-update-database.
    •  
"},{"location":"changelog/#known-issues_6","title":"Known issues","text":"
     
  • ParserBot: erroneous raw line recovery in error handling (#1850).
    •  
  • ruamel.yaml loader and dumper: human readability bug / support for comments (#2003).
    •  
"},{"location":"changelog/#233-2021-05-31","title":"2.3.3 (2021-05-31)","text":""},{"location":"changelog/#core_9","title":"Core","text":"
     
  • intelmq.lib.upgrade:
    •  
  • Added v233_feodotracker_browse for Abuse.ch Feodotracker Browse parser configuration adaption (PR#1941 by Sebastian Wagner).
    •  
"},{"location":"changelog/#bots_9","title":"Bots","text":""},{"location":"changelog/#parsers_7","title":"Parsers","text":"
     
  • intelmq.bots.parsers.microsoft.parser_ctip:
    •  
  • Add support for new field SourceIpInfo.SourceIpv4Int (PR#1940 by Sebastian Wagner).
    •  
  • Fix mapping of \"ConnectionType\" fields, this is not protocol.application. Now mapped to extra.*.connection_type (PR#1940 by Sebastian Wagner).
    •  
  • intelmq.bots.parsers.shadowserver._config:
    •  
  • Add support for the new feeds Honeypot-Amplification-DDoS-Events, Honeypot-Brute-Force-Events, Honeypot-Darknet, IP-Spoofer-Events, Sinkhole-Events, Sinkhole-HTTP-Events, Vulnerable-Exchange-Server, Sinkhole-Events-HTTP-Referer (PR#1950, PR#1952, PR#1953, PR#1954, PR#1970 by Birger Schacht and Sebastian Wagner, PR#1971 by Mikk Margus M\u00f6ll).
    •  
"},{"location":"changelog/#experts_9","title":"Experts","text":"
     
  • intelmq.bots.experts.splunk_saved_search.expert:
    •  
  • fixed erroneous string formatting (PR#1960 by Karl-Johan Karlsson).
    •  
"},{"location":"changelog/#outputs_7","title":"Outputs","text":"
     
  • intelmq.bots.outputs.smtp.output:
    •  
  • Handle empty \"fieldnames\" parameter by sending no attachment (PR#1932 by Sebastian Wagner).
    •  
"},{"location":"changelog/#documentation_8","title":"Documentation","text":"
     
  • dev/data-harmonization renamed to dev/data-format (by Sebastian Waldbauer)
    •  
  • Feeds:
    •  
  • Fixed Abuse.ch Feodotracker Browse parser configuration (PR#1941 by Sebastian Wagner fixes #1938).
    •  
"},{"location":"changelog/#tests_6","title":"Tests","text":"
     
  • intelmq.bots.parsers.html_table:
    •  
  • Added testcase for Abuse.ch Feodotracker Browse (PR#1941 by Sebastian Wagner).
    •  
"},{"location":"changelog/#tools_6","title":"Tools","text":"
     
  • intelmqsetup:
    •  
  • Set ownershop of state file path and its parent directory (PR#1911 by Sebastian Wagner).
    •  
"},{"location":"changelog/#known-issues_7","title":"Known issues","text":"
     
  • ParserBot: erroneous raw line recovery in error handling (#1850).
    •  
"},{"location":"changelog/#232-2021-04-27","title":"2.3.2 (2021-04-27)","text":""},{"location":"changelog/#core_10","title":"Core","text":"
     
  • intelmq.lib.harmonization:
    •  
  • TLP type: accept value \"yellow\" for TLP level AMBER.
    •  
"},{"location":"changelog/#bots_10","title":"Bots","text":""},{"location":"changelog/#collectors_7","title":"Collectors","text":"
     
  • intelmq.bots.collectors.shadowserver.collector_reports_api:
    •  
  • Handle timeouts by logging the error and continuing to next report (PR#1852 by Marius Karotkis and Sebastian Wagner, fixes #1823).
    •  
"},{"location":"changelog/#parsers_8","title":"Parsers","text":"
     
  • intelmq.bots.parsers.shadowserver.config:
    •  
  • Parse and harmonize field end_time as date in Feeds \"Drone-Brute-Force\" and \"Amplification-DDoS-Victim\" (PR#1833 by Mikk Margus M\u00f6ll).
    •  
  • Add conversion function convert_date_utc which assumes UTC and sanitizes the data to datetime (by Sebastian Wagner, fixes #1848).
    •  
  • intelmq.bots.parsers.shadowserver.parser_json:
    •  
  • Use the overwrite parameter for optionally overwriting the \"feed.name\" field (by Sebastian Wagner).
    •  
  • intelmq.bots.parsers.microsoft.parser_ctip:
    •  
  • Handle fields timestamp, timestamp_utc, source_ip, source_port, destination_ip, destination_port, computer_name, bot_id, asn, geo in Payload of CTIP Azure format (PR#1841, PR#1851 and PR#1879 by Sebastian Wagner).
    •  
  • intelmq.bots.parsers.shodan.parser:
    •  
  • Added support for unique keys and verified vulns (PR#1835 by Mikk Margus M\u00f6ll).
    •  
  • intelmq.bots.parsers.cymru.parser_cap_program:
    •  
  • Fix parsing in whitespace edge case in comments (PR#1870 by Alex Kaplan, fixes #1862).
    •  
"},{"location":"changelog/#experts_10","title":"Experts","text":"
     
  • intelmq.bots.experts.modify:
    •  
  • Add a new rule to the example configuration to change the type of malicious-code events to c2server if the malware name indicates c2 (PR#1854 by Sebastian Wagner).
    •  
  • intelmq.bots.experts.gethostbyname.expert:
    •  
  • Fix handling of parameter gaierrors_to_ignore with value None (PR#1890 by Sebastian Wagner, fixes #1886).
    •  
"},{"location":"changelog/#outputs_8","title":"Outputs","text":"
     
  • intelmq.bots.outputs.elasticsearch: Fix log message on required elasticsearch library message (by Sebastian Wagner).
    •  
"},{"location":"changelog/#documentation_9","title":"Documentation","text":"
     
  • dev/data-harmonization: Fix taxonomy name \"information gathering\" should be \"information-gathering\" (by Sebastian Wagner).
    •  
"},{"location":"changelog/#tests_7","title":"Tests","text":"
     
  • intelmq.tests.bots.parsers.microsoft.test_parser_ctip_azure:
    •  
  • Add test case for TLP level \"YELLOW\".
    •  
"},{"location":"changelog/#known-issues_8","title":"Known issues","text":"
     
  • ParserBot: erroneous raw line recovery in error handling (#1850).
    •  
"},{"location":"changelog/#231-2021-03-25","title":"2.3.1 (2021-03-25)","text":""},{"location":"changelog/#configuration_4","title":"Configuration","text":""},{"location":"changelog/#core_11","title":"Core","text":"
     
  • intelmq.lib.utils:
    •  
  • log: Handle null value for logging parameter logging_max_size (PR#1786 by Sebastian Wagner, fixes #1778).
    •  
  • intelmq.lib.pipeline:
    •  
  • Amqp._get_queues: Check virtual host when retrieving queue sizes. Fixes output of intelmqctl check for orphaned queues if AMQP is used and the AMQP user has access to more virtual hosts (PR#1830 by Sebastian Wagner, fixes #1746).
    •  
"},{"location":"changelog/#bots_11","title":"Bots","text":""},{"location":"changelog/#collectors_8","title":"Collectors","text":"
     
  • intelmq.bots.collectors.shadowserver.collector_reports_api: Added debug logging to show number of downloaded reports and download size (PR#1826 by Sebastian Wagner, partly addresses #1688 and #1823).
    •  
"},{"location":"changelog/#parsers_9","title":"Parsers","text":"
     
  • intelmq.bots.parsers.cymru.parser_cap_program:
    •  
  • Adapt parser to new upstream format for events of category \"bruteforce\" (PR#1795 by Sebastian Wagner, fixes 1794).
    •  
  • intelmq.bots.parsers.shodan.parser:
    •  
  • Support nested conversions, improved protocol detection and extended Shodan parser mappings (PR#1821 by Mikk Margus M\u00f6ll).
    •  
"},{"location":"changelog/#documentation_10","title":"Documentation","text":"
     
  • Add missing newlines at end of docs/_static/intelmq-manager/*.png.license files (PR#1785 by Sebastian Wagner, fixes #1777).
    •  
  • Ecosystem: Revise sections on intelmq-cb-mailgen and fody (PR#1792 by Bernhard Reiter).
    •  
  • intelmq-api: Add documentation about necessary write permission for the session database file (PR#1798 by Birger Schacht, fixes intelmq-api#23).
    •  
  • FAQ: Section on redis socket permissions: set only minimal necessary permissions (PR#1809 by Sebastian Wagner).
    •  
  • Add document on hardware requirements (PR#1811 by Sebastian Wagner).
    •  
  • Feeds: Added Shodan Country Stream (by Sebastian Wagner).
    •  
"},{"location":"changelog/#tests_8","title":"Tests","text":"
     
  • Add missing newlines at end of various test input files (PR#1785 by Sebastian Wagner, fixes #1777).
    •  
  • intelmq.tests.bots.parsers.shodan.test_parser: Add test cases for new code (PR#1821 by Mikk Margus M\u00f6ll).
    •  
  • intelmq.tests.lib.test_harmonization.test_datetime_convert: Only run this test in timezone UTC (PR#1825 by Sebastian Wagner).
    •  
"},{"location":"changelog/#tools_7","title":"Tools","text":"
     
  • intelmqsetup:
    •  
  • Also cover required directory layout and file permissions for intelmq-api (PR#1787 by Sebastian Wagner, fixes #1783).
    •  
  • Also cover webserver and sudoers configuration for intelmq-api and intelmq-manger (PR#1805 by Sebastian Wagner, fixes #1803).
    •  
  • intelmqctl:
    •  
  • Do not log an error message if logging to file is explicitly disabled, e.g. in calls from intelmsetup. The error message would not be useful for the user and is not necessary.
    •  
"},{"location":"changelog/#known-issues_9","title":"Known issues","text":"
     
  • Bots started with IntelMQ-API/Manager stop when the webserver is restarted (#952).
    •  
  • Corrupt dump files when interrupted during writing (#870).
    •  
  • CSV line recovery forces Windows line endings (#1597).
    •  
  • intelmqdump: Honor logging_path variable (#1605).
    •  
  • Timeout error in mail URL fetcher (#1621).
    •  
  • Shadowserver Parser: Drone feed has (also?) application protocol in type field (mapped to transport protocol) (#1763).
    •  
"},{"location":"changelog/#230-2021-03-04","title":"2.3.0 (2021-03-04)","text":"
IntelMQ no longer supports Python 3.5 (and thus Debian 9 and Ubuntu 16.04), the minimum supported Python version is 3.6.
"},{"location":"changelog/#configuration_5","title":"Configuration","text":""},{"location":"changelog/#core_12","title":"Core","text":"
     
  • intelmq.lib.bot:
    •  
  • ParserBot.recover_line_json_stream: Make line parameter optional, as it is not needed for this method (by Sebastian Wagner).
    •  
  • Bot.argparser: Added class method _create_argparser (returns argparse.ArgumentParser) for easy command line arguments parsing (PR#1586 by Filip Pokorn\u00fd).
    •  
  • Runtime configuration does not necessarily need a parameter entry for each block. Previously at least an empty block was required (PR#1604 by Filip Pokorn\u00fd).
    •  
  • Allow setting the pipeline host and the Redis cache host by environment variables for docker usage (PR#1669 by Sebastian Waldbauer).
    •  
  • Better logging message for SIGHUP handling if the handling of the signal is not delayed (by Sebastian Wagner).
    •  
  • intelmq.lib.upgrades:
    •  
  • Add upgrade function for removal of HPHosts Hosts file feed and intelmq.bots.parsers.hphosts parser (#1559, by Sebastian Wagner).
    •  
  • intelmq.lib.exceptions:
    •  
  • PipelineError: Remove unused code to format exceptions (by Sebastian Wagner).
    •  
  • intelmq.lib.utils:
    •  
  • create_request_session_from_bot:
       
    • Changed bot argument to optional, uses defaults.conf as fallback, renamed to create_request_session. Name create_request_session_from_bot will be removed in version 3.0.0 (PR#1524 by Filip Pokorn\u00fd).
      •  
    • Fixed setting of http_verify_cert from defaults configuration (PR#1758 by Birger Schacht).
      •  
     
    •  
  • log: Use RotatingFileHandler for allow log file rotation without external tools (PR#1637 by Vasek Bruzek).
    •  
  • intelmq.lib.harmonization:
    •  
  • The IPAddress type sanitation now accepts integer IP addresses and converts them to the string representation (by Sebastian Wagner).
    •  
  • DateTime.parse_utc_isoformat: Add parameter return_datetime to return datetime object instead of string ISO format (by Sebastian Wagner).
    •  
  • DateTime.convert: Fix utc_isoformat format, it pointed to a string and not a function, causing an exception when used (by Sebastian Wagner).
    •  
  • DateTime.from_timestamp: Ensure that time zone information (+00:00) is always present (by Sebastian Wagner).
    •  
  • DateTime.__parse now handles OverflowError exceptions from the dateutil library, happens for large numbers, e.g. telehpone numbers (by Sebastian Wagner).
    •  
  • intelmq.lib.upgrades:
    •  
  • Added upgrade function for CSV parser parameter misspelling (by Sebastian Wagner).
    •  
  • Check for existence of collector and parser for the obsolete Malware Domain List feed and raise warning if found (#1762, PR#1771 by Birger Schacht).
    •  
"},{"location":"changelog/#development_5","title":"Development","text":"
     
  • intelmq.bin.intelmq_gen_docs:
    •  
  • Add bot name to the resulting feed documentation (PR#1617 by Birger Schacht).
    •  
  • Merged into docs/autogen.py (PR#1622 by Birger Schacht).
    •  
"},{"location":"changelog/#bots_12","title":"Bots","text":""},{"location":"changelog/#collectors_9","title":"Collectors","text":"
     
  • intelmq.bots.collectors.eset.collector: Added (PR#1554 by Mikk Margus M\u00f6ll).
    •  
  • intelmq.bots.collectors.http.collector_http:
    •  
  • Added PGP signature check functionality (PR#1602 by sinus-x).
    •  
  • If status code is not 2xx, the request's and response's headers and body are logged in debug logging level (#1615, by Sebastian Wagner).
    •  
  • intelmq.bots.collectors.kafka.collector: Added (PR#1654 by Birger Schacht, closes #1634).
    •  
  • intelmq.bots.collectors.xmpp.collector: Marked as deprecated, see https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html (#1614, PR#1685 by Birger Schacht).
    •  
  • intelmq.bots.collectors.shadowserver.collector_api:
    •  
  • Added (#1683, PR#1700 by Birger Schacht).
    •  
  • Change file names in the report to .json instead of the original and wrong .csv (PR#1769 by Sebastian Wagner).
    •  
  • intelmq.bots.collectors.mail: Add content of the email's Date header as extra.email_date to the report in all email collectors (PR#1749 by aleksejsv and Sebastian Wagner).
    •  
  • intelmq.bots.collectors.http.collector_http_stream: Retry on common connection issues without raising exceptions (#1435, PR#1747 by Sebastian Waldbauer and Sebastian Wagner).
    •  
  • intelmq.bots.collectors.shodan.collector_stream: Retry on common connection issues without raising exceptions (#1435, PR#1747 by Sebastian Waldbauer and Sebastian Wagner).
    •  
  • intelmq.bots.collectors.twitter.collector_twitter:
    •  
  • Proper input validation in URLs using urllib. CWE-20, found by GitHub's CodeQL (PR#1754 by Sebastian Wagner).
    •  
  • Limit replacement (\"pastebin.com\", \"pastebin.com/raw\") to a maximum of one (PR#1754 by Sebastian Wagner).
    •  
"},{"location":"changelog/#parsers_10","title":"Parsers","text":"
     
  • intelmq.bots.parsers.eset.parser: Added (PR#1554 by Mikk Margus M\u00f6ll).
    •  
  • Ignore invalid \"NXDOMAIN\" IP addresses (PR#1573 by Mikk Margus M\u00f6ll).
    •  
  • intelmq.bots.parsers.hphosts: Removed, feed is unavailable (#1559, by Sebastian Wagner).
    •  
  • intelmq.bots.parsers.cznic.parser_haas: Added (PR#1560 by Filip Pokorn\u00fd and Edvard Rejthar).
    •  
  • intelmq.bots.parsers.cznic.parser_proki: Added (PR#1599 by sinus-x).
    •  
  • intelmq.bots.parsers.key_value.parser: Added (PR#1607 by Karl-Johan Karlsson).
    •  
  • intelmq.bots.parsers.generic.parser_csv: Added new parameter compose_fields (by Sebastian Wagner).
    •  
  • intelmq.bots.parsers.shadowserver.parser_json: Added (PR#1700 by Birger Schacht).
    •  
  • intelmq.bots.parsers.shadowserver.config:
    •  
  • Fixed mapping for Block list feed to accept network ranges in CIDR notation (#1720, PR#1728 by Sebastian Waldbauer).
    •  
  • Added mapping for new feed MSRDPUDP, Vulnerable-HTTP, Sinkhole DNS (#1716, #1726, #1733, PR#1732, PR#1735, PR#1736 by Sebastian Waldbauer).
    •  
  • Ignore value 0 for source.asn and destination.asn in all mappings to avoid parsing errors (PR#1769 by Sebastian Wagner).
    •  
  • intelmq.bots.parsers.abusech.parser_ip: Adapt to changes in the Feodo Tracker Botnet C2 IP Blocklist feed (PR#1741 by Thomas Bellus).
    •  
  • intelmq.bots.parsers.malwaredomainlist: Removed, as the feed is obsolete (#1762, PR#1771 by Birger Schacht).
    •  
"},{"location":"changelog/#experts_11","title":"Experts","text":"
     
  • intelmq.bots.experts.rfc1918.expert:
    •  
  • Add support for ASNs (PR#1557 by Mladen Markovic).
    •  
  • Speed improvements.
    •  
  • More output in debug logging mode (by Sebastian Wagner).
    •  
  • Checks parameter length on initialization and in check method (by Sebastian Wagner).
    •  
  • intelmq.bots.experts.gethostbyname.expert:
    •  
  • Added parameter fallback_to_url and set to True (PR#1586 by Edvard Rejthar).
    •  
  • Added parameter gaierrors_to_ignore to optionally ignore other gethostbyname errors (#1553).
    •  
  • Added parameter overwrite to optionally overwrite existing IP addresses (by Sebastian Wagner).
    •  
  • intelmq.bots.experts.asn_lookup.expert:
    •  
  • Added --update-database option (PR#1524 by Filip Pokorn\u00fd).
    •  
  • The script update-asn-data is now deprecated and will be removed in version 3.0.
    •  
  • intelmq.bots.experts.maxmind_geoip.expert:
    •  
  • Added --update-database option (PR#1524 by Filip Pokorn\u00fd).
    •  
  • Added license_key parameter (PR#1524 by Filip Pokorn\u00fd).
    •  
  • The script update-geoip-data is now deprecated and will be removed in version 3.0.
    •  
  • intelmq.bots.experts.tor_nodes.expert:
    •  
  • Added --update-database option (PR#1524 by Filip Pokorn\u00fd).
    •  
  • The script update-tor-nodes is now deprecated and will be removed in version 3.0.
    •  
  • intelmq.bots.experts.recordedfuture_iprisk.expert:
    •  
  • Added --update-database option (PR#1524 by Filip Pokorn\u00fd).
    •  
  • Added api_token parameter (PR#1524 by Filip Pokorn\u00fd).
    •  
  • The script update-rfiprisk-data is now deprecated and will be removed in version 3.0.
    •  
  • Added intelmq.bots.experts.threshold (PR#1608 by Karl-Johan Karlsson).
    •  
  • Added intelmq.bots.experts.splunk_saved_search.expert (PR#1666 by Karl-Johan Karlsson).
    •  
  • intelmq.bots.experts.sieve.expert:
    •  
  • Added possibility to give multiple queue names for the path directive (#1462, by Sebastian Wagner).
    •  
  • Added possibility to run actions without filtering expression (#1706, PR#1708 by Sebastian Waldbauer).
    •  
  • Added datetime math operations (#1680, PR#1696 by Sebastian Waldbauer).
    •  
  • intelmq.bots.experts.maxmind_geoip.expert:
    •  
  • Fixed handing over of overwrite parameter to event.add (PR#1743 by Birger Schacht).
    •  
"},{"location":"changelog/#outputs_9","title":"Outputs","text":"
     
  • intelmq.bots.outputs.rt: Added Request Tracker output bot (PR#1589 by Marius Urkis).
    •  
  • intelmq.bots.outputs.xmpp.output: Marked as deprecated, see https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html (#1614, PR#1685 by Birger Schacht).
    •  
  • intelmq.bots.outputs.smtp.output: Fix sending to multiple recipients when recipients are defined by event-data (#1759, PR#1760 by Sebastian Waldbauer and Sebastian Wagner).
    •  
"},{"location":"changelog/#documentation_11","title":"Documentation","text":"
     
  • Feeds:
    •  
  • Add ESET URL and Domain feeds (by Sebastian Wagner).
    •  
  • Remove unavailable HPHosts Hosts file feed (#1559 by Sebastian Wagner).
    •  
  • Added CZ.NIC HaaS feed (PR#1560 by Filip Pokorn\u00fd and Edvard Rejthar).
    •  
  • Added CZ.NIC Proki feed (PR#1599 by sinus-x).
    •  
  • Updated Abuse.ch URLhaus feed (PT#1572 by Filip Pokorn\u00fd).
    •  
  • Added CERT-BUND CB-Report Malware infections feed (PR#1598 by sinus-x and Sebastian Wagner).
    •  
  • Updated Turris Greylist feed with PGP verification information (by Sebastian Wagner).
    •  
  • Fixed parsing of the public field in the generated feeds documentation (PR#1641 by Birger Schacht).
    •  
  • Change the rate_limit parameter of some feeds from 2 days (129600 seconds) to one day (86400 seconds).
    •  
  • Update the cAPTure Ponmocup Domains feed documentation (PR#1574 by Filip Pokorn\u00fd and Sebastian Wagner).
    •  
  • Added Shadowserver Reports API (by Sebastian Wagner).
    •  
  • Change the rate_limit parameter for many feeds from 2 days to the default one day (by Sebastian Wagner).
    •  
  • Removed Malware Domain List feed, as the feed is obsolete (#1762, PR#1771 by Birger Schacht).
    •  
  • Bots:
    •  
  • Enhanced documentation of RFC1918 Expert (PR#1557 by Mladen Markovic and Sebastian Wagner).
    •  
  • Enhanced documentation of SQL Output (PR#1620 by Edvard Rejthar).
    •  
  • Updated documentation for MaxMind GeoIP, ASN Lookup, TOR Nodes and Recorded Future experts to reflect new --update-database option (PR#1524 by Filip Pokorn\u00fd).
    •  
  • Added documentation for Shadowserver API collector and parser (PR#1700 by Birger Schacht and Sebastian Wagner).
    •  
  • Add n6 integration documentation (by Sebastian Wagner).
    •  
  • Moved 'Orphaned Queues' section from the FAQ to the intelmqctl documentation (by Sebastian Wagner).
    •  
  • Generate documentation using Sphinx (PR#1622 by Birger Schacht).
    •  
  • The documentation is now available at https://intelmq.readthedocs.io/en/latest/
    •  
  • Refactor documentation and fix broken syntax (#1639, PRs #1638 #1640 #1642 by Birger Schacht).
    •  
  • Integrate intelmq-manager and intelmq-api user documentation to provide unified documentation place (PR#1714 & PR#1714 by Birger Schacht).
    •  
"},{"location":"changelog/#packaging_5","title":"Packaging","text":"
     
  • Fix paths in the packaged logcheck rules (by Sebastian Wagner).
    •  
  • Build the sphinx documentation on package build (PR#1701 by Birger Schacht).
    •  
  • Ignore non-zero exit-codes for the intelmqctl check call in postinst (#1748, by Sebastian Wagner).
    •  
"},{"location":"changelog/#tests_9","title":"Tests","text":"
     
  • Added tests for intelmq.lib.exceptions.PipelineError (by Sebastian Wagner).
    •  
  • intelmq.tests.bots.collectors.http_collector.test_collector: Use requests_mock to mock all requests and do not require a local webserver (by Sebastian Wagner).
    •  
  • intelmq.tests.bots.outputs.restapi.test_output:
    •  
  • Use requests_mock to mock all requests and do not require a local webserver (by Sebastian Wagner).
    •  
  • Add a test for checking the response status code (by Sebastian Wagner).
    •  
  • intelmq.tests.bots.collectors.mail.test_collector_url: Use requests_mock to mock all requests and do not require a local webserver (by Sebastian Wagner).
    •  
  • intelmq.tests.bots.experts.ripe.test_expert: Use requests_mock to mock all requests and do not require a local webserver (by Sebastian Wagner).
    •  
  • The test flag (environment variable) INTELMQ_TEST_LOCAL_WEB is no longer used (by Sebastian Wagner).
    •  
  • Added tests for intelmq.harmonization.DateTime.parse_utc_isoformat and convert_fuzzy (by Sebastian Wagner).
    •  
  • Move from Travis to GitHub Actions (PR#1707 by Birger Schacht).
    •  
  • intelmq.lib.test:
    •  
  • test_static_bot_check_method checks the bot's static check(parameters) method for any exceptions, and a valid formatted return value (#1505, by Sebastian Wagner).
    •  
  • setUpClass: Skip tests if cache was requests with use_cache member, but Redis is deactivated with the environment variable INTELMQ_SKIP_REDIS (by Sebastian Wagner).
    •  
  • intelmq.tests.bots.experts.cymru_whois.test_expert:
    •  
  • Switch from example.com to ns2.univie.ac.at for hopefully more stable responses (#1730, PR#1731 by Sebastian Waldbauer).
    •  
  • Do not test for exact expected values in the 6to4 network test, as the values are changing regularly (by Sebastian Wagner).
    •  
  • intelmq.tests.bots.parsers.abusech: Remove tests cases of discontinued feeds (PR#1741 by Thomas Bellus).
    •  
  • Activate GitHub's CodeQL Code Analyzing tool as GitHub Action (by Sebastian Wagner).
    •  
"},{"location":"changelog/#tools_8","title":"Tools","text":"
     
  • intelmqdump:
       
    • Check if given queue is configured upon recovery (#1433, PR#1587 by Mladen Markovic).
      •  
     
    •  
  • intelmqctl:
    •  
  • intelmq list queues: --sum, --count, -s flag for showing total count of messages (#1408, PR#1581 by Mladen Markovic).
    •  
  • intelmq check: Added a possibility to ignore queues from the orphaned queues check (by Sebastian Wagner).
    •  
  • Allow setting the pipeline host by environment variables for docker usage (PR#1669 by Sebastian Waldbauer).
    •  
"},{"location":"changelog/#contrib_3","title":"Contrib","text":"
     
  • EventDB:
    •  
  • Add SQL script for keeping track of the oldest inserted/update \"time.source\" information (by Sebastian Wagner).
    •  
  • Cron Jobs: The script intelmq-update-data has been renamed to intelmq-update-database (by Filip Pokorn\u00fd).
    •  
  • Dropped utterly outdated contrib modules (by Sebastian Wagner):
    •  
  • ansible
    •  
  • vagrant
    •  
  • vagrant-ansible
    •  
  • logrotate:
    •  
  • Do not use the deprecated \"copytruncate\" option as intelmq re-opens the log anyways (by Sebastian Wagner).
    •  
  • Set file permissions to 0644 (by Sebastian Wagner).
    •  
"},{"location":"changelog/#known-issues_10","title":"Known issues","text":"
     
  • Bots started with IntelMQ-API/Manager stop when the webserver is restarted (#952).
    •  
  • Corrupt dump files when interrupted during writing (#870).
    •  
  • CSV line recovery forces Windows line endings (#1597).
    •  
  • Timeout error in mail URL fetcher (#1621).
    •  
  • AMQP pipeline: get_queues needs to check vhost of response (#1746).
    •  
"},{"location":"changelog/#223-2020-12-23","title":"2.2.3 (2020-12-23)","text":""},{"location":"changelog/#documentation_12","title":"Documentation","text":"
     
  • Bots/Sieve expert: Add information about parenthesis in if-expressions (#1681, PR#1687 by Birger Schacht).
    •  
"},{"location":"changelog/#harmonization","title":"Harmonization","text":"
     
  • See NEWS.md for information on a fixed bug in the taxonomy expert.
    •  
"},{"location":"changelog/#bots_13","title":"Bots","text":""},{"location":"changelog/#collectors_10","title":"Collectors","text":"
     
  • intelmq.bots.rt.collector_rt: Log the size of the downloaded file in bytes on debug logging level.
    •  
"},{"location":"changelog/#parsers_11","title":"Parsers","text":"
     
  • intelmq.bots.parsers.cymru.parser_cap_program:
    •  
  • Add support for protocols 47 (GRE) and 59 (IPv6-NoNxt).
    •  
  • Add support for field additional_asns in optional information column.
    •  
  • intelmq.bots.parsers.microsoft.parser_ctip:
    •  
  • Fix mapping of DestinationIpInfo.DestinationIpConnectionType field (contained a typo).
    •  
  • Explicitly ignore field DestinationIpInfo.DestinationIpv4Int as the data is already in another field.
    •  
  • intelmq.bots.parsers.generic.parser_csv:
    •  
  • Ignore line having spaces or tabs only or comment having leading tabs or spaces (PR#1669 by Brajneesh).
    •  
  • Data fields containing - are now ignored and do not raise an exception anymore (#1651, PR#74 by Sebastian Waldbauer).
    •  
"},{"location":"changelog/#experts_12","title":"Experts","text":"
     
  • intelmq.bots.experts.taxonomy.expert: Map type scanner to information-gathering instead of information gathering. See NEWS file for more information.
    •  
"},{"location":"changelog/#tests_10","title":"Tests","text":"
     
  • Travis: Deactivate tests with optional requirements on Python 3.5, as the build fails because of abusix/querycontacts version conflicts on dnspython.
    •  
"},{"location":"changelog/#known-issues_11","title":"Known issues","text":"
     
  • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952).
    •  
  • Corrupt dump files when interrupted during writing (#870).
    •  
"},{"location":"changelog/#222-2020-10-28","title":"2.2.2 (2020-10-28)","text":""},{"location":"changelog/#core_13","title":"Core","text":"
     
  • intelmq.lib.upgrades:
    •  
  • Add upgrade function for renamed Shadowserver feed name \"Blacklisted-IP\"/\"Blocklist\".
    •  
"},{"location":"changelog/#bots_14","title":"Bots","text":""},{"location":"changelog/#parsers_12","title":"Parsers","text":"
     
  • intelmq.bots.parsers.shadowserver:
    •  
  • Rename \"Blacklisted-IP\" feed to \"Blocklist\", old name is still valid until IntelMQ version 3.0 (PR#1588 by Thomas Hungenberg).
    •  
  • Added support for the feeds Accessible Radmin and CAIDA IP Spoofer (PR#1600 by sinus-x).
    •  
  • intelmq.bots.parsers.anubisnetworks.parser: Fix parsing error where dst.ip was not equal to comm.http.host.
    •  
  • intelmq/bots/parsers/danger_rulez/parser: correctly skip malformed rows by defining variables before referencing (PR#1601 by Tomas Bellus).
    •  
  • `intelmq.bots.parsers.misp.parser: Fix MISP Event URL (#1619, PR#1618 by Nedfire23).
    •  
  • intelmq.bots.parsers.microsoft.parser_ctip:
    •  
  • Add support for DestinationIpInfo.* and Signatures.Sha256 fields, used by the ctip-c2 feed (PR#1623 by Mikk Margus M\u00f6ll).
    •  
  • Use extra.payload.text for the feed's field Payload if the content cannot be decoded (PR#1610 by Giedrius Ramas).
    •  
"},{"location":"changelog/#experts_13","title":"Experts","text":"
     
  • intelmq.bots.experts.cymru_whois:
    •  
  • Fix cache key calculation which previously led to duplicate keys and therefore wrong results in rare cases. The cache key calculation is intentionally not backwards-compatible (#1592, PR#1606).
    •  
  • The bot now caches and logs (as level INFO) empty responses from Cymru (PR#1606).
    •  
"},{"location":"changelog/#documentation_13","title":"Documentation","text":"
     
  • README:
    •  
  • Add Core Infrastructure Initiative Best Practices Badge.
    •  
  • Bots:
    •  
  • Generic CSV Parser: Add note on escaping backslashes (#1579).
    •  
  • Remove section of non-existing \"Copy Extra\" Bot.
    •  
  • Explain taxonomy expert.
    •  
  • Add documentation on n6 parser.
    •  
  • Gethostbyname expert: Add documentation how errors are treated.
    •  
  • Feeds:
    •  
  • Fixed bot modules of Calidog CertStream feed.
    •  
  • Add information on Microsoft CTIP C2 feed.
    •  
"},{"location":"changelog/#packaging_6","title":"Packaging","text":"
     
  • In Debian packages, intelmqctl check and intelmqctl upgrade-config are executed in the \"postinst\" step (#1551, PR#1624 by Birger Schacht).
    •  
  • Require requests<2.26 for Python 3.5, as 2.25.x will be the last release series of the requests library with support for Python 3.5.
    •  
"},{"location":"changelog/#tests_11","title":"Tests","text":"
     
  • intelmq.tests.lib.test_pipeline: Skip TestAmqp.test_acknowledge on Travis with Python 3.8.
    •  
  • intelmq.tests.bots.outputs.elasticsearch.test_output: Refresh index intelmq manually to fix random test failures (#1593, PR#1595 by Zach Stone).
    •  
"},{"location":"changelog/#tools_9","title":"Tools","text":"
     
  • intelmqctl check:
    •  
  • For disabled bots which do not have any pipeline connections, do not raise an error, but only warning.
    •  
  • Fix check on source/destination queues for bots as well the orphaned queues.
    •  
"},{"location":"changelog/#contrib_4","title":"Contrib","text":"
     
  • Bash completion scripts: Check both /opt/intelmq/ as well as LSB-paths (/etc/intelmq/ and /var/log/intelmq/) for loading bot information (#1561, PR#1628 by Birger Schacht).
    •  
"},{"location":"changelog/#known-issues_12","title":"Known issues","text":"
     
  • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952).
    •  
  • Corrupt dump files when interrupted during writing (#870).
    •  
"},{"location":"changelog/#221-2020-07-30","title":"2.2.1 (2020-07-30)","text":""},{"location":"changelog/#core_14","title":"Core","text":"
     
  • intelmq.lib.upgrades:
    •  
  • Add upgrade function for changed configuration of the feed \"Abuse.ch URLhaus\" (#1571, PR#1572 by Filip Pokorn\u00fd).
    •  
  • Add upgrade function for removal of HPHosts Hosts file feed and intelmq.bots.parsers.hphosts parser (#1559).
    •  
  • intelmq.lib.harmonization:
       
    • For IP Addresses, explicitly reject IPv6 addresses with scope ID (due to changed behavior in Python 3.9, #1550).
      •  
     
    •  
"},{"location":"changelog/#development_6","title":"Development","text":"
     
  • Ignore line length (E501) in code-style checks altogether.
    •  
"},{"location":"changelog/#bots_15","title":"Bots","text":""},{"location":"changelog/#collectors_11","title":"Collectors","text":"
     
  • intelmq.bots.collectors.misp: Fix access to actual MISP object (PR#1548 by Tomas Bellus @tomas321)
    •  
  • intelmq.bots.collectors.stomp: Remove empty client.pem file.
    •  
"},{"location":"changelog/#parsers_13","title":"Parsers","text":"
     
  • intelmq.bots.parsers.shadowserver.config:
    •  
  • Add support for Accessible-CoAP feed (PR #1555 by Thomas Hungenberg).
    •  
  • Add support for Accessible-ARD feed (PR #1584 by Tomas Bellus @tomas321).
    •  
  • intelmq.bots.parser.anubisnetworks.parser: Ignore \"TestSinkholingLoss\" events, these are not intended to be sent out at all.
    •  
  • intelmq.bots.parsers.generic.parser_csv: Allow values of type dictionary for parameter type_translation.
    •  
  • intelmq.bots.parsers.hphosts: Removed, feed is unavailable (#1559).
    •  
  • intelmq.bots.parsers.cymru.parser_cap_program: Add support for comment \"username\" for \"scanner\" category.
    •  
  • intelmq.bots.parsers.malwareurl.parser: Check for valid FQDN and IP address in URL and IP address columns (PR#1585 by Marius Urkis).
    •  
"},{"location":"changelog/#experts_14","title":"Experts","text":"
     
  • intelmq.bots.experts.maxmind_geoip: On Python < 3.6, require maxminddb < 2, as that version does no longer support Python 3.5.
    •  
"},{"location":"changelog/#outputs_10","title":"Outputs","text":"
     
  • intelmq.bots.outputs.udp: Fix error handling on sending, had a bug itself.
    •  
"},{"location":"changelog/#documentation_14","title":"Documentation","text":"
     
  • Feeds:
    •  
  • Update documentation of feed \"Abuse.ch URLhaus\" (#1571, PR#1572 by Filip Pokorn\u00fd).
    •  
  • Bots:
    •  
  • Overhaul of all bots' description fields (#1570).
    •  
  • User-Guide:
    •  
  • Overhaul pipeline configuration section and explain named queues better (#1577).
    •  
"},{"location":"changelog/#tests_12","title":"Tests","text":"
     
  • intelmq.tests.bots.experts.cymru: Adapt test_empty_result, remove test_unicode_as_name and test_country_question_mark (#1576).
    •  
"},{"location":"changelog/#tools_10","title":"Tools","text":"
     
  • intelmq.bin.intelmq_gen_docs: Format parameters of types lists with double quotes around values to produce conform JSON, ready to copy and paste the value into the IntelMQ Manager's bot parameter form.
    •  
  • intelmq.bin.intelmqctl:
    •  
  • debug: In JSON mode, use dictionaries instead of lists.
    •  
  • debug: Add PATH to the paths shown.
    •  
  • check: Show $PATH environment variable if executable cannot be found.
    •  
"},{"location":"changelog/#contrib_5","title":"Contrib","text":"
     
  • malware_name_mapping: Change MISP Threat Actors URL to new URL (branch master -> main) in download script.
    •  
"},{"location":"changelog/#known-issues_13","title":"Known issues","text":"
     
  • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952).
    •  
  • Corrupt dump files when interrupted during writing (#870).
    •  
  • Bash completion scripts search in wrong directory in packages (#1561).
    •  
  • Cymru Expert: Wrong Cache-Key Calculation (#1592).
    •  
"},{"location":"changelog/#220-2020-06-18","title":"2.2.0 (2020-06-18)","text":"
Dropped support for Python 3.4.
"},{"location":"changelog/#core_15","title":"Core","text":"
     
  • __init__: Changes to the path-handling, see User Guide, section /opt and LSB paths for more information
    •  
  • The environment variable INTELMQ_ROOT_DIR can be used to set custom root directories instead of /opt/intelmq/ (#805) in case of non LSB-path installations.
    •  
  • The environment variable ROOT_DIR can be used to set custom root directories instead of / (#805) in case of LSB-path installations.
    •  
  • intelmq.lib.exceptions: Added MissingDependencyError for show error messages about a missing library and how to install it (#1471).
    •  
  • Added optional parameter installed to show the installed version.
    •  
  • Added optional parameter additional_text to show arbitrary text.
    •  
  • Adding more type annotations for core libraries.
    •  
  • intelmq.lib.pipeline.Pythonlist.sleep: Drop deprecated method.
    •  
  • intelmq.lib.utils: write_configuration: Append a newline at end of configuration/file to allow proper comparisons & diffs.
    •  
  • intelmq.lib.test: BotTestCase drops privileges upon initialization (#1489).
    •  
  • intelmq.lib.bot:
    •  
  • New class OutputBot:
       
    • Method export_event to format/export events according to the parameters given by the user.
      •  
     
    •  
  • ParserBot: New methods parse_json_stream and recover_line_json_stream.
    •  
  • ParserBot.recover_line_json: Fix format by adding a list around the line data.
    •  
  • Bot.send_message: In debugging log level, the path to which the message is sent is now logged too.
    •  
"},{"location":"changelog/#bots_16","title":"Bots","text":"
     
  • Bots with dependencies: Use of intelmq.lib.exceptions.MissingDependencyError.
    •  
"},{"location":"changelog/#collectors_12","title":"Collectors","text":"
     
  • intelmq.bots.collectors.misp.collector: Deprecate parameter misp_verify in favor of generic parameter http_verify_cert.
    •  
  • intelmq.bots.collectors.tcp.collector: Drop compatibility with Python 3.4.
    •  
  • intelmq.bots.collectors.stomp.collector:
    •  
  • Check the stomp.py version and show an error message if it does not match.
    •  
  • For stomp.py versions >= 5.0.0 redirect the stomp.PrintingListener output to debug logging.
    •  
  • intelmq.bots.collectors.microsoft.collector_azure: Support current Python library azure-storage-blob>= 12.0.0, configuration is incompatible and needs manual change. See NEWS file and bot's documentation for more details.
    •  
  • intelmq.bots.collectors.amqp.collector_amqp: Require pika minimum version 1.0.
    •  
  • intelmq.bots.collectors.github_api.collector_github_contents_api: Added (PR#1481).
    •  
"},{"location":"changelog/#parsers_14","title":"Parsers","text":"
     
  • intelmq.bots.parsers.autoshun.parser: Drop compatibility with Python 3.4.
    •  
  • intelmq.bots.parsers.html_table.parser: Drop compatibility with Python 3.4.
    •  
  • intelmq.bots.parsers.shadowserver.parser: Add support for MQTT and Open-IPP feeds (PR#1512, PR#1544).
    •  
  • intelmq.bots.parsers.taichung.parser:
    •  
  • Migrate to ParserBot.
    •  
  • Also parse geolocation information if available.
    •  
  • intelmq.bots.parsers.cymru.parser_full_bogons:
    •  
  • Migrate to ParserBot.
    •  
  • Add last updated information in raw.
    •  
  • intelmq.bots.parsers.anubisnetworks.parser: Add new parameter use_malware_familiy_as_classification_identifier.
    •  
  • intelmq.bots.parsers.microsoft.parser_ctip: Compatibility for new CTIP data format used provided by the Azure interface.
    •  
  • intelmq.bots.parsers.cymru.parser_cap_program: Support for openresolver type.
    •  
  • intelmq.bots.parsers.github_feed.parser: Added (PR#1481).
    •  
  • intelmq.bots.parsers.urlvir.parser: Removed, as the feed is discontinued (#1537).
    •  
"},{"location":"changelog/#experts_15","title":"Experts","text":"
     
  • intelmq.bots.experts.csv_converter: Added as converter to CSV.
    •  
  • intelmq.bots.experts.misp: Added (PR#1475).
    •  
  • intelmq.bots.experts.modify: New parameter maximum_matches.
    •  
"},{"location":"changelog/#outputs_11","title":"Outputs","text":"
     
  • intelmq.bots.outputs.amqptopic:
    •  
  • Use OutputBot and export_event.
    •  
  • Allow formatting the routing key with event data by the new parameter format_routing_key (boolean).
    •  
  • intelmq.bots.outputs.file: Use OutputBot and export_event.
    •  
  • intelmq.bots.outputs.files: Use OutputBot and export_event.
    •  
  • intelmq.bots.outputs.misp.output_feed: Added, creates a MISP Feed (PR#1473).
    •  
  • intelmq.bots.outputs.misp.output_api: Added, pushes to MISP via the API (PR#1506, PR#1536).
    •  
  • intelmq.bots.outputs.elasticsearch.output: Dropped ElasticSearch version 5 compatibility, added version 7 compatibility (#1513).
    •  
"},{"location":"changelog/#documentation_15","title":"Documentation","text":"
     
  • Document usage of the INTELMQ_ROOT_DIR environment variable.
    •  
  • Added document on MISP integration possibilities.
    •  
  • Feeds:
    •  
  • Added \"Full Bogons IPv6\" feed.
    •  
  • Remove discontinued URLVir Feeds (#1537).
    •  
"},{"location":"changelog/#packaging_7","title":"Packaging","text":"
     
  • setup.py do not try to install any data to /opt/intelmq/ as the behavior is inconsistent on various systems and with intelmqsetup we have a tool to create the structure and files anyway.
    •  
  • debian/rules:
    •  
  • Provide a blank state file in the package.
    •  
  • Patches:
    •  
  • Updated fix-intelmq-paths.patch.
    •  
"},{"location":"changelog/#tests_13","title":"Tests","text":"
     
  • Travis: Use intelmqsetup here too.
    •  
  • Install required build dependencies for the Debian package build test.
    •  
  • This version is no longer automatically tested on Python < 3.5.
    •  
  • Also run the tests on Python 3.8.
    •  
  • Run the Debian packaging tests on Python 3.5 and the code-style test on 3.8.
    •  
  • Added tests for the new bot intelmq.bots.outputs.misp.output_feed (#1473).
    •  
  • Added tests for the new bot intelmq.bots.experts.misp.expert (#1473).
    •  
  • Added tests for intelmq.lib.exceptions.
    •  
  • Added tests for intelmq.lib.bot.OutputBot and intelmq.lib.bot.OutputBot.export_event.
    •  
  • Added IPv6 tests for intelmq.bots.parsers.cymru.parser_full_bogons.
    •  
  • Added tests for intelmq.lib.bot.ParserBot's new methods parse_json_stream and recover_line_json_stream.
    •  
  • intelmq.tests.test_conf: Set encoding to UTF-8 for reading the feeds.yaml file.
    •  
"},{"location":"changelog/#tools_11","title":"Tools","text":"
     
  • intelmqctl:
    •  
  • upgrade-config:
       
    • Allow setting the state file location with the --state-file parameter.
      •  
    • Do not require a second run anymore, if the state file is newly created (#1491).
      •  
    • New parameter no_backup/--no-backup to skip creation of .bak files for state and configuration files.
      •  
     
    •  
  • Only require psutil for the IntelMQProcessManager, not for process manager independent calls like upgrade-config or check.
    •  
  • Add new command debug to output some information for debugging. Currently implemented:
       
    • paths
      •  
    • environment variables
      •  
     
    •  
  • IntelMQController: New argument --no-file-logging to disable logging to file.
    •  
  • If dropping privileges does not work, intelmqctl will now abort (#1489).
    •  
  • intelmqsetup:
    •  
  • Add argument parsing and an option to skip setting file ownership, possibly not requiring root permissions.
    •  
  • Call intelmqctl upgrade-config and add argument for the state file path (#1491).
    •  
  • intelmq_generate_misp_objects_templates.py: Tool to create a MISP object template (#1470).
    •  
  • intelmqdump: New parameter -t or --truncate to optionally give the maximum length of raw data to show, 0 for no truncating.
    •  
"},{"location":"changelog/#contrib_6","title":"Contrib","text":"
     
  • Added development-tools.
    •  
  • ElasticSearch: Dropped version 5 compatibility, added version 7 compatibility (#1513).
    •  
  • Malware Name Mapping Downloader:
    •  
  • New parameter --mwnmp-ignore-adware.
    •  
  • The parameter --add-default supports an optional parameter to define the default value.
    •  
"},{"location":"changelog/#known-issues_14","title":"Known issues","text":"
     
  • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952).
    •  
  • Corrupt dump files when interrupted during writing (#870).
    •  
"},{"location":"changelog/#213-2020-05-26","title":"2.1.3 (2020-05-26)","text":""},{"location":"changelog/#requirements","title":"Requirements","text":"
     
  • The python library requests is (again) listed as dependency of the core (#1519).
    •  
"},{"location":"changelog/#core_16","title":"Core","text":"
     
  • intelmq.lib.upgrades:
    •  
  • Harmonization upgrade: Also check and update regular expressions.
    •  
  • Add function to migrate the deprecated parameter attach_unzip to extract_files for the mail attachment collector.
    •  
  • Add function to migrate changed Taichung URL feed.
    •  
  • Check for discontinued Abuse.CH Zeus Tracker feed.
    •  
  • intelmq.lib.bot:
    •  
  • ParserBot.recover_line: Parameter line needs to be optional, fix usage of fallback value self.current_line.
    •  
  • start: Handle decoding errors in the pipeline different so that the bot is not stuck in an endless loop (#1494).
    •  
  • start: Only acknowledge a message in case of errors, if we actually had a message to dump, which is not the case for collectors.
    •  
  • _dump_message: Dump messages with encoding errors base64 encoded, not in JSON format as it's not possible to decode them (#1494).
    •  
  • intelmq.lib.test:
    •  
  • BotTestCase.run_bot: Add parameters allowed_error_count and allowed_warning_count to allow set the number per run, not per test class.
    •  
  • Set source_pipeline_broker and destination_pipeline_broker to pythonlist instead of the old broker, fixes intelmq.tests.lib.test_bot.TestBot.test_pipeline_raising.
    •  
  • Fix test for (allowed) errors and warnings.
    •  
  • intelmq.lib.exceptions:
    •  
  • InvalidKey: Add KeyError as parent class.
    •  
  • DecodingError: Added, string representation has all relevant information on the decoding error, including encoding, reason and the affected string (#1494).
    •  
  • intelmq.lib.pipeline:
    •  
  • Decode messages in Pipeline.receive not in the implementation's _receive so that the internal counter is correct in case of decoding errors (#1494).
    •  
  • intelmq.lib.utils:
    •  
  • decode: Raise new DecodingError if decoding fails.
    •  
"},{"location":"changelog/#harmonization_1","title":"Harmonization","text":"
     
  • protocol.transport: Adapt regular expression to allow the value nvp-ii (protocol 11).
    •  
"},{"location":"changelog/#bots_17","title":"Bots","text":""},{"location":"changelog/#collectors_13","title":"Collectors","text":"
     
  • intelmq.bots.collectors.mail.collector_mail_attach:
    •  
  • Fix handling of deprecated parameter name attach_unzip.
    •  
  • Fix handling of attachments without filenames (#1538).
    •  
  • intelmq.bots.collectors.stomp.collector: Fix compatibility with stomp.py versions > 4.1.20 and catch errors on shutdown.
    •  
  • intelmq.bots.collectors.microsoft:
    •  
  • Update REQUIREMENTS.txt temporarily fixing deprecated Azure library (#1530, PR#1532).
    •  
  • intelmq.bots.collectors.microsoft.collector_interflow: Add method for printing the file list.
    •  
"},{"location":"changelog/#parsers_15","title":"Parsers","text":"
     
  • intelmq.bots.parsers.cymru.parser_cap_program: Support for protocol 11 (nvp-ii) and conficker type.
    •  
  • intelmq.bots.parsers.taichung.parser: Support more types/classifications:
    •  
  • Application Compromise: Apache vulnerability & SQL injections
    •  
  • Brute-force: MSSQL & SSH password guess attacks; Office 365, SSH & SIP attacks
    •  
  • C2 Sever: Attack controller
    •  
  • DDoS
    •  
  • DoS: DNS, DoS, Excess connection
    •  
  • IDS Alert / known vulnerability exploitation: backdoor
    •  
  • Malware: Malware Proxy
    •  
  • Warn on new unknown types.
    •  
  • intelmq.bots.parsers.bitcash.parser: Removed as feed is discontinued.
    •  
  • intelmq.bots.parsers.fraunhofer.parser_ddosattack_cnc and intelmq.bots.parsers.fraunhofer.parser_ddosattack_target: Removed as feed is discontinued.
    •  
  • intelmq.bots.parsers.malwaredomains.parser: Correctly classify C&C and phishing events.
    •  
  • intelmq.bots.parsers.shadowserver.parser: More verbose error message for missing report specification (#1507).
    •  
  • intelmq.bots.parsers.n6.parser_n6stomp: Always add n6 field name as malware.name independent of category.
    •  
  • intelmq.bots.parsers.anubisnetworks: Update parser with new data format.
    •  
  • intelmq.bots.parsers.bambenek: Add new feed URLs with Host faf.bambenekconsulting.com (#1525, PR#1526).
    •  
  • intelmq.bots.parsers.abusech.parser_ransomware: Removed, as the feed is discontinued (#1537).
    •  
  • intelmq.bots.parsers.nothink.parser: Removed, as the feed is discontinued (#1537).
    •  
  • intelmq.bots.parsers.n6.parser: Remove not allowed characters in the name field for malware.name and write original value to event_description.text instead.
    •  
"},{"location":"changelog/#experts_16","title":"Experts","text":"
     
  • intelmq.bots.experts.cymru_whois.lib: Fix parsing of AS names with Unicode characters.
    •  
"},{"location":"changelog/#outputs_12","title":"Outputs","text":"
     
  • intelmq.bots.outputs.mongodb:
    •  
  • Set default port 27017.
    •  
  • Use different authentication mechanisms per MongoDB server version to fix compatibility with server version >= 3.4 (#1439).
    •  
"},{"location":"changelog/#documentation_16","title":"Documentation","text":"
     
  • Feeds:
    •  
  • Remove unavailable feed Abuse.CH Zeus Tracker.
    •  
  • Remove the field status, offline feeds should be removed.
    •  
  • Add a new field public to differentiate between private and public feeds.
    •  
  • Adding documentation URLs to nearly all feeds.
    •  
  • Remove unavailable Bitcash.cz feed.
    •  
  • Remove unavailable Fraunhofer DDos Attack feeds.
    •  
  • Remove unavailable feed Abuse.CH Ransomware Tracker (#1537).
    •  
  • Update information on Bambenek Feeds, many require a license now (#1525).
    •  
  • Remove discontinued Nothink Honeypot Feeds (#1537).
    •  
  • Developers Guide: Fix the instructions for /opt/intelmq file permissions.
    •  
"},{"location":"changelog/#packaging_8","title":"Packaging","text":"
     
  • Patches: fix-logrotate-path.patch: also include path to rotated file in patch.
    •  
  • Fix paths from /opt to LSB for setup.py and contrib/logrotate/intelmq in build process (#1500).
    •  
  • Add runtime dependency debianutils for the program which, which is required for intelmqctl.
    •  
"},{"location":"changelog/#tests_14","title":"Tests","text":"
     
  • Dropping Travis tests for 3.4 as required libraries dropped 3.4 support.
    •  
  • intelmq.tests.bots.experts.cymru_whois:
    •  
  • Drop missing ASN test, does not work anymore.
    •  
  • IPv6 to IPv4 test: Test for two possible results.
    •  
  • intelmq.lib.test: Fix compatibility of logging capture with Python >= 3.7 by reworking the whole process (#1342).
    •  
  • intelmq.bots.collectors.tcp.test_collector: Removing custom mocking and bot starting, not necessary anymore.
    •  
  • Added tests for intelmq.bin.intelmqctl.IntelMQProcessManager._interpret_commandline.
    •  
  • Fix and split tests.bots.experts.ripe.test_expert.test_ripe_stat_error_json.
    •  
  • Added tests for invalid encodings in input messages in intelmq.tests.lib.test_bot and intelmq.tests.lib.test_pipeline (#1494).
    •  
  • Travis: Explicitly enable RabbitMQ management plugin.
    •  
  • intelmq.tests.lib.test_message: Fix usage of the parameter blacklist for Message hash tests (#1539).
    •  
"},{"location":"changelog/#tools_12","title":"Tools","text":"
     
  • intelmqsetup: Copy missing BOTS file to IntelMQ's root directory (#1498).
    •  
  • intelmq_gen_docs: Feed documentation generation: Handle missing/empty parameters.
    •  
  • intelmqctl:
    •  
  • IntelMQProcessManager: For the status of running bots also check the bot ID of the commandline and ignore the path of the executable (#1492).
    •  
  • IntelMQController: Fix exit codes of check command for JSON output (now 0 on success and 1 on error, was swapped, #1520).
    •  
  • intelmqdump:
    •  
  • Handle base64-type messages for show, editor and recovery actions.
    •  
"},{"location":"changelog/#contrib_7","title":"Contrib","text":"
     
  • intelmq/bots/experts/asn_lookup/update-asn-data: Use pyasn_util_download.py to download the data instead from RIPE, which cannot be parsed currently (#1517, PR#1518, https://github.com/hadiasghari/pyasn/issues/62).
    •  
"},{"location":"changelog/#known-issues_15","title":"Known issues","text":"
     
  • HTTP stream collector: retry on regular connection problems? (#1435).
    •  
  • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952).
    •  
  • Reverse DNS: Only first record is used (#877).
    •  
  • Corrupt dump files when interrupted during writing (#870).
    •  
"},{"location":"changelog/#212-2020-01-28","title":"2.1.2 (2020-01-28)","text":""},{"location":"changelog/#core_17","title":"Core","text":"
     
  • __init__: Resolve absolute path for STATE_FILE_PATH variable (resolves ..).
    •  
  • intelmq.lib.utils:
    •  
  • log: Do not raise an exception if logging to neither file nor syslog is requested.
    •  
  • logging StreamHandler: Colorize all warning and error messages red.
    •  
  • logging FileHandler: Strip all shell colorizations from the messages (#1436).
    •  
  • intelmq.lib.message:
    •  
  • Message.to_json: Set sort_keys=True to get reproducible results.
    •  
  • drop_privileges: Handle situations where the user or group intelmq does not exist.
    •  
  • intelmq.lib.pipeline:
    •  
  • Amqp._send and Amqp._acknowledge: Log traceback in debug mode in case of errors and necessary re-connections.
    •  
  • Amqp._acknowledge: Reset delivery tag if acknowledge was successful.
    •  
"},{"location":"changelog/#bots_18","title":"Bots","text":""},{"location":"changelog/#collectors_14","title":"Collectors","text":"
     
  • intelmq.bots.collectors.misp.collector:
    •  
  • Add compatibility with current pymisp versions and versions released after January 2020 (PR #1468).
    •  
"},{"location":"changelog/#parsers_16","title":"Parsers","text":"
     
  • intelmq.bots.parsers.shadowserver.config: Add some missing fields for the feed accessible-rdp (#1463).
    •  
  • intelmq.bots.parsers.shadowserver.parser:
    •  
  • Feed-detection based on file names: The prefixed date is optional now.
    •  
  • Feed-detection based on file names: Re-detect feed for every report received (#1493).
    •  
"},{"location":"changelog/#experts_17","title":"Experts","text":"
     
  • intelmq.bots.experts.national_cert_contact_certat: Handle empty responses by server (#1467).
    •  
  • intelmq.bots.experts.maxmind_geoip: The script update-geoip-data now requires a license key as second parameter because of upstream changes (#1484)).
    •  
"},{"location":"changelog/#outputs_13","title":"Outputs","text":"
     
  • intelmq.bots.outputs.restapi.output: Fix logging of response body if response status code was not ok.
    •  
"},{"location":"changelog/#documentation_17","title":"Documentation","text":"
     
  • Remove some hardcoded /opt/intelmq/ paths from code comments and program outputs.
    •  
"},{"location":"changelog/#packaging_9","title":"Packaging","text":"
     
  • debian/rules: Only replace /opt/intelmq/ with LSB-paths in some certain files, not the whole tree, avoiding wrong replacements.
    •  
  • debian/rules and debian/intelmq.install: Do install the examples configuration directly instead of working around the abandoned examples directory.
    •  
"},{"location":"changelog/#tests_15","title":"Tests","text":"
     
  • lib/test_utils: Skip some tests on Python 3.4 because contextlib.redirect_stdout and contextlib.redirect_sterr are not supported on this version.
    •  
  • Travis: Stop running tests with all optional dependencies on Python 3.4, as more and more libraries are dropping support for it. Tests on the core and code without non-optional requirements are not affected.
    •  
  • tests.bots.parsers.html_table: Make tests independent of current year.
    •  
"},{"location":"changelog/#tools_13","title":"Tools","text":"
     
  • intelmqctl upgrade-config: Fix missing substitution in error message \"State file %r is not writable.\".
    •  
"},{"location":"changelog/#known-issues_16","title":"Known issues","text":"
     
  • bots trapped in endless loop if decoding of raw message fails (#1494)
    •  
  • intelmqctl status of processes: need to check bot id too (#1492)
    •  
  • MongoDB authentication: compatibility on different MongoDB and pymongo versions (#1439)
    •  
  • ctl: shell colorizations are logged (#1436)
    •  
  • http stream collector: retry on regular connection problems? (#1435)
    •  
  • tests: capture logging with context manager (#1342)
    •  
  • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
    •  
  • n6 parser: mapping is modified within each run (#905)
    •  
  • reverse DNS: Only first record is used (#877)
    •  
  • Corrupt dump files when interrupted during writing (#870)
    •  
"},{"location":"changelog/#211-2019-11-11","title":"2.1.1 (2019-11-11)","text":""},{"location":"changelog/#configuration_6","title":"Configuration","text":"
     
  • Default configuration:
    •  
  • Remove discontinued feed \"Feodo Tracker Domains\" from default configuration.
    •  
  • Add \"Feodo Tracker Browse\" feed to default configuration.
    •  
"},{"location":"changelog/#core_18","title":"Core","text":"
     
  • intelmq.lib.pipeline: AMQP: using port 15672 as default (like RabbitMQ's defaults) for the monitoring interface for getting statistical data (intelmqctl_rabbitmq_monitoring_url).
    •  
  • intelmq.lib.upgrades: Added a generic upgrade function for harmonization, checking of all message types, it's fields and their types.
    •  
  • intelmq.lib.utils:
    •  
  • TimeoutHTTPAdapter: A subclass of requests.adapters.HTTPAdapter with the possibility to set the timeout per adapter.
    •  
  • create_request_session_from_bot: Use the TimeoutHTTPAdapter with the user-defined timeout. Previously the timeout was not functional.
    •  
"},{"location":"changelog/#bots_19","title":"Bots","text":""},{"location":"changelog/#parsers_17","title":"Parsers","text":"
     
  • intelmq.bots.parsers.shadowserver.parser: Fix logging message if the parameter feedname is not present.
    •  
  • intelmq.bots.parsers.shodan.parser: Also add field classification.identifier ('network-scan') in minimal mode.
    •  
  • intelmq.bots.parsers.spamhaus.parser_cert: Add support for category 'misc'.
    •  
  • intelmq.bots.parsers.cymru.parser_cap_program:
    •  
  • Add support for phishing events without URL.
    •  
  • Add support for protocols >= 143 (unassigned, experiments, testing, reserved), saving the number to extra, as the data would be bogus.
    •  
  • intelmq.bots.parsers.microsoft.parser_bingmurls:
    •  
  • Save the Tags data as source.geolocation.cc.
    •  
"},{"location":"changelog/#experts_18","title":"Experts","text":"
     
  • intelmq.bots.experts.modify.expert: Fix bug with setting non-string values (#1460).
    •  
"},{"location":"changelog/#outputs_14","title":"Outputs","text":"
     
  • intelmq.bots.outputs.smtp:
    •  
  • Allow non-existent field in text formatting by using a default value None instead of throwing errors.
    •  
  • Fix Authentication (#1464).
    •  
  • Fix sending to multiple recipients (#1464).
    •  
"},{"location":"changelog/#documentation_18","title":"Documentation","text":"
     
  • Feeds:
    •  
  • Fix configuration of Feodo Tracker Browse feed.
    •  
  • Bots:
    •  
  • Sieve expert: Document behavior of != with lists.
    •  
"},{"location":"changelog/#tests_16","title":"Tests","text":"
     
  • Adaption and extension of the test cases to the changes.
    •  
"},{"location":"changelog/#tools_14","title":"Tools","text":"
     
  • intelmq.bin.intelmqctl:
    •  
  • check: Check if running the upgrade function for harmonization is necessary.
    •  
  • upgrade-config: Run the upgrade function for harmonization.
    •  
  • intelmqctl restart did throw an error as the message for restarting was not defined (#1465).
    •  
"},{"location":"changelog/#known-issues_17","title":"Known issues","text":"
     
  • MongoDB authentication: compatibility on different MongoDB and pymongo versions (#1439)
    •  
  • ctl: shell colorizations are logged (#1436)
    •  
  • http stream collector: retry on regular connection problems? (#1435)
    •  
  • tests: capture logging with context manager (#1342)
    •  
  • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
    •  
  • n6 parser: mapping is modified within each run (#905)
    •  
  • reverse DNS: Only first record is used (#877)
    •  
  • Corrupt dump files when interrupted during writing (#870)
    •  
"},{"location":"changelog/#210-2019-10-15","title":"2.1.0 (2019-10-15)","text":""},{"location":"changelog/#core_19","title":"Core","text":"
     
  • intelmq.lib.harmonization:
    •  
  • Use correct parent classes.
    •  
  • Add DateTime.convert as interface for all existing conversion functions.
    •  
  • add DateTime.convert_from_format.
    •  
  • add DateTime.convert_from_format_midnight.
    •  
  • add DateTime.convert_fuzzy.
    •  
  • intelmq.lib.pipeline:
    •  
  • Redis: Use single connection client if calling bot is not multithreaded. Gives a small speed advantage.
    •  
  • Require the bot instance as parameter for all pipeline classes.
    •  
  • New internal variable _has_message to keep the state of the pipeline.
    •  
  • Split receive and acknowledge into public-facing and private methods.
    •  
  • Add reject_message method to the Pipeline class for explicit re-queue of messages.
    •  
  • AMQP:
       
    • Make exchange configurable.
      •  
    • If exchange is set, the queues are not declared, the queue name is for routing used by exchanges.
      •  
     
    •  
  • intelmq.lib.bot:
    •  
  • Log message after successful bot initialization, no log message anymore for ready pipeline.
    •  
  • Use existing current message if receive is called and the current message still exists.
    •  
  • Fix handling of received messaged after a SIGHUP that happened during a blocking receiving connection using explicit rejection (#1438).
    •  
  • New method _parse_common_parameters called before init to parse commonly used argument. Currently supported: extract_files.
    •  
  • intelmq.lib.test:
    •  
  • Fix the tests broker by providing the testing pipeline.
    •  
  • intelmq.lib.utils:
    •  
  • unzip:
       
    • new parameter return_names to optionally return the file names.
      •  
    • support for zip
      •  
    • new parameters try_zip, try_gzip and try_tar to control which compressions are tried.
      •  
    • rewritten to an iterative approach
      •  
     
    •  
  • add file_name_from_response to extract a file name from a Response object for downloaded files.
    •  
  • intelmq.lib.upgrades: Added v210_deprecations for deprecated parameters.
    •  
"},{"location":"changelog/#harmonization_2","title":"Harmonization","text":"
     
  • Add extra to reports.
    •  
"},{"location":"changelog/#bots_20","title":"Bots","text":""},{"location":"changelog/#collectors_15","title":"Collectors","text":"
     
  • intelmq.bots.collectors.http.collector_http:
    •  
  • More extensive usage of intelmq.lib.utils.unzip.
    •  
  • Save the file names in the report if files have been extracted form an archive.
    •  
  • intelmq.bots.collectors.rt.collector_rt:
    •  
  • Save ticket information/metadata in the extra fields of the report.
    •  
  • Support for RT 3.8 and RT 4.4.
    •  
  • New parameters extract_attachment and extract_download for generic archive extraction and consistency. The parameter unzip_attachment is deprecated.
    •  
  • intelmq.bots.collectors.mail.*: Save email information/metadata in the extra fields of the report. See the bots documentation for a complete list of provided data.
    •  
  • intelmq.bots.collectors.mail.collector_mail_attach: Check for existence/validity of the attach_regex parameter.
    •  
  • Use the lib's unzip function for uncompressing attachments and use the .
    •  
  • intelmq.bots.collectors.mail.collector_mail_url: Save the file name of the downloaded file as extra.file_name.
    •  
  • intelmq.bots.collectors.amqp.collector_amqp: New collector to collect data from (remote) AMQP servers, for bot IntelMQ as well as external data.
    •  
  • use default SSL context for client purposes, fixes compatibility with python < 3.6 if TLS is used.
    •  
"},{"location":"changelog/#parsers_18","title":"Parsers","text":"
     
  • intelmq.bots.parsers.html_table.parser:
    •  
  • New parameter \"html_parser\".
    •  
  • Use time conversion functions directly from intelmq.lib.harmonization.DateTime.convert.
    •  
  • Limit lxml dependency on 3.4 to < 4.4.0 (incompatibility).
    •  
  • intelmq.bots.parsers.netlab_360.parser: Add support for hajime scanners.
    •  
  • intelmq.bots.parsers.hibp.parser_callback: A new parser to parse data retrieved from a HIBP Enterprise Subscription.
    •  
  • intelmq.bots.parsers.shadowserver.parser:
    •  
  • Ability to detect the feed base on the reports's field extra.file_name, so the parameter feedname is no longer required and one configured parser can parse any feed (#1442).
    •  
"},{"location":"changelog/#experts_19","title":"Experts","text":"
     
  • Add geohash expert.
    •  
  • intelmq.bots.experts.generic_db_lookup.expert
    •  
  • new optional parameter engine with postgresql (default) and sqlite (new) as possible values.
    •  
"},{"location":"changelog/#outputs_15","title":"Outputs","text":"
     
  • Add intelmq.bots.outputs.touch.output.
    •  
  • intelmq.bots.outputs.postgresql.output:
    •  
  • deprecated in favor of intelmq.bots.outputs.sql.output
    •  
  • Compatibility shim will be available in the 2.x series.
    •  
  • intelmq.bots.outputs.sql.output added generic SQL output bot. Comparted to
    •  
  • new optional parameter engine with postgresql (default) and sqlite (new) as possible values.
    •  
  • intelmq.bots.outputs.stomp.output: New parameters message_hierarchical, message_jsondict_as_string, message_with_type, single_key.
    •  
"},{"location":"changelog/#documentation_19","title":"Documentation","text":"
     
  • Feeds:
    •  
  • Add ViriBack feed.
    •  
  • Add Have I Been Pwned Enterprise Callback.
    •  
  • intelmq.tests.bots.outputs.amqptopic.test_output: Added.
    •  
  • Move the documentation of most bots from separate README files to the central Bots.md and feeds.yaml files.
    •  
"},{"location":"changelog/#tests_17","title":"Tests","text":"
     
  • Travis:
    •  
  • Use UTC timezone.
    •  
  • Tests for utils.unzip.
    •  
  • Add a new asset: Zip archive with two files, same as with .tar.gz archive.
    •  
  • Added tests for the Mail Attachment & Mail URL collectors.
    •  
  • Ignore logging-tests on Python 3.7 temporarily (#1342).
    •  
"},{"location":"changelog/#tools_15","title":"Tools","text":"
     
  • intelmqctl:
    •  
  • Use green and red text color for some interactive output to indicate obvious errors or the absence of them.
    •  
  • intelmqdump:
    •  
  • New edit action v to modify a message saved in the dump (#1284).
    •  
"},{"location":"changelog/#contrib_8","title":"Contrib","text":"
     
  • malware name mapping:
    •  
  • Add support for MISP treat actors data, see it's README for more information.
       
    • And handle empty synonyms in misp's galxies data.
      •  
     
    •  
  • Move apply-Script to the new EventDB directory
    •  
  • EventDB: Scripts for applying malware name mapping and domain suffixes to an EventDB.
    •  
"},{"location":"changelog/#known-issues_18","title":"Known issues","text":"
     
  • MongoDB authentication: compatibility on different MongoDB and pymongo versions (#1439)
    •  
  • ctl: shell colorizations are logged (#1436)
    •  
  • http stream collector: retry on regular connection problems? (#1435)
    •  
  • tests: capture logging with context manager (#1342)
    •  
  • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
    •  
  • n6 parser: mapping is modified within each run (#905)
    •  
  • reverse DNS: Only first record is used (#877)
    •  
  • Corrupt dump files when interrupted during writing (#870)
    •  
"},{"location":"changelog/#202-2019-10-14","title":"2.0.2 (2019-10-14)","text":""},{"location":"changelog/#core_20","title":"Core","text":"
     
  • intelmq.lib.bot.CollectorBot: Support the deprecated parameter feed until version 2.2 as the documentation was not properly updated (#1445).
    •  
  • intelmq.lib.bot.Bot:
    •  
  • _dump_message: Wait for up to 60 seconds instead of 50 if the dump file is locked (the log message was said 60, but the code was for only 50).
    •  
  • intelmq.lib.upgrades.v202_fixes
    •  
  • Migration of deprecated parameter feed for Collectors.
    •  
  • Ripe expert parameter query_ripe_stat_ip was not correctly configured in v110_deprecations, now use query_ripe_stat_asn as default if it does not exist.
    •  
  • intelmq.lib.upgrades.v110_deprecations: Fix upgrade of ripe expert configuration.
    •  
  • intelmq.lib.bot_debugger:
    •  
  • Fix handling of empty messages generated by parser when user wanted to show the result by \"--show-sent\" flag.
    •  
  • Fix handling of sent messages for bots using the path_permissive parameter (#1453).
    •  
  • intelmq.lib.pipeline.Amqp:
    •  
  • use default SSL context for client purposes, fixes compatibility with python < 3.6 if TLS is used.
    •  
  • Reconnect once on sending messages if disconnect detected.
    •  
"},{"location":"changelog/#bots_21","title":"Bots","text":""},{"location":"changelog/#collectors_16","title":"Collectors","text":"
     
  • intelmq.bots.collectors.api.collector_api:
    •  
  • Handle non-existing IO loop in shutdown.
    •  
  • Close socket on shutdown, fixes reloading.
    •  
  • Marked as non-threadable.
    •  
  • intelmq.bots.collectors.rt.collector_rt: Check for matching URLs if no attachment_regex is given.
    •  
  • intelmq.bots.collectors.stomp.collector_stomp: Handle disconnects by actively reconnecting.
    •  
"},{"location":"changelog/#parsers_19","title":"Parsers","text":"
     
  • intelmq.bots.cymru.parser_cap_program: Fix parsing of the new $certname_$date.txt report format (#1443):
    •  
  • Support protocol ICMP.
    •  
  • Fix error message for unsupported protocols.
    •  
  • Support fields destination_port_numbers, port.
    •  
  • Support for all proxy types without ports.
    •  
  • Use Country Code of AS as source.geolocation.cc.
    •  
  • Support for 'scanner' and 'spam' categories.
    •  
  • Handle bogus lines with missing separator.
    •  
  • Fix bug preventing use of old format after using the new format.
    •  
  • Handle postfix (total_count:..) for destination port numbers.
    •  
"},{"location":"changelog/#experts_20","title":"Experts","text":"
     
  • intelmq.bots.experts.cymru_whois.expert: Add optional parameter overwrite, current behavior was True, default if not given is True now, will change to False in 3.0.0 (#1452, #1455).
    •  
  • intelmq.bots.experts.modify.expert: Add optional parameter overwrite, current behavior was True, default if not given is True now, will change to False in 3.0.0 (#1452, #1455).
    •  
  • intelmq.bots.experts.reverse_dns.expert: Add optional parameter overwrite, current behavior was True, default if not given is True now, will change to False in 3.0.0 (#1452, #1455).
    •  
"},{"location":"changelog/#outputs_16","title":"Outputs","text":"
     
  • intelmq.bots.outputs.amqptopic.output: use default SSL context for client purposes, fixes compatibility with python < 3.6 if TLS is used.
    •  
"},{"location":"changelog/#packaging_10","title":"Packaging","text":"
     
  • Rules:
    •  
  • Exclude intelmqsetup tool in packages
    •  
  • Include update-rfiprisk-data in packages
    •  
"},{"location":"changelog/#tests_18","title":"Tests","text":"
     
  • Tests for intelmq.lib.upgrades.v202_fixes.
    •  
  • Tests for intelmq.lib.upgrades.v110_deprecations.
    •  
  • Extended tests for intelmq.bots.parser.cymru.parser_cap_program.
    •  
"},{"location":"changelog/#tools_16","title":"Tools","text":"
     
  • intelmqctl:
    •  
  • More and more precise logging messages for botnet starting and restarting, enable and disable.
    •  
  • No error message for disabled bots on botnet reload.
    •  
  • Fix upgrade-conf is state file is empty or not existing.
    •  
  • Use arpgarse's store_true action for flags instead of store_const.
    •  
  • If the loading of the defaults configuration failed, a variable definition was missing and causing an exception (#1456).
    •  
"},{"location":"changelog/#contrib_9","title":"Contrib","text":"
     
  • Check MK Statistics Cronjob:
    •  
  • Use statistics_* parameters.
    •  
  • Make file executable
    •  
  • Handle None values in *.temporary.* keys and treat them as 0.
    •  
  • systemd:
    •  
  • Add PIDFile parameter to service file.
    •  
"},{"location":"changelog/#known-issues_19","title":"Known issues","text":"
     
  • MongoDB authentication: compatibility on different MongoDB and pymongo versions (#1439)
    •  
  • ctl: shell colorizations are logged (#1436)
    •  
  • http stream collector: retry on regular connection problems? (#1435)
    •  
  • tests: capture logging with context manager (#1342)
    •  
  • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
    •  
  • n6 parser: mapping is modified within each run (#905)
    •  
  • reverse DNS: Only first record is used (#877)
    •  
  • Corrupt dump files when interrupted during writing (#870)
    •  
"},{"location":"changelog/#201-2019-08-23","title":"2.0.1 (2019-08-23)","text":""},{"location":"changelog/#core_21","title":"Core","text":"
     
  • intelmq.lib.harmonization:
    •  
  • IPAddress: Remove Scope/Zone IDs for IPv6 addresses in sanitation.
    •  
  • All types: Handle None for validation and sanitation gracefully.
    •  
  • intelmq.lib.bot:
    •  
  • fix parameters of ParserBot and CollectorBot constructors, allowing intelmqctl run with these bots again (#1414).
    •  
  • Also run rate_limit after retry counter reset (#1431).
    •  
  • __version_info__:
    •  
  • is now available in the top level module.
    •  
  • uses integer values now instead of strings for numerical version parts
    •  
  • Also provide (empty) ROOT_DIR for non-pip installations.
    •  
  • intelmq.lib.upgrades: New library file upgrades with upgrade functions.
    •  
  • intelmq.lib.utils:
    •  
  • New function setup_list_logging for intelmqctl check an possibly others.
       
    • Fix return values (#1423).
      •  
     
    •  
  • New function version_smaller for version comparisons.
    •  
  • New function lazy_int for version conversions.
    •  
  • parse_logline: Handle thread IDs.
    •  
  • log takes a new argument logging_level_stream for the logging level of the console handler.
    •  
  • New constant LOG_FORMAT_SIMPLE, used by intelmqctl.
    •  
  • New function write_configuration to write dicts to files in the correct json formatting.
    •  
  • New function create_request_session_from_bot.
    •  
  • intelmq.lib.pipeline:
    •  
  • AMQP:
       
    • Actually use source/destination_pipeline_amqp_virtual_host parameter.
      •  
    • Support for SSL with source/destination_pipeline_ssl parameter.
      •  
     
    •  
  • pipeline base class: add missing dummy methods.
    •  
  • Add missing return types.
    •  
  • Redis: Evaluate return parameter of queue/key deletion.
    •  
  • Variable STATE_FILE_PATH added.
    •  
"},{"location":"changelog/#development_7","title":"Development","text":"
     
  • intelmq.bin.intelmq_gen_docs: For yaml use safe_load instead of unsafe load.
    •  
"},{"location":"changelog/#harmonization_3","title":"Harmonization","text":"
     
  • IPAddress type: Remove Scope/Zone IDs for IPv6 addresses in sanitation.
    •  
  • TLP: Sanitation handles now more cases: case-insensitive prefixes and arbitrary whitespace between the prefix and the value (#1420).
    •  
"},{"location":"changelog/#bots_22","title":"Bots","text":""},{"location":"changelog/#collectors_17","title":"Collectors","text":"
     
  • intelmq.bots.collectors.http.collector_http: Use utils.create_request_session_from_bot.
    •  
  • intelmq.bots.collectors.http.collector_http_stream: Use utils.create_request_session_from_bot and thus fix some retries on connection timeouts.
    •  
  • intelmq.bots.collectors.mail.collector_mail_url: Use utils.create_request_session_from_bot.
    •  
  • intelmq.bots.collectors.microsoft.collector_interflow: Use utils.create_request_session_from_bot and thus fix retries on connection timeouts.
    •  
  • intelmq.bots.collectors.rt.collector_rt: Use utils.create_request_session_from_bot and thus fix retries on connection timeouts.
    •  
  • intelmq.bots.collectors.twitter.collector_twitter: Use utils.create_request_session_from_bot and thus fix retries on connection timeouts for non-twitter connections.
    •  
"},{"location":"changelog/#parsers_20","title":"Parsers","text":"
     
  • intelmq.bots.parsers.n6.parser_n6stomp: use malware-generic instead of generic-n6-drone for unknown infected system events.
    •  
  • intelmq.bots.parsers.abusech.parser_ip: Support LastOnline column in feodo feed (#1400) and use it for time.source if available.
    •  
  • Use lower case malware names as default, should not make a difference in practice.
    •  
  • Fix handling of CSV header for feodotracker (#1417, #1418).
    •  
  • intelmq.bots.parsers.netlab_360.parser: Detect feeds with https:// too.
    •  
"},{"location":"changelog/#experts_21","title":"Experts","text":"
     
  • intelmq.bots.experts.generic_db_lookup: Recommend psycopg2-binary package.
    •  
  • intelmq.bots.experts.modify.expert:
    •  
  • Compile regular expressions (all string rules) at initialization, improves the speed.
    •  
  • Warn about old configuration style deprecation.
    •  
  • intelmq.bots.experts.do_portal.expert:
    •  
  • Use utils.create_request_session_from_bot and thus fix retries on connection timeouts (#1432).
    •  
  • Treat \"502 Bad Gateway\" as timeout which can be retried.
    •  
  • intelmq.bots.experts.ripe.expert: Use utils.create_request_session_from_bot and thus fix retries on connection timeouts.
    •  
  • intelmq.bots.experts.url2fqdn.expert: Support for IP addresses in hostnames (#1416).
    •  
  • intelmq.bots.experts.national_cert_contact_certat.expert: Use utils.create_request_session_from_bot and thus fix retries on connection timeouts.
    •  
"},{"location":"changelog/#outputs_17","title":"Outputs","text":"
     
  • intelmq.bots.outputs.postgresql: Recommend psycopg2-binary package.
    •  
  • intelmq.bots.outputs.amqptopic:
    •  
  • Shutdown: Close connection only if connection exists.
    •  
  • Add support for pika > 1. Pika changed the way it indicates (Non-)Acknowledgments of sent messages.
    •  
  • Gracefully handle unroutable messages and give advice.
    •  
  • Support for connections without authentication.
    •  
  • Replace deprecated parameter type with exchange_type for exchange_declare, supporting pika >= 0.11 (#1425).
    •  
  • New parameters message_hierarchical_output, message_with_type, message_jsondict_as_string.
    •  
  • New parameter use_ssl for SSL connections.
    •  
  • New parameter single_key for sending single fields instead of the full event.
    •  
  • intelmq.bots.outputs.mongodb.output: Support for pymongo >= 3.0.0 (#1063, PR#1421).
    •  
  • intelmq.bots.outputs.file: time.* field serialization: support for microseconds.
    •  
  • intelmq.bots.outputs.mongodb.output: Support for authentication in pymongo >= 3.5 (#1062).
    •  
  • intelmq.bots.outputs.restapi.output: Use utils.create_request_session_from_bot and thus fix retries on connection timeouts.
    •  
"},{"location":"changelog/#documentation_20","title":"Documentation","text":"
     
  • Add certbund-contact to the ecosystem document.
    •  
  • Rename the IDEA expert to \"IDEA Converter\".
    •  
  • Add the new configuration upgrade function to the docs.
    •  
  • User Guide:
    •  
  • Clarify on Uninstallation
    •  
"},{"location":"changelog/#packaging_11","title":"Packaging","text":"
     
  • Do not execute the tcp collector tests during Debian and Ubuntu builds as they fail there.
    •  
"},{"location":"changelog/#tests_19","title":"Tests","text":"
     
  • intelmq.lib.test: Disable statistics for test runs of bots.
    •  
  • contrib.malware_name_mapping: Added tests.
    •  
  • Travis: Also run tests of contrib.
    •  
"},{"location":"changelog/#tools_17","title":"Tools","text":"
     
  • intelmqsetup: Only change directory ownerships if necessary.
    •  
  • intelmqctl:/**---
    •  
  • Provide new command upgrade-conf to upgrade configuration to a newer version.
       
    • Makes backups of configurations files on its own.
      •  
    • Also checks for previously skipped or new functions of older versions and catches up.
      •  
     
    •  
  • Provides logging level on class layer.
    •  
  • Fix -q flag for intelmqctl list queues by renaming its alternative name to --non-zero to avoid a name collision with the global --quiet parameter.
    •  
  • For console output the string intelmqctl: at the beginning of each line is no longer present.
    •  
  • check: Support for the state file added. Checks if it exists and all upgrade functions have been executed successfully.
    •  
  • Waits for up to 2 seconds when stopping a bot (#1434).
    •  
  • Exits early on restart when stopping a bot did not work (#1434).
    •  
  • intelmqctl run process -m debugging: Mock acknowledge method if incoming message is mocked too, otherwise a different message is acknowledged.
    •  
  • Queue listing for AMQP: Support non-default monitoring URLs, see User-Guide.
    •  
"},{"location":"changelog/#contrib_10","title":"Contrib","text":"
     
  • logcheck rules: Adapt ignore rule to cover the instance IDs of bot names.
    •  
  • malware name mapping:
    •  
  • Ignore lines in mapping starting with '#'.
    •  
  • Optionally include malpedia data.
    •  
  • Fix command line parsing for not arguments (#1427).
    •  
  • bash-completion: Support for intelmqctl upgrade-config added.
    •  
"},{"location":"changelog/#known-issues_20","title":"Known issues","text":"
     
  • http stream collector: retry on regular connection problems? (#1435)
    •  
  • tests: capture logging with context manager (#1342)
    •  
  • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
    •  
  • n6 parser: mapping is modified within each run (#905)
    •  
  • reverse DNS: Only first record is used (#877)
    •  
  • Corrupt dump files when interrupted during writing (#870)
    •  
"},{"location":"changelog/#200-2019-05-22","title":"2.0.0 (2019-05-22)","text":"
See also the changelog for 2.0.0.beta1 below.
"},{"location":"changelog/#configurations","title":"Configurations","text":"
     
  • Defaults: New parameters statistics_host, statistics_port, statistics_databasae, statistics_password for statistics redis database (#1402).
    •  
"},{"location":"changelog/#core_22","title":"Core","text":"
     
  • Add more and fix some existing type annotations.
    •  
  • intelmq.lib.bot:
    •  
  • Use statistics_* parameters for bot's statistics (#1402).
    •  
  • Introduce collector_empty_process for collectors with an empty process() method, hardcoded 1s minimum sleep time, preventing endless loops, causing high load (#1364).
    •  
  • Allow to disable multithreading by initialization parameter, used by intelmqctl / the bot debugger (#1403).
    •  
  • intelmq.lib.pipeline: redis: OOM can also be low memory, add this to log message (#1405).
    •  
  • intelmq.lib.harmonization: ClassificationType: Update RSIT mapping (#1380):
    •  
  • replace botnet drone with infected-system
    •  
  • replace infected system with infected-system
    •  
  • replace ids alert with ids-alert
    •  
  • replace c&c with c2server
    •  
  • replace malware configuration with malware-configuration
    •  
  • sanitize replaces these values on the fly
    •  
  • Allow using non-opt/ (LSB) paths with environment variable INTELMQ_PATHS_NO_OPT.
    •  
  • Disable/disallow threading for all collectors and some other bots.
    •  
"},{"location":"changelog/#development_8","title":"Development","text":"
     
  • Applied isort to all core files and core-related test files, sorting the imports there (every thing except bots and bots' tests).
    •  
"},{"location":"changelog/#harmonization_4","title":"Harmonization","text":"
     
  • See the Core section for the changes in the allowed values for classification.type.
    •  
"},{"location":"changelog/#bots_23","title":"Bots","text":"
     
  • Use the new RSIT types in several bots, see above
    •  
"},{"location":"changelog/#parsers_21","title":"Parsers","text":"
     
  • intelmq.bots.parsers.spamhaus.parser_cert: Added support for extortion events.
    •  
"},{"location":"changelog/#experts_22","title":"Experts","text":"
     
  • added intelmq.bots.experts.do_portal.expert.
    •  
"},{"location":"changelog/#outputs_18","title":"Outputs","text":"
     
  • intelmq.bots.outputs.elasticsearch.output: Support for TLS added (#1406).
    •  
  • intelmq.bots.outputs.tcp.output: Support non-intelmq counterparts again. New parameter counterpart_is_intelmq, see NEWS.md for more information (#1385).
    •  
"},{"location":"changelog/#packaging_12","title":"Packaging","text":"
     
  • Update IntelMQ path fix patch after INTELMQ_PATHS_NO_OPT introduction, provide INTELMQ_PATHS_OPT environment variable for packaged instances.
    •  
"},{"location":"changelog/#tests_20","title":"Tests","text":"
     
  • test_conf: For yaml use safe_load instead of unsafe load.
    •  
  • Travis: Switch distribution from trusty to xenial, adapt scripts.
    •  
  • Add Python 3.7 to tests.
    •  
  • Don't use Cerberus 1.3 because of https://github.com/pyeve/cerberus/issues/489
    •  
  • Add tests for intelmqctl.lib.upgrades.
    •  
"},{"location":"changelog/#tools_18","title":"Tools","text":"
     
  • intelmqdump: Fix creation of pipeline object by providing a logger.
    •  
  • intelmqctl: Disable multithreading for interactive runs / the bot debugger (#1403).
    •  
"},{"location":"changelog/#known-issues_21","title":"Known issues","text":"
     
  • tests: capture logging with context manager (#1342)
    •  
  • pymongo 3.0 deprecates used insert method  (#1063)
    •  
  • pymongo >= 3.5: authentication changes  (#1062)
    •  
  • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
    •  
  • n6 parser: mapping is modified within each run (#905)
    •  
  • reverse DNS: Only first record is used (#877)
    •  
  • Corrupt dump files when interrupted during writing (#870)
    •  
"},{"location":"changelog/#200beta1-2019-04-10","title":"2.0.0.beta1 (2019-04-10)","text":"
There are some features considered as beta and marked as such in the documentation, do not use them in production yet.
"},{"location":"changelog/#removals-of-deprecated-code","title":"Removals of deprecated code:","text":"
     
  • Removed compatibility shim intelmq.bots.collectors.n6.collector_stomp, use intelmq.bots.collectors.stomp.collector instead (see #1124).
    •  
  • Removed compatibility shim intelmq.bots.parsers.cymru_full_bogons.parser, use intelmq.bots.parsers.cymru.parser_full_bogons instead.
    •  
  • Removed compatibility shim handling deprecated parameter feed for collectors. Use name instead.
    •  
  • Removed deprecated and unused method intelmq.lib.pipeline.Pipeline.sleep.
    •  
  • Removed support for deprecated parameter query_ripe_stat in intelmq.bots.experts.ripe.expert, use query_ripe_stat_asn and query_ripe_stat_ip instead (#1291).
    •  
  • Removed deprecated and unused function intelmq.lib.utils.extract_tar.
    •  
"},{"location":"changelog/#core_23","title":"Core","text":"
     
  • lib/pipeline:
    •  
  • Allow setting the broker of source and destination independently.
    •  
  • Support for a new AMQP broker. See User Guide for configuration. (#1179)
    •  
  • lib/bot:
    •  
  • Dump messages locks the dump file using Unix file locks (#574).
    •  
  • Print idle/rate limit time also in human readable format (#1332).
    •  
  • set_request_parameters: Use {} as default proxy value instead of None. Allows updating of existing proxy dictionaries.
    •  
  • Bots drop privileges if they run as root.
    •  
  • Save statistics on successfully and failed processed messages in the redis database 3.
    •  
  • lib/utils
    •  
  • Function unzip to extract files from gz-zipped and/or tar-archives.
    •  
  • New class ListHandler: new handler for logging purpose which saves the messages in a list.
    •  
  • Add function seconds_to_human.
    •  
  • Add function drop_privileges.
    •  
  • parse_relative: Strip string before parsing.
    •  
  • parse_logline: Do not convert the timestamps to UTC, leave them as is.
    •  
  • lib/cache:
    •  
  • Allow ttl to be None explicitly.
    •  
  • Overwrite existing cache keys in the database instead of discarding the new data.
    •  
  • lib/bot:
    •  
  • Basic, but easy-to-configure multi-threading using python's threading library. See the User-Guide for more information (#111, #186).
    •  
  • bin/intelmqctl:
    •  
  • Support for Supervisor as process manager (#693, #1360).
    •  
"},{"location":"changelog/#development_9","title":"Development","text":"
     
  • upgraded all files to python3-only syntax, e.g. use super() instead of super(..., ...) in all files. Migration from old to new string formatting has not been applied if the resulting code would be longer.
    •  
"},{"location":"changelog/#bots_24","title":"Bots","text":""},{"location":"changelog/#collectors_18","title":"Collectors","text":"
     
  • added intelmq.bots.parsers.opendxl.collector (#1265).
    •  
  • added intelmq.bots.collectors.api: collecting data using an HTTP API (#123, #1187).
    •  
  • added intelmq.bots.collectors.rsync (#1286).
    •  
  • intelmq.bots.collectors.http.collector_http:
    •  
  • Add support for uncompressing of gz-zipped-files (#1270).
    •  
  • Add time-delta support for time formatted URLs (#1366).
    •  
  • intelmq.collectors.blueliv.collector_crimeserver: Allow setting the API URL by parameter (#1336).
    •  
  • intelmq.collectors.mail:
    •  
  • Use internal lib for functionality.
    •  
  • Add intelmq.bots.collectors.mail.collector_mail_body.
    •  
  • Support for ssl_ca_certificate parameter (#1362).
    •  
"},{"location":"changelog/#parsers_22","title":"Parsers","text":"
     
  • added intelmq.bots.parsers.mcafee.parser_atd (#1265).
    •  
  • intelmq.bots.parsers.generic.parser_csv:
    •  
  • New parameter columns_required to optionally ignore parse errors for columns.
    •  
  • added intelmq.bots.parsers.cert_eu.parser_csv (#1287).
    •  
  • Do not overwrite the local time.observation with the data from the feed. The feed's field 'observation time' is now saved in the field extra.cert_eu_time_observation.
    •  
  • Fix parsing of asn (renamed to source asn, source.asn internally) and handle existing feed.accuracy for parsing confidence.
    •  
  • Update columns and mapping to current (2019-04-02) data.
    •  
  • added intelmq.bots.parsers.surbl.surbl
    •  
  • added intelmq.bots.parsers.html_table (#1381).
    •  
  • intelmq.bots.parsers.netlab_360.parser: Handle empty lines containing blank characters (#1393).
    •  
  • intelmq.bots.parsers.n6.parser_n6stomp: Handle events without IP addresses.
    •  
  • intelmq.bots.parsers.cymru.parser_cap_program: Handle new feed format.
    •  
  • intelmq.bots.parsers.shadowserver:
    •  
  • Add support for the Accessible-FTP feed (#1391).
    •  
  • intelmq.bots.parsers.dataplane.parser:
    •  
  • Fix parse errors and log more context (#1396).
    •  
  • added intelmq.bots.parsers.fraunhofer.parser_ddosattack_cnc.py and intelmq.bots.parsers.fraunhofer.parser_ddosattack_target.py (#1373).
    •  
"},{"location":"changelog/#experts_23","title":"Experts","text":"
     
  • added intelmq.bots.experts.recordedfuture_iprisk (#1267).
    •  
  • added intelmq.bots.experts.mcafee.expert_mar (1265).
    •  
  • renamed intelmq.bots.experts.ripencc_abuse_contact.expert to intelmq.bots.experts.ripe.expert, compatibility shim will be removed in version 3.0.
    •  
  • Added support for geolocation information in ripe expert with a new parameter query_ripe_stat_geolocation (#1317).
    •  
  • Restructurize the expert and code de-duplicataion (#1384).
    •  
  • Handle '?' in geolocation country data (#1384).
    •  
  • intelmq.bots.experts.ripe.expert:
    •  
  • Use a requests session (#1363).
    •  
  • Set the requests parameters once per session.
    •  
  • intelmq.bots.experts.maxmind_geoip.expert: New parameter use_registered to use the registered country (#1344).
    •  
  • intelmq.bots.experts.filter.expert: Support for paths (#1208).
    •  
"},{"location":"changelog/#outputs_19","title":"Outputs","text":"
     
  • added intelmq.bots.experts.mcafee.output_esm (1265).
    •  
  • added intelmq.bots.outputs.blackhole (#1279).
    •  
  • intelmq.bots.outputs.restapi.expert:
    •  
  • Set the requests parameters once per session.
    •  
  • intelmq.bots.outputs.redis:
    •  
  • New parameter hierarchichal_output (#1388).
    •  
  • New parameter with_type.
    •  
  • intelmq.bots.outputs.amqptopic.output: Compatibility with pika 1.0.0 (#1084, #1394).
    •  
"},{"location":"changelog/#documentation_21","title":"Documentation","text":"
     
  • added documentation for feeds
    •  
  • CyberCrime Tracker
    •  
  • Feodo Tracker Latest
    •  
  • Feeds: Document abuse.ch URLhaus feed (#1379).
    •  
  • Install and Upgrading: Use intelmqsetup tool.
    •  
  • Added an ecosystem overview document describing related software.
    •  
"},{"location":"changelog/#tests_21","title":"Tests","text":"
     
  • Add tests of AMQP broker.
    •  
  • Travis: Change the ownership of /opt/intelmq to the current user.
    •  
"},{"location":"changelog/#tools_19","title":"Tools","text":"
     
  • intelmqctl check: Now uses the new ListHandler from utils to handle the logging in JSON output mode.
    •  
  • intelmqctl run: The message that a running bot has been stopped, is not longer a warning, but an informational message. No need to inform sysadmins about this intended behavior.
    •  
  • intelmqdump: Inspecting dumps locks the dump file using unix file locks (#574).
    •  
  • intelmqctl:
    •  
  • After the check if the program runs as root, it tries to drop privileges. Only if this does not work, a warning is shown.
    •  
  • intelmqsetup: New tool for initializing an IntelMQ environment.
    •  
"},{"location":"changelog/#contrib_11","title":"Contrib","text":"
     
  • malware_name_mapping:
    •  
  • Added the script apply_mapping_eventdb.py to apply the mapping to an EventDB.
    •  
  • Possibility to add local rules using the download tool.
    •  
  • check_mk:
    •  
  • Added scripts for monitoring queues and statistics.
    •  
"},{"location":"changelog/#known-issues_22","title":"Known issues","text":"
     
  • Multi-threaded bots require multiple SIGTERMs (#1403)
    •  
  • Stats can't be saved with AMQP if redis is password-protected (#1402)
    •  
  • Update taxonomies to current RSIT and vice-versa (#1380)
    •  
  • stomp collector bot constantly uses 100% of CPU (#1364)
    •  
  • tests: capture logging with context manager (#1342)
    •  
  • Consistent message counter log messages for all kind of bots (#1278)
    •  
  • pymongo 3.0 deprecates used insert method  (#1063)
    •  
  • pymongo >= 3.5: authentication changes  (#1062)
    •  
  • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
    •  
  • n6 parser: mapping is modified within each run (#905)
    •  
  • reverse DNS: Only first record is used (#877)
    •  
  • Corrupt dump files when interrupted during writing (#870)
    •  
"},{"location":"changelog/#112-2019-03-25","title":"1.1.2 (2019-03-25)","text":""},{"location":"changelog/#core_24","title":"Core","text":"
     
  • intelmq.lib.bot:
    •  
  • Bot.__handle_sighup: Handle exceptions in shutdown method of bots.
    •  
"},{"location":"changelog/#harmonization_5","title":"Harmonization","text":"
     
  • FQDN: Disallow : in FQDN values to prevent values like '10.0.0.1:8080' (#1235).
    •  
"},{"location":"changelog/#bots_25","title":"Bots","text":""},{"location":"changelog/#collectors_19","title":"Collectors","text":"
     
  • intelmq.bots.collectors.stomp.collector
    •  
  • Fix name of shutdown method, was ineffective in the past.
    •  
  • Ignore NotConnectedException errors on disconnect during shutdown.
    •  
  • intelmq.bots.collectors.mail.collector_mail_url: Decode body if it is bytes (#1367).
    •  
  • intelmq.bots.collectors.tcp.collector: Timeout added. More stable version.
    •  
"},{"location":"changelog/#parsers_23","title":"Parsers","text":"
     
  • intelmq.bots.parsers.shadowserver:
    •  
  • Add support for the Amplification-DDoS-Victim, HTTP-Scanners, ICS-Scanners and Accessible-Ubiquiti-Discovery-Service feeds (#1368, #1383)
    •  
  • intelmq.bots.parsers.microsoft.parser_ctip:
    •  
  • Workaround for mis-formatted data in networkdestinationipv4 field (since 2019-03-14).
    •  
  • Ignore \"hostname\" (\"destination.fqdn\") if it contains invalid data.
    •  
  • intelmq.bots.parsers.shodan.parser:
    •  
  • In minimal_mode:
       
    • Fix the parsing, previously only source.geolocation.cc and extra.shodan was correctly filled with information.
      •  
    • Add a classification.type = 'other' to all events.
      •  
    • Added tests for this mode.
      •  
     
    •  
  • Normal mode:
       
    • Fix the parsing of timestamp to `time.source in the normal mode, previously no timezone information has been added and thus every event raised an exception.
      •  
    • ISAKMP: Ignore isakmp.aggressive, as the content is same as isakmp or less.
      •  
     
    •  
  • intelmq.bots.parsers.abusech.parser_ip: Re-structure the bot and support new format of the changed \"Feodo Tracker Domains\" feed.
    •  
  • intelmq.bots.parsers.n6.parser:
    •  
  • Add parsing for fields \"confidence\", \"expires\" and \"source\".
    •  
  • Add support for type \"bl-other\" (category \"other\").
    •  
"},{"location":"changelog/#experts_24","title":"Experts","text":"
     
  • intelmq.bots.experts.sieve.expert: Fix key definition to allow field names with numbers (malware.hash.md5/sha1, #1371).
    •  
"},{"location":"changelog/#outputs_20","title":"Outputs","text":"
     
  • intelmq.bots.outputs.tcp.output: Timeout added. When no separator used, awaits that every message is acknowledged by a simple \"Ok\" string to ensure more stability.
    •  
"},{"location":"changelog/#documentation_22","title":"Documentation","text":"
     
  • Install: Update operating system versions
    •  
  • Sieve Expert: Fix elsif -> elif.
    •  
  • Rephrase the description of time.* fields.
    •  
  • Feeds: New URL and format of the \"Feodo Tracker IPs\" feed. \"Feodo Tracker Domains\" has been discontinued.
    •  
"},{"location":"changelog/#packaging_13","title":"Packaging","text":""},{"location":"changelog/#tests_22","title":"Tests","text":"
     
  • Add missing __init__.py files in 4 bot's test directories. Previously these tests have never been executed.
    •  
  • intelmq.lib.test: Allow bot test class names with an arbitrary postfix separated by an underscore. E.g. TestShodanParserBot_minimal.
    •  
"},{"location":"changelog/#tools_20","title":"Tools","text":"
     
  • intelmqctl:
    •  
  • status: Show commandline differences if a program with the expected PID could be found, but they do not match (previous output was None).
    •  
  • Use logging level from defaults configuration if possible, otherwise intelmq's internal default. Previously, DEBUG was used unconditionally.
    •  
"},{"location":"changelog/#known-issues_23","title":"Known issues","text":"
     
  • Bots started with IntelMQ-Manager stop when the webserver is restarted (#952).
    •  
  • stomp collector bot constantly uses 100% of CPU (#1364).
    •  
"},{"location":"changelog/#111-2019-01-15","title":"1.1.1 (2019-01-15)","text":""},{"location":"changelog/#core_25","title":"Core","text":"
     
  • lib/harmonization.py: Change parse_utc_isoformat of DateTime class from private to public (related to #1322).
    •  
  • lib/utils.py: Add new function object_pair_hook_bots.
    •  
  • lib.bot.py:
    •  
  • ParserBot's method recover_line_csv now also handles given tempdata.
    •  
  • Bot.acknowledge_message() deletes __current_message to free the memory, saves memory in idling parsers with big reports.
    •  
  • start(): Warn once per run if error_dump_message is set to false.
    •  
  • Bot.start(), ParserBot.process(): If errors happen on bots without destination pipeline, the on_error path has been queried and lead to an exception being raised.
    •  
  • start(): If error_procedure is pass and on pipeline errors, the bot retries forever (#1333).
    •  
  • lib/message.py:
    •  
  • Fix add('extra', ..., overwrite=True): old extra fields have not been deleted previously (#1335).
    •  
  • Do not ignore empty or ignored (as defined in _IGNORED_VALUES) values of extra.* fields for backwards compatibility (#1335).
    •  
  • lib/pipeline.py (Redis.receive): Wait in 1s steps if redis is busy loading its snapshot from disk (#1334).
    •  
"},{"location":"changelog/#default-configuration","title":"Default configuration","text":"
     
  • Set error_dump_message to true by default in defaults.conf.
    •  
  • Fixed typo in defaults.conf: proccess_manager -> process_manager
    •  
"},{"location":"changelog/#development_10","title":"Development","text":"
     
  • bin/rewrite_config_files.py: Fix ordering of BOTS file (#1327).
    •  
"},{"location":"changelog/#harmonization_6","title":"Harmonization","text":"
Update allowed classification fields to 2018-09-26 version (#802, #1350, #1380). New values for classification.type are per taxonomy: - Taxonomy 'intrusions':   - \"application-compromise\"   - \"burglary\"   - \"privileged-account-compromise\"   - \"unprivileged-account-compromise\" - Taxonomy 'fraud':   - \"copyright\"   - \"masquerade\"   - \"unauthorized-use-of-resources\" - Taxonomy 'information content security':   - \"data-loss\" - Taxonomy 'vulnerable':   - \"ddos-amplifier\"   - \"information-disclosure\"   - \"potentially-unwanted-accessible\"   - \"vulnerable-system\"   - \"weak-crypto\" - Taxonomy 'availability':   - \"dos\"   - \"outage\"   - \"sabotage\" - Taxonomy 'abusive-content':   - \"harmful-speech\"   - \"violence\" - Taxonomy 'malicious code':   - \"malware-distribution\" - Taxonomy 'information-gathering':   - \"social-engineering\"   - \"sniffing\" - Taxonomy 'information content security':   - \"Unauthorised-information-access\"   - \"Unauthorised-information-modification\"
"},{"location":"changelog/#bots_26","title":"Bots","text":""},{"location":"changelog/#collectors_20","title":"Collectors","text":"
     
  • intelmq.bots.collectors.http.collector_http:
    •  
  • Fix parameter name extract_files in BOTS (#1331).
    •  
  • Fix handling of extract_files parameter if the value is an empty string.
    •  
  • Handle not installed dependency library requests gracefully.
    •  
  • Explain extract_files parameter in docs and use a sane default in BOTS file.
    •  
  • intelmq.bots.collectors.mail.collector_mail_url:
    •  
  • Handle HTTP status codes != 2xx the same as HTTP timeouts: No exception, but graceful handling.
    •  
  • Handle HTTP errors (bad status code and timeouts) with error_procedure == 'pass' but marking the mail as read and logging the error.
    •  
  • Handle not installed dependency library requests gracefully.
    •  
  • intelmq.bots.collectors.http.collector_http_stream:
    •  
  • Handle not installed dependency library requests gracefully.
    •  
  • intelmq.bots.collectors.microsoft.collector_interflow:
    •  
  • Handle not installed dependency library requests gracefully.
    •  
  • intelmq.bots.collectors.rt.collector_rt:
    •  
  • Handle not installed dependency library requests gracefully.
    •  
  • added intelmq.bots.collectors.shodan.collector_stream for collecting shodan stream data (#1096).
    •  
  • Correctly check the version of the shodan library, it resulted in wrong comparisons with two digit numbers.
    •  
  • intelmq.bots.collectors.microsoft.collector_interflow:
    •  
  • Add check if Cache's TTL is big enough compared to not_older_than and throw an error otherwise.
    •  
"},{"location":"changelog/#parsers_24","title":"Parsers","text":"
     
  • intelmq.bots.parsers.misp: Fix Object attribute (#1318).
    •  
  • intelmq.bots.parsers.cymru.parser_cap_program:
    •  
  • Add support for new format (extra data about botnet of 'bots').
    •  
  • Handle AS number 0.
    •  
  • intelmq.bots.parsers.shadowserver:
    •  
  • Spam URL reports: remove src_naics, src_sic columns.
    •  
  • fix parsing of 'spam' events in ShadowServer's 'Botnet Drone Hadoop' Report (#1271).
    •  
  • Add support in parser to ignore some columns in config file by using False as intelmq key.
    •  
  • Add support for the Outdated-DNSSEC-Key and Outdated-DNSSEC-Key-IPv6 feeds.
    •  
  • Add support for the Accessible-Rsync feed.
    •  
  • Document support for the Open-LDAP-TCP feed.
    •  
  • Add support for Accessible-HTTP and Open-DB2-Discovery-Service (#1349).
    •  
  • Add support for Accessible-AFP (#1351).
    •  
  • Add support for Darknet (#1353).
    •  
  • intelmq.bots.parsers.generic.parser_csv: If the skip_header parameter was set to True, the header was not part of the raw field as returned by the recover_line method. The header is now saved and handled correctly by the fixed recovery method.
    •  
  • intelmq.bots.parsers.cleanmx.parser: Use field first instead of firsttime for time.source (#1329, #1348).
    •  
  • intelmq.bots.parsers.twitter.parser: Support for url-normalize >= 1.4.1 and recommend it. Added new optional parameter default_scheme, passed to url-normalize (#1356).
    •  
"},{"location":"changelog/#experts_25","title":"Experts","text":"
     
  • intelmq.bots.experts.national_cert_contact_certat.expert:
    •  
  • Handle not installed dependency library requests gracefully.
    •  
  • intelmq.bots.experts.ripencc_abuse_contact.expert:
    •  
  • Handle not installed dependency library requests gracefully.
    •  
  • intelmq.bots.experts.sieve.expert:
    •  
  • check method: Load missing harmonization, caused an error for every check.
    •  
  • Add text and more context to error messages.
    •  
  • README: Fix 'modify' to 'update' (#1340).
    •  
  • Handle empty rules file (#1343).
    •  
  • intelmq.bots.experts.idea.expert: Add mappings for new harmonization classification.type values, see above.
    •  
"},{"location":"changelog/#outputs_21","title":"Outputs","text":"
     
  • intelmq.bots.outputs.redis:
    •  
  • Fix sending password to redis server.
    •  
  • Fix for redis-py >= 3.0.0: Convert Event to string explicitly (#1354).
    •  
  • Use Redis class instead of deprecated StrictRedis for redis-py >= 3.0.0 (#1355).
    •  
  • intelmq.bots.outputs.mongodb:
    •  
  • New parameter replacement_char (default: '_') for non-hierarchical output as dots in key names are not allowed (#1324, #1322).
    •  
  • Save value of fields time.observation and time.source as native datetime object, not as string (#1322).
    •  
  • intelmq.bots.outputs.restapi.output:
    •  
  • Handle not installed dependency library requests gracefully.
    •  
"},{"location":"changelog/#documentation_23","title":"Documentation","text":"
     
  • FAQ
    •  
  • Explanation and solution on orphaned queues.
    •  
  • Section on how and why to remove raw data.
    •  
  • Add or fix the tables of contents for all documentation files.
    •  
  • Feeds:
    •  
  • Fix Autoshun Feed URL (#1325).
    •  
  • Add parameters name and provider to intelmq/etc/feeds.yaml, docs/Feeds.md and intelmq/bots/BOTS (#1321).
    •  
  • Add SECURITY.md file.
    •  
"},{"location":"changelog/#packaging_14","title":"Packaging","text":"
     
  • Change the maintainer from Sasche Wilde to Sebastian Wagner (#1320).
    •  
"},{"location":"changelog/#tests_23","title":"Tests","text":"
     
  • intelmq.tests.lib.test_bot: Skip test_logging_level_other on python 3.7 because of unclear behavior related to copies of loggers (#1269).
    •  
  • intelmq.tests.bots.collectors.rt.test_collector: Remove test because the REST interface of the instance has been closed (see also https://github.com/CZ-NIC/python-rt/issues/28).
    •  
"},{"location":"changelog/#tools_21","title":"Tools","text":"
     
  • intelmqctl check: Shows more detailed information on orphaned queues.
    •  
  • intelmqctl:
    •  
  • Correctly determine the status of bots started with intelmqctl run.
    •  
  • Fix output of errors during bot status determination, making it compatible to IntelMQ Manager.
    •  
  • check subcommand: Show bot ID for messages also in JSON output.
    •  
  • run [bot-id] process -m [message] works also with bots without a configured source pipeline (#1307).
    •  
"},{"location":"changelog/#contrib_12","title":"Contrib","text":"
     
  • elasticsearch/elasticmapper: Add tlp field (#1308).
    •  
  • feeds-config-generator/intelmq_gen_feeds_conf:
    •  
  • Add parameters to write resulting configuration directly to files (#1321).
    •  
  • Handle collector's feed.name and feed.provider (#1314).
    •  
"},{"location":"changelog/#known-issues_24","title":"Known issues","text":"
     
  • Bots started with IntelMQ-Manager stop when the webserver is restarted (#952).
    •  
  • Tests: capture logging with context manager (#1342).
    •  
  • stomp collector bot constantly uses 100% of CPU (#1364).
    •  
"},{"location":"changelog/#110-2018-09-05","title":"1.1.0 (2018-09-05)","text":"
     
  • Support for Python 3.3 has been dropped in IntelMQ and some dependencies of it. Python 3.3 reached its end of life and Python 3.4 or newer is a hard requirement now.
    •  
  • The list of feeds docs/Feeds.md has now a machine-readable equivalent YAML file in intelmq/etc/feeds.yaml   A tool to convert from yaml to md has been added.
    •  
"},{"location":"changelog/#tools_22","title":"Tools","text":"
     
  • intelmq_gen_feeds_docs added to bin directory, allows generating the Feeds.md documentation file from feeds.yaml
    •  
  • intelmq_gen_docs merges both intelmq_gen_feeds_docs and intelmq_gen_harm_docs in one file and automatically updates the documentation files.
    •  
"},{"location":"changelog/#intelmqctl","title":"intelmqctl","text":"
     
  • intelmqctl start prints the bot's last error messages if the bot failed to start (#1021).
    •  
  • intelmqctl start message \"is running\" is printed every time. (Until now, it wasn't said when a bot was just starting.)
    •  
  • intelmqctl start/stop/restart/reload/status now has a \"--group\" flag which allows you to specify the group of the bots that should be influenced by the command.
    •  
  • intelmqctl check checks for defaults.conf completeness if the shipped file from the package can be found.
    •  
  • intelmqctl check shows errors for non-importable bots.
    •  
  • intelmqctl list bots -q only prints the IDs of enabled bots.
    •  
  • intelmqctl list queues-and-status prints both queues and bots statuses (so that it can be used in eg. intelmq-manager).
    •  
  • intelmqctl run parameter for showing a sent message.
    •  
  • intelmqctl run if message is sent to a non-default path, it is printed out.
    •  
  • intelmqctl restart bug fix; returned some half-nonsense, now returns return state of start and stop operation in a list (#1226).
    •  
  • intelmqctl check: New parameter --no-connections to prevent the command from making connections e.g. to the redis pipeline.s
    •  
  • intelmqctl list queues: don't display named paths among standard queues.
    •  
  • The process status test failed if the PATH did not include the bot executables and the which command failed. Then the proccess's command line could not be compared correctly. The fix warns of this and adds a new status 'unknown' (#1297).
    •  
"},{"location":"changelog/#contrib_13","title":"Contrib","text":"
     
  • tool feeds-config-generator to automatically generate the collector and parser runtime and pipeline configurations.
    •  
  • malware_name_mapping: Download and convert tool for malware family name mapping has been added.
    •  
  • Added a systemd script which creates systemd units for bots (#953).
    •  
  • contrib/cron-jobs/update-asn-data, contrib/cron-jobs/update-geoip-data, contrib/cron-jobs/update-tor-nodes: Errors produce proper output.
    •  
"},{"location":"changelog/#core_26","title":"Core","text":"
     
  • lib/bot
    •  
  • use SIGTERM instead of SIGINT to stop bots (#981).
    •  
  • Bots can specify a static method check(parameters) which can perform individual checks specific to the bot.     These functions will be called by intelmqctl check if the bot is configured with the given parameters
    •  
  • top level bot parameters (description, group, module, name) are exposed as members of the class.
    •  
  • The parameter feed for collectors is deprecated for 2.0 and has been replaced by the more consistent name (#1144).
    •  
  • bug: allow path parameter for CollectorBot class.
    •  
  • Handle errors better when the logger could not be initialized.
    •  
  • ParserBot:
       
    • For the csv parsing methods, ParserBot.csv_params is now used for all these methods.
      •  
    • ParserBot.parse_csv_dict now saves the field names in ParserBot.csv_fieldnames.
      •  
    • ParserBot.parse_csv_dict now saves the raw current line in ParserBot.current_line.
      •  
    • ParserBot.recover_line_csv_dict now uses the raw current line.
      •  
     
    •  
  • lib/message:
    •  
  • Subitems in fields of type JSONDict (see below) can be accessed directly. E.g. you can do:     event['extra.foo'] = 'bar'     event['extra.foo'] # gives 'bar'     It is still possible to set and get the field as whole, however this may be removed or changed in the future:     event['extra'] = '{\"foo\": \"bar\"}'     event['extra'] # gives '{\"foo\": \"bar\"}'     \"Old\" bots and configurations compatible with 1.0.x do still work.     Also, the extra field is now properly exploded when exporting events, analogous to all other fields.     The in operator works now for both - the old and the new - behavior.
    •  
  • Message.add: The parameter overwrite accepts now three different values: True, False and None (new).     True: An existing value will be overwritten     False: An existing value will not be overwritten (previously an exception has been raised when the value was given).     None (default): If the value exists an KeyExists exception is thrown (previously the same as False).     This allows shorter code in the bots, as an 'overwrite' configuration parameter can be directly passed to the function.
    •  
  • The message class has now the possibility to return a default value for non-existing fields, see Message.set_default_value.
    •  
  • Message.get behaves the same like Message.__getitem__ (#1305).
    •  
  • Add RewindableFileHandle to utils making handling of CSV files more easy (optionally)
    •  
  • lib/pipeline:
    •  
  • you may now define more than one destination queues path the bot should pass the message to, see Pipelines (#1088, #1190).
    •  
  • the special path \"_on_error\" can be used to pass messages to different queues in case of processing errors (#1133).
    •  
  • lib/harmonization: Accept AS prefix for ASN values (automatically stripped).
    •  
  • added intelmq.VAR_STATE_PATH for variable state data of bots.
    •  
"},{"location":"changelog/#bots_27","title":"Bots","text":"
     
  • Removed print statements from various bots.
    •  
  • Replaced various occurrences of self.logger.error() + self.stop() with raise ValueError.
    •  
"},{"location":"changelog/#collectors_21","title":"Collectors","text":"
     
  • bots.collectors.mail:
    •  
  • New parameters; sent_from: filter messages by sender, sent_to: filter messages by recipient
    •  
  • More debug logs
    •  
  • bots.collectors.n6.collector_stomp: renamed to bots.collectors.stomp.collector (#716)
    •  
  • bots.collectors.rt:
    •  
  • New parameter search_requestor to search for field Requestor.
    •  
  • Empty strings and null as value for search parameters are ignored.
    •  
  • Empty parameters attachment_regex and url_regex handled.
    •  
  • bots.collectors.http.collector_http: Ability to optionally use the current time in parameter http_url, added parameter http_url_formatting.
    •  
  • bots.collectors.stomp.collector: Heartbeat timeout is now logged with log level info instead of warning.
    •  
  • added intelmq.bots.collectors.twitter.collector_twitter
    •  
  • added intelmq.bots.collectors.tcp.collector that can be bound to another IntelMQ instance by a TCP output
    •  
  • bots.collectors.microsoft.collector_interflow: added for MS interflow API
    •  
  • Automatic ungzipping for .gz files.
    •  
  • added intelmq.bots.collectors.calidog.collector_certstream for collecting certstream data (#1120).
    •  
  • added intelmq.bots.collectors.shodan.collector_stream for collecting shodan stream data (#1096).
    •  
  • Add proxy support.
    •  
  • Fix handling of parameter countries.
    •  
"},{"location":"changelog/#parsers_25","title":"Parsers","text":"
     
  • bots.parsers.shadowserver:
    •  
  • changed feednames. Please refer to it's README for the exact changes.
    •  
  • If the conversion function fails for a line, an error is raised and the offending line will be handled according to the error handling configuration.     Previously errors like these were only logged and ignored otherwise.
    •  
  • add support for the feeds
       
    • Accessible-Hadoop (#1231)
      •  
    • Accessible ADB (#1285)
      •  
     
    •  
  • Remove deprecated parameter override, use overwrite instead (#1071).
    •  
  • The raw values now are exactly the input with quotes unchanged, the ParserBot methods are now used directly (#1011).
    •  
  • The Generic CSV Parser bots.parsers.generic.parser_csv:
    •  
  • It is possible to filter the data before processing them using the new parameters filter_type and filter_text.
    •  
  • It is possible to specify multiple columns using | character in parameter columns.
    •  
  • The parameter time_format now supports 'epoch_millis' for seconds since the Epoch, milliseconds are supported but not used.
    •  
  • renamed bots.parsers.cymru_full_bogons.parser to bots.parsers.cymru.parser_full_bogons, compatibility shim will be removed in version 2.0
    •  
  • added bots.parsers.cymru.parser_cap_program
    •  
  • added intelmq.bots.parsers.zoneh.parser for ZoneH feeds
    •  
  • added intelmq.bots.parsers.sucuri.parser
    •  
  • added intelmq.bots.parsers.malwareurl.parser
    •  
  • added intelmq.bots.parsers.threatminer.parser
    •  
  • added intelmq.bots.parsers.webinspektor.parser
    •  
  • added intelmq.bots.parsers.twitter.parser
    •  
  • added intelmq.bots.parsers.microsoft.parser_ctip
    •  
  • ignore the invalid IP '0.0.0.0' for the destination
    •  
  • fix the raw/dumped messages, did not contain the paling list previously.
    •  
  • use the new harmonization field tlp instead of extra.tlp.
    •  
  • bots.parsers.alienvault.parser_otx: Save TLP data in the new harmonization field tlp.
    •  
  • added intelmq.bots.parsers.openphish.parser_commercial
    •  
  • added intelmq.bots.parsers.microsoft.parser_bingmurls
    •  
  • added intelmq.bots.parsers.calidog.parser_certstream for parsing certstream data (#1120).
    •  
  • added intelmq.bots.parsers.shodan.parser for parsing shodan data (#1096).
    •  
  • change the classification type from 'botnet drone' to 'infected system' in various parses.
    •  
  • intelmq.bots.parsers.spamhaus.parser_cert: Added support for all known bot types.
    •  
"},{"location":"changelog/#experts_26","title":"Experts","text":"
     
  • Added sieve expert for filtering and modifying events (#1083)
    •  
  • capable of distributing the event to appropriate named queues
    •  
  • bots.experts.modify
    •  
  • default rulesets: all malware name mappings have been migrated to the Malware Name Mapping repository ruleset. See the new added contrib tool for download and conversion.
    •  
  • new parameter case_sensitive (default: True)
    •  
  • Added wait expert for sleeping
    •  
  • Added domain suffix expert to extract the TLD/Suffix from a domain name.
    •  
  • bots.experts.maxmind_geoip: New (optional) parameter overwrite, by default false. The current default was to overwrite!
    •  
  • intelmq.bots.experts.ripencc_abuse_contact:
    •  
  • Extend deprecated parameter compatibility query_ripe_stat until 2.0 because of a logic bug in the compatibility code, use query_ripe_stat_asn and query_ripe_stat_ip instead (#1071, #1291).
    •  
  • Handle HTTP status code 404 for DB AS queries.
    •  
  • Add caching capability.
    •  
  • intelmq/bots/experts/asn_lookup/update-asn-data: Errors produce proper output on stdout/stderr.
    •  
  • intelmq/bots/experts/maxmind_geoip/update-geoip-data: Errors produce proper output on stdout/stderr.
    •  
  • intelmq/bots/experts/tor_nodes/update-tor-nodes: Errors produce proper output on stdout/stderr.
    •  
"},{"location":"changelog/#outputs_22","title":"Outputs","text":"
     
  • bots.outputs.file:
    •  
  • String formatting can be used for file names with new parameter format_filename.
    •  
  • New parameter single_key to only save one field.
    •  
  • New parameter encoding_errors_mode with default value 'strict' to handle encoding errors for the files written.
    •  
"},{"location":"changelog/#harmonization_7","title":"Harmonization","text":"
     
  • Renamed JSON to JSONDict and added a new type JSON. JSONDict saves data internally as JSON, but acts like a dictionary. JSON accepts any valid JSON.
    •  
  • fixed regex for protocol.transport it previously allowed more values than it should have.
    •  
  • New ASN type. Like integer but checks the range.
    •  
  • added destination.urlpath and source.urlpath to harmonization.
    •  
  • New field tlp for tlp level specification.
    •  
  • New TLP type. Allows all four tlp levels, removes 'TLP:' prefix and converts to upper case.
    •  
  • Added new classification.type 'vulnerable client'
    •  
  • Added (destination|source).domain_suffix to hold the TLD/domain suffix.
    •  
  • New allowed value for classification.type: infected system for taxonomy malicious code (#1197).
    •  
"},{"location":"changelog/#requirements_1","title":"Requirements","text":"
     
  • Requests is no longer listed as dependency of the core. For depending bots the requirement is noted in their REQUIREMENTS.txt file.
    •  
"},{"location":"changelog/#documentation_24","title":"Documentation","text":"
     
  • Use Markdown for README again, as pypi now supports it.
    •  
  • Developers Guide: Add instructions for pre-release testing.
    •  
"},{"location":"changelog/#packaging_15","title":"Packaging","text":"
     
  • Add logcheck configuration to the packages.
    •  
  • Fix packaging of bash completion script.
    •  
"},{"location":"changelog/#tests_24","title":"Tests","text":"
     
  • Travis now correctly stops if a requirement could not be installed (#1257).
    •  
  • New tests for validating etc/feeds.yaml and bots/BOTS using cerberus and schemes are added (#1166).
    •  
  • New test for checking if docs/Feeds.md is up to date with etc/feeds.yaml.
    •  
"},{"location":"changelog/#known-bugs","title":"Known bugs","text":"
     
  • contrib: feeds-config-generator does not add feed name as parameter (#1314).
    •  
  • bot debugger requires configured source pipeline (#1307).
    •  
  • shadowserver parser: drone feed has spam events (#1271).
    •  
  • debug log level on python 3.7 not applied (#1269).
    •  
  • bots.experts.sieve does not support textX (#1246).
    •  
  • Bots started with IntelMQ-Manager stop when the webserver is restarted (#952).
    •  
"},{"location":"changelog/#106-bugfix-release-2018-08-31","title":"1.0.6 Bugfix release (2018-08-31)","text":""},{"location":"changelog/#bots_28","title":"Bots","text":""},{"location":"changelog/#collectors_22","title":"Collectors","text":"
     
  • bots.collectors.rt.collector_rt: Log ticket id for downloaded reports.
    •  
"},{"location":"changelog/#parsers_26","title":"Parsers","text":"
     
  • bots.parsers.shadowserver:
    •  
  • if required fields do not exist in data, an exception is raised, so the line will be dumped and not further processed.
    •  
  • fix a bug in the parsing of column cipher_suite in ssl poodle reports (#1288).
    •  
"},{"location":"changelog/#experts_27","title":"Experts","text":"
     
  • Reverse DNS Expert: ignore all invalid results and use first valid one (#1264).
    •  
  • intelmq/bots/experts/tor_nodes/update-tor-nodes: Use check.torproject.org as source as internet2.us is down (#1289).
    •  
"},{"location":"changelog/#outputs_23","title":"Outputs","text":"
     
  • bots.output.amqptopic:
    •  
  • The default exchange must not be declared (#1295).
    •  
  • Unencodable characters are prepended by backslashes by default. Otherwise Unicode characters can't be encoded and sent (#1296).
    •  
  • Gracefully close AMQP connection on shutdown of bot.
    •  
"},{"location":"changelog/#documentation_25","title":"Documentation","text":"
     
  • Bots: document redis cache parameters.
    •  
  • Installation documentation: Ubuntu needs universe repositories.
    •  
"},{"location":"changelog/#packaging_16","title":"Packaging","text":"
     
  • Dropped support for Ubuntu 17.10, it reached its End of Life as of 2018-07-19.
    •  
"},{"location":"changelog/#tests_25","title":"Tests","text":"
     
  • Drop tests for Python 3.3 for the mode with all requirements, as some optional dependencies do not support Python 3.3 anymore.
    •  
  • lib.test: Add parameter compare_raw (default: True) to assertMessageEqual, to optionally skip the comparison of the raw field.
    •  
  • Add tests for RT collector.
    •  
  • Add tests for Shadowserver Parser:
    •  
  • SSL Poodle Reports.
    •  
  • Helper functions.
    •  
"},{"location":"changelog/#tools_23","title":"Tools","text":"
     
  • intelmqctl list now sorts the output of bots and queues (#1262).
    •  
  • intelmqctl: Correctly handle the corner cases with collectors and outputs for getting/sending messages in the bot debugger (#1263).
    •  
  • intelmqdump: fix ordering of dumps in a file in runtime. All operations are applied to a sorted list (#1280).
    •  
"},{"location":"changelog/#contrib_14","title":"Contrib","text":"
     
  • cron-jobs/update-tor-nodes: Use check.torproject.org as source as internet2.us is down (#1289).
    •  
"},{"location":"changelog/#known-issues_25","title":"Known issues","text":"
     
  • shadowserver parser: drone feed has spam events (#1271).
    •  
"},{"location":"changelog/#105-bugfix-release-2018-06-21","title":"1.0.5 Bugfix release (2018-06-21)","text":""},{"location":"changelog/#core_27","title":"Core","text":"
     
  • lib/message: Report() can now create a Report instance from Event instances (#1225).
    •  
  • lib/bot:
    •  
  • The first word in the log line Processed ... messages since last logging. is now adaptable and set to Forwarded in the existing filtering bots (#1237).
    •  
  • Kills oneself again after proper shutdown if the bot is XMPP collector or output (#970). Previously these two bots needed two stop commands to get actually stopped.
    •  
  • lib/utils: log: set the name of the py.warnings logger to the bot name (#1184).
    •  
"},{"location":"changelog/#harmonization_8","title":"Harmonization","text":"
     
  • Added new types unauthorized-command and unauthorized-login to intrusions taxonomy.
    •  
"},{"location":"changelog/#bots_29","title":"Bots","text":""},{"location":"changelog/#collectors_23","title":"Collectors","text":"
     
  • bots.collectors.mail.collector_mail_url: handle empty downloaded reports (#988).
    •  
  • bots.collectors.file.collector_file: handle empty files (#1244).
    •  
"},{"location":"changelog/#parsers_27","title":"Parsers","text":"
     
  • Shadowserver parser:
    •  
  • SSL FREAK: Remove optional column device_serial and add several new ones.
    •  
  • Fixed HTTP URL parsing for multiple feeds (#1243).
    •  
  • Spamhaus CERT parser:
    •  
  • add support for smtpauth, l_spamlink, pop, imap, rdp, smb, iotscan, proxyget, iotmicrosoftds, automatedtest, ioturl, iotmirai, iotcmd, iotlogin and iotuser (#1254).
    •  
  • fix extra.destination.local_port -> extra.source.local_port.
    •  
"},{"location":"changelog/#experts_28","title":"Experts","text":"
     
  • bots.experts.filter: Pre-compile regex at bot initialization.
    •  
"},{"location":"changelog/#tests_26","title":"Tests","text":"
     
  • Ensure that the bots did process all messages (#291).
    •  
"},{"location":"changelog/#tools_24","title":"Tools","text":"
     
  • intelmqctl:
    •  
  • intelmqctl run has a new parameter -l --loglevel to overwrite the log level for the run (#1075).
    •  
  • intelmqctl run [bot-id] message send can now send report messages (#1077).
    •  
  • intelmqdump:
    •  
  • has now command completion for bot names, actions and queue names in interactive console.
    •  
  • automatically converts messages from events to reports if the queue the message is being restored to is the source queue of a parser (#1225).
    •  
  • is now capable to read messages in dumps that are dictionaries as opposed to serialized dicts as strings and does not convert them in the show command (#1256).
    •  
  • truncated messages are no longer used/saved to the file after being shown (#1255).
    •  
  • now again denies recovery of dumps if the corresponding bot is running. The check was broken (#1258).
    •  
  • now sorts the dump by the time of the dump. Previously, the list was in random order (#1020).
    •  
"},{"location":"changelog/#known-issues_26","title":"Known issues","text":"
no known issues
"},{"location":"changelog/#104-bugfix-release-2018-04-20","title":"1.0.4 Bugfix release (2018-04-20)","text":"
     
  • make code style compatible to pycodestyle 2.4.0
    •  
  • fixed permissions of some files (they were executable but shouldn't be)
    •  
"},{"location":"changelog/#core_28","title":"Core","text":"
     
  • lib/harmonization:
    •  
  • FQDN validation now handles None correctly (raised an Exception).
    •  
  • Fixed several sanitize() methods, the generic sanitation method were called by is_valid, not the sanitize methods (#1219).
    •  
"},{"location":"changelog/#bots_30","title":"Bots","text":"
     
  • Use the new pypi website at https://pypi.org/ everywhere.
    •  
"},{"location":"changelog/#parsers_28","title":"Parsers","text":"
     
  • Shadowserver parser:
    •  
  • The fields url and http_url now handle HTTP URL paths and HTTP requests for all feeds (#1204).
    •  
  • The conversion function validate_fqdn now handles empty strings correctly.
    •  
  • Feed 'drone (hadoop)':
       
    • Correct validation of field cc_dns, will now only be added as destination.fqdn if correct FQDN, otherwise ignored. Previously this field could be saved in extra containing an IP address.
      •  
    • Adding more mappings for added columns.
      •  
     
    •  
  • Added feeds:
       
    • Drone-Brute-Force
      •  
    • IPv6-Sinkhole-HTTP-Drone
      •  
     
    •  
  • A lot of newly added fields and fixed conversions.
    •  
  • Optional fields can now use one column multiple times.
    •  
  • Add newly added columns of Ssl-Scan feed to parser
    •  
  • Spamhaus CERT parser:
    •  
  • fix parsing and classification for bot names 'openrelay', 'iotrdp', 'sshauth', 'telnetauth', 'iotcmd', 'iotuser', 'wpscanner', 'w_wplogin', 'iotscan'     see the NEWS file - Postgresql section - for all changes.
    •  
  • CleanMX phishing parser: handle FQDNs in IP column (#1162).
    •  
"},{"location":"changelog/#experts_29","title":"Experts","text":"
     
  • bots.experts.ripencc_abuse_contact: Add existing parameter mode to BOTS file.
    •  
"},{"location":"changelog/#tools_25","title":"Tools","text":"
     
  • intelmqctl check: Fixed and extended message for 'run_mode' check.
    •  
  • intelmqctl start botnet. When using --type json, no non-JSON information about wrong bots are output because that would confuse eg. intelmq-manager
    •  
"},{"location":"changelog/#tests_27","title":"Tests","text":"
     
  • lib/bot: No dumps will be written during tests (#934).
    •  
  • lib/test: Expand regular expression on python version to match pre-releases (debian testing).
    •  
"},{"location":"changelog/#packaging_17","title":"Packaging","text":"
     
  • Static data is now included in source tarballs, development files are excluded
    •  
"},{"location":"changelog/#known-issues_27","title":"Known issues","text":"
     
  • bots.collectors/outputs.xmpp must be killed two times (#970).
    •  
  • When running bots with intelmqctl run [bot-id] the log level is always INFO (#1075).
    •  
  • intelmqctl run [bot-id] message send [msg] does only support Events, not Reports (#1077).
    •  
  • A warning issued by the python warnings module is logged without the bot-id (#1184).
    •  
"},{"location":"changelog/#103-bugfix-release-2018-02-05","title":"1.0.3 Bugfix release (2018-02-05)","text":""},{"location":"changelog/#contrib_15","title":"Contrib","text":"
     
  • logrotate: use sudo for postrotate script
    •  
  • cron-jobs: use the scripts in the bots' directories and link them (#1056, #1142)
    •  
"},{"location":"changelog/#core_29","title":"Core","text":"
     
  • lib.harmonization: Handle idna encoding error in FQDN sanitation (#1175, #1176).
    •  
  • lib.bot:
    •  
  • Bots stop when redis gives the error \"OOM command not allowed when used memory > 'maxmemory'.\" (#1138).
    •  
  • warnings of bots are caught by the logger (#1074, #1113).
    •  
  • Fixed exitcodes 0 for graceful shutdowns .
    •  
  • better handling of problems with pipeline and especially it's initialization (#1178).
    •  
  • All parsers using ParserBot's methods now log the sum of successfully parsed and failed lines at the end of each run (#1161).
    •  
"},{"location":"changelog/#harmonization_9","title":"Harmonization","text":"
     
  • Rule for harmonization keys is enforced (#1104, #1141).
    •  
  • New allowed values for classification.type: tor & leak (see n6 parser below ).
    •  
"},{"location":"changelog/#bots_31","title":"Bots","text":""},{"location":"changelog/#collectors_24","title":"Collectors","text":"
     
  • bots.collectors.mail.collector_mail_attach: Support attachment file parsing for imbox versions newer than 0.9.5 (#1134).
    •  
"},{"location":"changelog/#parsers_29","title":"Parsers","text":"
     
  • All CSV parsers ignore NULL-bytes now, because the csv-library cannot handle it (#967, #1114).
    •  
  • bots.parsers.shadowserver.parser: Add Accessible Cisco Smart Install (#1122).
    •  
  • bots.parsers.cleanmx.parser: Handle new columns first and last, rewritten for XML feed. See NEWS.md for upgrade instructions (#1131, #1136, #1163).
    •  
  • bots.parsers.n6.parser: Fix classification mappings. See NEWS file for changes values (#738, #1127).
    •  
"},{"location":"changelog/#experts_30","title":"Experts","text":"
     
  • bots.experts.modify default ruleset: changed conficker rule to catch more spellings.
    •  
"},{"location":"changelog/#outputs_24","title":"Outputs","text":"
     
  • bots.outputs.smtp.output: Fix STARTTLS, threw an exception (#1152, #1153).
    •  
"},{"location":"changelog/#documentation_26","title":"Documentation","text":"
     
  • Release.md add release procedure documentation
    •  
  • Bots.md: fix example configuration for modify expert
    •  
"},{"location":"changelog/#tools_26","title":"Tools","text":"
     
  • intelmqctl now exits with exit codes > 0 when errors happened or the operation was not successful. Also, the status operation exits with 1, if bots are stopped, but enabled. (#977, #1143)
    •  
  • intelmctl check checks for valid run_mode in runtime configuration (#1140).
    •  
"},{"location":"changelog/#tests_28","title":"Tests","text":"
     
  • tests.lib.test_pipeline: Redis tests clear all queues before and after tests (#1086).
    •  
  • Repaired debian package build on travis (#1169).
    •  
  • Warnings are not allowed by default, an allowed count can be specified (#1129).
    •  
  • tests.bots.experts.cymru_whois/abusix: Skipped on travis because of ongoing problems.
    •  
"},{"location":"changelog/#packaging_18","title":"Packaging","text":"
     
  • cron jobs: fix paths of executables
    •  
"},{"location":"changelog/#known-issues_28","title":"Known issues","text":"
     
  • bots.collectors/outputs.xmpp must be killed two times (#970).
    •  
  • When running bots with intelmqctl run [bot-id] the log level is always INFO (#1075).
    •  
  • intelmqctl run [bot-id] message send [msg] does only support Events, not Reports (#1077).
    •  
  • python3 setup.py sdist does not include static files in the resulting tarballs (#1146).
    •  
  • bots.parsers.cleanmx.parser: The cleanMX feed may have FQDNs as IPs in rare cases, such lines are dumped (#1162).
    •  
"},{"location":"changelog/#102-bugfix-release-2017-11-09","title":"1.0.2 Bugfix release (2017-11-09)","text":""},{"location":"changelog/#core_30","title":"Core","text":"
     
  • lib.message.add: parameter force has finally been removed, should have been gone in 1.0.0.rc1 already
    •  
"},{"location":"changelog/#bots_32","title":"Bots","text":"
     
  • collectors.mail.collector_mail_url: Fix bug which prevented marking emails seen due to disconnects from server (#852).
    •  
  • parsers.spamhaus.parser_cert: Handle/ignore 'AS?' in feed (#1111)
    •  
"},{"location":"changelog/#packaging_19","title":"Packaging","text":"
     
  • The following changes have been in effect for the built packages already since version 1.0.0
    •  
  • Support building for more distributions, now supported: CentOS 7, Debian 8 and 9, Fedora 25 and 26, RHEL 7, openSUSE Leap 42.2 and 42.3 and Tumbleweed, Ubuntu 14.04 and 16.04
    •  
  • Use LSB-paths for created packages (/etc/intelmq/, /var/lib/intelmq/, /run/intelmq/) (#470). Does does not affect installations with setuptools/pip.
    •  
  • Change the debian package format from native to quilt
    •  
  • Fix problems in postint and postrm scripts
    •  
  • Use systemd-tmpfile for creation of /run/intelmq/
    •  
"},{"location":"changelog/#documentation_27","title":"Documentation","text":"
     
  • Add disclaimer on maxmind database in bot documentation and code and the cron-job (#1110)
    •  
"},{"location":"changelog/#101-bugfix-release-2017-08-30","title":"1.0.1 Bugfix release (2017-08-30)","text":""},{"location":"changelog/#documentation_28","title":"Documentation","text":"
     
  • Feeds: use more https:// URLs
    •  
  • minor fixes
    •  
"},{"location":"changelog/#bots_33","title":"Bots","text":"
     
  • bots/experts/ripencc_abuse_contact/expert.py: Use HTTPS URLs for rest.db.ripe.net
    •  
  • bots/outputs/file/output.py: properly close the file handle on shutdown
    •  
  • bots/parser/shadowserver: If conversion of a value via conversion function fails, only log the function name, not the representation string (#1157).
    •  
"},{"location":"changelog/#core_31","title":"Core","text":"
     
  • lib/bot: Bots will now log the used intelmq version at startup
    •  
"},{"location":"changelog/#tools_27","title":"Tools","text":"
     
  • intelmqctl: To check the status of a bot, the command line of the running process is compared to the actual executable of the bot. Otherwise unrelated programs with the same PID are detected as running bot.
    •  
  • intelmqctl: enable, disable, check, clear now support the JSON output
    •  
"},{"location":"changelog/#100-stable-release-2017-08-04","title":"1.0.0 Stable release (2017-08-04)","text":""},{"location":"changelog/#core_32","title":"Core","text":"
     
  • Fixes a thrown FileNotFound exception when stopping bots started with intelmqctl run ...
    •  
"},{"location":"changelog/#harmonization_10","title":"Harmonization","text":"
     
  • leading dots in FQDNs are rejected and removed in sanitation (#1022, #1030)
    •  
"},{"location":"changelog/#bots_34","title":"Bots","text":"
     
  • shadowserver parser Accessible-SMB: smb_implant is converted to bool
    •  
"},{"location":"changelog/#100rc1-release-candidate-2017-07-05","title":"1.0.0.rc1 Release candidate (2017-07-05)","text":""},{"location":"changelog/#core_33","title":"Core","text":"
     
  • Changing the value of an existing field to None deletes the field.
    •  
  • Message.update now behaves like dict.update. The old behavior is implemented in Message.change
    •  
  • Deprecated http_ssl_proxy has been dropped, use https_proxy instead
    •  
  • Deprecated http_timeout has been dropped, use http_timeout_sec instead
    •  
  • Deprecated parameters force and ignore of Message.add have been removed
    •  
  • Deprecated method Message.contains has been removed
    •  
  • Drop support for deprecated configuration files startup.conf and system.conf
    •  
"},{"location":"changelog/#development_11","title":"Development","text":"
     
  • We are now testing with and without optional libraries/lowest recommended versions and most current versions of required libraries
    •  
  • Tests shadowserver with more data and checks for warnings and errors
    •  
  • Tests: if bots log warnings this counts as failure if not allowed explicitly
    •  
  • Tests: Bot preparation can be skipped
    •  
"},{"location":"changelog/#documentation_29","title":"Documentation","text":"
     
  • The branching/releasing mechanism has been documented
    •  
"},{"location":"changelog/#bots_35","title":"Bots","text":""},{"location":"changelog/#collectors_25","title":"Collectors","text":"
     
  • HTTP collectors: If http_username and http_password are both given and empty or null, 'None:None' has been used to authenticate. It is now checked that the username evaluates to non-false/null before adding the authentication. (fixes #1017)
    •  
  • Dropped unmaintained and undocumented FTP(S) collectors bots.collectors.ftp. Also, the FTPS collector had a license conflict (#842).
    •  
  • bots.collectors.http.collector_http_stream: drop deprecated parameter url in favor of http_url
    •  
"},{"location":"changelog/#parsers_30","title":"Parsers","text":"
     
  • Removed bots.parsers.openbl as the source is offline since end of may (#1018, https://twitter.com/sshblorg/status/854669263671615489)
    •  
  • Removed bots.parsers.proxyspy as the source is offline (#1031)
    •  
  • Shadowserver: Added Accessible SMB
    •  
  • bots.experts.ripencc_abuse_contact now has the two additional parameters query_ripe_stat_asn and query_ripe_stat_ip.   Deprecated parameter query_ripe_stat. New parameter mode.
    •  
  • bots.experts.certat_contact has been renamed to bots.experts.national_cert_contact_certat (#995)
    •  
  • bots.experts.cymru_whois ignores registry other (#996)
    •  
  • bots.parsers.alienvault.parser_otx: handle timestamps without floating point seconds
    •  
"},{"location":"changelog/#experts_31","title":"Experts","text":"
     
  • bots.experts.deduplicator: New parameter bypass to deactivate deduplication, default: False
    •  
"},{"location":"changelog/#v100dev8-beta-release-2017-06-14","title":"v1.0.0.dev8 Beta release (2017-06-14)","text":""},{"location":"changelog/#general-changes","title":"General changes","text":"
     
  • It's now configurable how often the bots are logging how much events they have sent, based on both the amount and time. (fixes #743)
    •  
  • switch from pycodestyle to pep8
    •  
"},{"location":"changelog/#configuration_7","title":"Configuration","text":"
     
  • Added log_processed_messages_count (500) and log_processed_messages_seconds (900) to defaults.conf.
    •  
  • http_timeout has been renamed to http_timeout_sec and http_timeout_max_tries has been added.    This setting is honored by bots.collectors.http.* and bots.collectors.mail.collector_mail_url, bots.collectors.rt (only http_timeout_sec), bots.outputs.restapi.output and bots.experts.ripencc_abuse_contact.
    •  
"},{"location":"changelog/#documentation_30","title":"Documentation","text":"
     
  • Minor fixes
    •  
  • Dropped install scripts, see INSTALL.md for more detailed instructions and explanations
    •  
  • Better structure of INSTALL.md
    •  
  • Better documentation of packages
    •  
"},{"location":"changelog/#tools_28","title":"Tools","text":"
     
  • added a bot debugger (#975)
    •  
  • missing bot executable is detected and handled by intelmqctl (#979)
    •  
"},{"location":"changelog/#core_34","title":"Core","text":"
     
  • fix bug which prevented dumps to be written if the file did not exist (#986)
    •  
  • Fix reload of bots regarding logging
    •  
  • type annotations for all core libraries
    •  
"},{"location":"changelog/#bots_36","title":"Bots","text":"
     
  • added bots.experts.idea, bots.outputs.files
    •  
  • possibility to split large csv Reports into Chunks, currently possible for mail url and file collector
    •  
  • elasticsearch output supports HTTP Basic Auth
    •  
  • bots.collectors.mail.collector_mail_url and bots collectors.file.collector can split large reports (#680)
    •  
  • bots.parsers.shadowserver support the VNC feed
    •  
  • handling of HTTP timeouts, see above #859
    •  
  • bots.parsers.bambenek saves the malware name
    •  
  • bots.parsers.fraunhofer.parser_dga saves the malware name
    •  
  • bots.parsers.shadowserver handles NULL bytes
    •  
  • bots.parsers.abusech.parser_ransomware handles the IP 0.0.0.0 specially
    •  
"},{"location":"changelog/#harmonization_11","title":"Harmonization","text":"
     
  • New field named output to support export to foreign formats
    •  
"},{"location":"changelog/#v100dev7-beta-release-2017-05-09","title":"v1.0.0.dev7 Beta release (2017-05-09)","text":""},{"location":"changelog/#documentation_31","title":"Documentation","text":"
     
  • more verbose installation and upgrade instructions
    •  
"},{"location":"changelog/#bots_37","title":"Bots","text":""},{"location":"changelog/#collectors_26","title":"Collectors","text":"
     
  • bots.collectors.alienvault_otx: OTX library has been removed, install it as package instead
    •  
"},{"location":"changelog/#parsers_31","title":"Parsers","text":"
     
  • API keys will be removed from feed.url if possible
    •  
  • intelmq.bots.parsers.shadowserver.config:
    •  
  • Added support for Compromised-Website, Open-Netis, NTP-Version, Sandbox-URL, Spam-URL, Vulnerable-ISAKMP, Botnet-CCIP, Accessible-RDP, Open-LDAP, Blacklisted-IP, Accessible-Telnet, Accessible-CWMP (#748).
    •  
"},{"location":"changelog/#experts_32","title":"Experts","text":"
     
  • added bots.experts.field_reducer, bots.outputs.smtp.
    •  
  • bots.experts.deduplicator: ignore_keys has been renamed to filter_keys and filter_type has been removed.
    •  
  • bots.experts.modify: The configuration is now list-based for a consistent ordering.
    •  
  • bots.experts.tor_node as an optional parameter overwrite.
    •  
"},{"location":"changelog/#harmonization_12","title":"Harmonization","text":"
     
  • New parameter and field named feed.documentation to link to documentation of the feed
    •  
  • classification.taxonomy is lower case only
    •  
"},{"location":"changelog/#v100dev6-beta-release-2017-01-11","title":"v1.0.0.dev6 Beta release (2017-01-11)","text":"
Changes between 0.9 and 1.0.0.dev6
"},{"location":"changelog/#general-changes_1","title":"General changes","text":"
     
  • Dropped support for Python 2, Python >= 3.3 is needed
    •  
  • Dropped startup.conf and system.conf. Sections in BOTS can be copied directly to runtime.conf now.
    •  
  • Support two run modes: 'stream' which is the current implementation and a new one 'scheduled' which allows scheduling via cron or systemd.
    •  
  • Helper classes for parser bots
    •  
  • moved intelmq/conf to intelmq/etc
    •  
  • cleanup in code and repository
    •  
  • All bots capable of reloading on SIGHUP
    •  
  • packages
    •  
  • pip wheel format instead of eggs
    •  
  • unittests for library and bots
    •  
  • bots/BOTS now contains only generic and specific collectors. For a list of feeds, see docs/Feeds.md
    •  
"},{"location":"changelog/#tools_29","title":"Tools","text":"
     
  • DEV: intelmq_gen_harm_docs: added to generate Harmonization documentation
    •  
  • intelmq_psql_initdb: creates a table for a postgresql database using the harmonization fields
    •  
  • intelmqctl: reworked argument parsing, many bugfixes
    •  
  • intelmqdump: added to inspect dumped messages and reinsert them into the queues
    •  
  • DEV: rewrite_config_files: added to rewrite configuration files with consistent style
    •  
"},{"location":"changelog/#bots_38","title":"Bots","text":""},{"location":"changelog/#collectors_27","title":"Collectors","text":"
     
  • added alienvault, alienvault otx, bitsight, blueiv, file, ftp, misp, n6, rtir, xmpp collector
    •  
  • removed hpfeeds collector
    •  
  • removed microsoft DCU collector
    •  
  • renamed and reworked URL collector to HTTP
    •  
  • reworked Mail collectors
    •  
"},{"location":"changelog/#parsers_32","title":"Parsers","text":"
     
  • source specific parsers added: abusech, alienvault, alienvault otx, anubisnetworks, autoshun, bambenek, bitcash, bitsight, blocklistde, blueliv, ci army, cleanmx, cymru_full_bogons, danger_rulez, dataplane, dshield (asn, block and domain), dyn, fraunhofer_dga, hphosts, malc0de, malwaredomains, misp, n6, netlab_360, nothink, openphish, proxyspy, spamhaus cert, taichung, turris, urlvir
    •  
  • generic parsers added: csv, json
    •  
  • specific parsers dropped: abusehelper (broken), arbor (source unavailable), bruteforceblocker, certeu, dragonresearchgroup parser (discontinued), hpfeeds, microsoft_dcu (broken), taichungcitynetflow, torexitnode parser
    •  
  • renamed intelmq.bots.parsers.spamhaus.parser to intelmq.bots.parsers.spamhaus.parser_drop.   renamed intelmq.bots.parsers.malwarepatrol.parser-dansguardian tointelmq.bots.parsers.malwarepatrol.parser_dansguardian`
    •  
  • renamed intelmq.bots.parsers.taichungcitynetflow.parser tointelmq.bots.parsers.taichung.parser`
    •  
  • major rework of shadowserver parsers
    •  
  • enhanced all parsers
    •  
"},{"location":"changelog/#experts_33","title":"Experts","text":"
     
  • Added experts: asnlookup, cert.at contact lookup, filter, generic db lookup, gethostbyname, modify, reverse dns, rfc1918, tor_nodes, url2fqdn
    •  
  • removed experts: contactdb, countrycodefilter (obsolete), sanitizer (obsolete)
    •  
  • renamed intelmq.bots.experts.abusix.abusix to intelmq.bots.experts.abusix.expert intelmq.bots.experts.asnlookup.asnlookup to intelmq.bots.experts.asn_lookup.expert intelmq.bots.experts.cymru.expert to intelmq.bots.experts.cymru_whois.expert intelmq.bots.experts.deduplicator.deduplicator to intelmq.bots.experts.deduplicator.expert intelmq.bots.experts.geoip.geopip to intelmq.bots.experts.maxmind_geoip.expert intelmq.bots.experts.ripencc.ripencc to intelmq.bots.experts.ripencc_abuse_contact.expert intelmq.bots.experts.taxonomy.taxonomy to intelmq.bots.experts.taxonomy.expert
    •  
  • enhanced all experts
    •  
  • changed configuration syntax for intelmq.bots.experts.modify to a more simple variant
    •  
"},{"location":"changelog/#outputs_25","title":"Outputs","text":"
     
  • added: amqp, elasticsearch, redis, restapi, smtp, stomp, tcp, udp, xmpp
    •  
  • removed: debug, intelmqmailer (broken), logcollector
    •  
  • enhanced all outputs
    •  
"},{"location":"changelog/#bug-fixes","title":"Bug fixes","text":"
     
  • FIX: all bots handle message which are None
    •  
  • FIX: various encoding issues resolved in core and bots
    •  
  • FIX: time.observation is generated in collectors, not in parsers
    •  
"},{"location":"changelog/#other-enhancements-and-changes","title":"Other enhancements and changes","text":"
     
  • TST: testing framework for core and tests. Newly introduced components should always come with proper unit tests.
    •  
  • ENH: intelmqctl has shortcut parameters and can clear queues
    •  
  • STY: code obeys PEP8, new code should always be properly formatted
    •  
  • DOC: Updated user and dev guide
    •  
  • Removed Message.contains, Message.update methods Message.add ignore parameter
    •  
"},{"location":"changelog/#configuration_8","title":"Configuration","text":"
     
  • ENH: New parameter and field named accuracy to represent the accuracy of each feed
    •  
  • Consistent naming \"overwrite\" to switch overwriting capabilities of bots (as opposed to override)
    •  
  • Renamed http_ssl_proxy to https_proxy
    •  
  • parameter hierarchical_output for many output bots
    •  
  • deduplicator bot has a new required parameter to configure deduplication mode filter_type
    •  
  • deduplicator bot key ignore_keys was renamed to filter_keys
    •  
  • The tor_nodes expert has a new parameter overwrite, which is by default false.
    •  
"},{"location":"changelog/#harmonization_13","title":"Harmonization","text":"
     
  • ENH: Additional data types: integer, float and Boolean
    •  
  • ENH: Added descriptions and matching types to all fields
    •  
  • DOC: harmonization documentation has same fields as configuration, docs are generated from configuration
    •  
  • BUG: FQDNs are only allowed in IDN representation
    •  
  • ENH: Removed UUID Type (duplicate of String)
    •  
  • ENH: New type LowercaseString and UppercaseString, doing automatic conversion
    •  
  • ENH: Removed UUID Type (duplicate of String)
    •  
  • ENH: FQDNs are converted to lowercase
    •  
  • ENH: regex, iregex and length checks when data is added to messages
    •  
"},{"location":"changelog/#most-important-changes","title":"Most important changes:","text":"
     
  • (source|destination).bgp_prefix is now (source|destination).network
    •  
  • (source|destination).cc is now (source|destination).geolocation.cc
    •  
  • (source|destination).reverse_domain_name is (source|destination).reverse_dns
    •  
  • (source|destination).abuse_contact is lower case only
    •  
  • misp_id changed to misp.event_uuid
    •  
  • protocol.transport added, a fixed list of values is allowed
    •  
  • protocol.application is lower case only
    •  
  • webshot_url is now screenshot_url
    •  
  • additional_information renamed to extra, must be JSON
    •  
  • os.name, os.version, user_agent removed in favor of extra
    •  
  • all hashes are lower case only
    •  
  • added malware.hash.(md5|sha1|sha256), removed malware.hash
    •  
  • New parameter and field named feed.accuracy to represent the accuracy of each feed
    •  
  • New parameter and field named feed.provider to document the name of the source of each feed
    •  
  • New field classification.identifier -classification.taxonomy is now lower case only
    •  
"},{"location":"changelog/#known-issues_29","title":"Known issues","text":"
     
  • Harmonization: hashes are not normalized and classified, see also issue #394 and pull #634
    •  
"},{"location":"changelog/#contrib_16","title":"Contrib","text":"
     
  • ansible and vagrant scripts added
    •  
  • bash-completion for shells add
    •  
  • cron job scripts to update lookup data added
    •  
  • logcheck example rules added
    •  
  • logrotate configuration added
    •  
"},{"location":"changelog/#20160618","title":"2016/06/18","text":"
     
  • improvements in pipeline:
    •  
  • PipelineFactory to give possibility to easily add a new broker (Redis, ZMQ, etc..)
    •  
  • Splitter feature: if this option is enable, will split the events in source queue to multiple destination queues
    •  
  • add different messages support:
    •  
  • the system is flexible to define a new type of message like 'tweet' without change anything in bot.py, pipeline.py. Just need to add a new class in message.py and harmonization.conf
    •  
  • add harmonization support
    •  
  • in harmonization.conf is possible to define the fields of a specific message in json format.
    •  
  • the harmonization.py has data types witch contains sanitize and validation methods that will make sure that the values are correct to be part of an event.
    •  
  • Error Handling
    •  
  • multiple parameters in configuration which gives possibility to define how bot will handle some errors. Example of parameters:
    •  
  • error_procedure - retry or pass in case of error
    •  
  • error_retry_delay - time in seconds to retry
    •  
  • error_max_retries - number of retries
    •  
  • error_log_message - log or not the message in error log
    •  
  • error_log_exception - log or not the exception in error log
    •  
  • error_dump_message - log or not the message in dump log to be fixed and re-insert in pipeline
    •  
  • Exceptions
    •  
  • custom exceptions for IntelMQ
    •  
  • Defaults configurations
    •  
  • new configuration file to specify the default parameters which will be applied to all bots. Bots can overwrite the configurations.
    •  
  • New bots/feeds
    •  
"},{"location":"changelog/#20150603-aaron","title":"2015/06/03 (aaron)","text":"
     
  • fixed the license to AGPL in setup.py
    •  
  • moved back the documentation from the wiki repo to docs/. See #205.
    •  
  • added python-zmq as a setup requirement in UserGuide . See #206
    •  
"},{"location":"community/","title":"Community","text":""},{"location":"community/#intelmq-organizational-structure","title":"IntelMQ Organizational Structure","text":"
The central IntelMQ components are maintained by multiple people and organizations in the IntelMQ community. Please note that some components of the IntelMQ Universe can have a different project governance, but all are part of the IntelMQ universe and community.
"},{"location":"community/#intelmq-enhancement-proposals-iep","title":"IntelMQ Enhancement Proposals (IEP)","text":"
Major changes, including architecture, strategy and the internal data format, require so-called IEPs, IntelMQ Enhancement Proposals. Their name is based on the famous \"PEPs\" of Python.
 
IEPs are collected in the separate IEP Repository.
"},{"location":"community/#code-reviews-and-merging","title":"Code-Reviews and Merging","text":"
Every line of code checked in for the IntelMQ Core, is checked by at least one trusted developer (excluding the author of the changes) of the IntelMQ community. Afterwards, the code can be merged. Currently, these three contributors, have the permission to push and merging code to IntelMQ Core, Manager and API:
 
     
  • Aaron Kaplan (aaronkaplan)
    •  
  • Sebastian Wagner (sebix)
    •  
  • Sebastian Waldbauer (waldbauer-certat)
    •  
 
Additionally, these people significantly contributed to IntelMQ:
 
     
  • Bernhard Reiter
    •  
  • Birger Schacht
    •  
  • Edvard Rejthar
    •  
  • Filip Pokorn\u00fd
    •  
  • Karl-Johan Karlsson
    •  
  • Marius Karotkis
    •  
  • Marius Urkus
    •  
  • Mikk Margus M\u00f6ll
    •  
  • navtej
    •  
  • Pavel K\u00e1cha
    •  
  • Robert \u0160efr
    •  
  • Tomas Bellus
    •  
  • Zach Stone
    •  
"},{"location":"community/#short-history","title":"Short history","text":"
In 2013 and 2014 Aaron Kaplan (back then working at CERT.at) was researching ways to improve the automation of handling and distributing (IT security) incident reports across a whole country as part of the job of a national CERT. We would get many notifications of vulnerable systems, hacked systems, phishing domains, etc etc. The amount of reports we were getting required an automated solution. Back then, Aaron and a couple of other people looked at a tool called \"Abusehelper\". There was an open source version of Abusehelper, but it was deemed quite complex and complicated at that time.
 
Frustration with this tool led to discussions amongst multiple CERTs.
 
The idea and overall concept of an free, truly open source, simple (KISS principle! Keep it simple, stupid) community owned and maintained, extendible software for automated incident handling was born at an meeting of several European CSIRTs in Heraklion, Greece, in 2014. Following the event, Tom\u00e1s Lima \"SYNchroACK\" (working at CERT.pt back then) created IntelMQ from scratch. IntelMQ was born on June 24th, 2014. A major support came from CERT.pt at this early stage. Aaron Kaplan (CERT.at until 2020) engaged in the long-term advancement and from 2015 on, CERT.at took the burden of the maintenance and development (Sebastian Wagner 2015-2021 at CERT.at). From 2016 onward, CERT.at started projects, initiated and lead by Aaron Kaplan, receiving CEFF-funding from the European Union to support IntelMQ's development. IntelMQ became a software component of the EU-funded MeliCERTes framework for CSIRTs. In 2020, IntelMQ's organizational structure and architectural development gained new thrive by the newly founded Board and the start of the IEP process, creating more structure and more transparency in the IntelMQ community's decisions.
"},{"location":"help/","title":"Help","text":""},{"location":"help/#getting-help","title":"Getting help","text":"
In case you are lost, you need assistance or something is not discussed in this guide, you can ask the community for help. To be most efficient in seeking help, please describe your problem or question with all necessary information, for example:
 
     
  • Name and version of the operating system
    •  
  • Way of installation (deb/rpm packages, PyPI, docker, local git repository)
    •  
  • Used bots and configuration
    •  
  • Logs of bots or terminal output
    •  
  • Any other useful messages, screenshots
    •  
 
Please report any errors and suggest improvements via issues. Thank you!
"},{"location":"help/#github","title":"GitHub","text":"
GitHub offers a discussion platform where you can ask questions and seek assistance.
 
To report bugs, GitHub issues are the ideal place to do so. Every IntelMQ component has it's own repository on GitHub, with a separate Issue tracker.
 
To participate on GitHub, you first need to create an account on the platform.
"},{"location":"help/#mailing-list","title":"Mailing list","text":"
The most traditional way is to ask your question, make a proposal or discuss a topic on the mailing IntelMQ Users mailing list. You need to subscribe to the mailing list before posting, but the archive is publicly available: IntelMQ Users Archive.
"},{"location":"help/#assistance","title":"Assistance","text":"
If your organisation is a member of the CSIRTs Network, you are eligible for support in the MeliCERTes project. You can also ask on for individual support, some members offer support, including, but not limited to:
 
     
  • Aaron Kaplan (founder of IntelMQ)
    •  
  • Institute for Common Good Technology (chairmen Sebastian Wager is an IntelMQ maintainer and developer)
    •  
  • Intevation GmbH (Develops and maintains several IntelMQ components)
    •  
"},{"location":"overview/","title":"Overview","text":""},{"location":"overview/#overview","title":"Overview","text":"
The complete IntelMQ universe consists of the following components:
 
     
  • IntelMQ
    •  
  • IntelMQ API
    •  
  • IntelMQ Manager
    •  
  • additional tools
    •  
  • useful scripts
    •  
"},{"location":"overview/#intelmq","title":"IntelMQ","text":"
This project contains the core functionality.
 
The Core includes all the components required for processing data feeds. This includes the bots, configuration, pipeline, the internal data format, management tools etc.
 
\u2192 Repository: IntelMQ
"},{"location":"overview/#intelmq-api","title":"IntelMQ API","text":"
This is an extension of IntelMQ providing hug based REST API for remote management.
 
\u2192 Repository: IntelMQ API
"},{"location":"overview/#intelmq-manager","title":"IntelMQ Manager","text":"
The Manager is the most known software and can be seen as the face of IntelMQ. It's goal is to provide an intuitive web interface to allow non-programmers to specify the data flow in IntelMQ.
 
\u2192 Repository: IntelMQ Manager
 
"},{"location":"overview/#additional-tools","title":"Additional tools","text":"
Here you can find a list of additional tools. If you think something is missing, please let us know!
 
Unless stated otherwise, the tools are maintained by the IntelMQ community.
"},{"location":"overview/#intelmq-webinput-csv","title":"IntelMQ Webinput CSV","text":"
A web-based interface to ingest CSV data into IntelMQ with on-line validation and live feedback.
 
This interface allows inserting \"one-shot\" data feeds into IntelMQ without the need to configure bots in IntelMQ.
 
Developed and maintained by CERT.at.
 
\u2192 Repository: intelmq-webinput-csv
 
"},{"location":"overview/#intelmq-mailgen","title":"IntelMQ Mailgen","text":"
A solution allowing an IntelMQ setup with a complex contact database, managed by a web interface and sending out aggregated email reports. In different words: To send grouped notifications to network owners using SMTP.
 
Developed and maintained by Intevation, initially funded by BSI.
 
It consists of the following three components, which can also be used on their own.
"},{"location":"overview/#intelmq-certbund-contact","title":"IntelMQ CertBUND Contact","text":"
The certbund-contact consists of two IntelMQ expert bots, which fetch and process the information from the contact database, and scripts to import RIPE data into the contact database. Based on user-defined rules, the experts determine to which contact the event is to be sent to, and which e-mail template and attachment format to use.
 
\u2192 Repository: intelmq-certbund-contact
"},{"location":"overview/#intelmq-fody","title":"IntelMQ Fody","text":"
Fody is a web based interface for Mailgen. It allows to read and edit contacts, query sent mails (tickets) and call up data from the PostgreSQL database.
 
It can also be used to just query the database without using Mailgen.
 
 
\u2192 Repository: intelmq-fody
 
\u2192 Repository: intelmq-fody-backend
"},{"location":"overview/#intelmq-mailgen_1","title":"intelmq-mailgen","text":"
Sends emails with grouped event data to the contacts determined by the certbund-contact. Mails can be encrypted with PGP.
 
\u2192 Repository: intelmq-mailgen
"},{"location":"overview/#constituency-portal-tuency","title":"\"Constituency Portal\" tuency","text":"
A web application helping CERTs to enable members of their constituency to self-administrate how they get warnings related to their network objects (IP addresses, IP ranges, autonomous systems, domains). tuency is developed by Intevation for CERT.at.
 
If features organizational hierarchies, contact roles, self-administration and network objects per organization (Autonomous systems, network ranges, (sub)domains, RIPE organization handles). A network object claiming and approval process prevents abuse. An hierarchical rule-system on the network objects allow fine-grained settings. The tagging system for contacts and organization complement the contact-management features of the portal. Authentication is based on keycloak, which enables the re-use of the user accounts in the portal. The integrated API enables IntelMQ to query the portal for the right abuse contact and notification settings with the intelmq.bots.experts.tuency.expert expert bot.
 
 
\u2192 Repository: tuency
"},{"location":"overview/#constituency-portal-do-portal-deprecated","title":"\"Constituency Portal\" do-portal (deprecated)","text":"
Warning
 
The do-portal is deprecated and succeeded by tuency.
 
A contact portal with organizational hierarchies, role functionality and network objects based on RIPE, allows self-administration by the contacts. Can be queried from IntelMQ and integrates the stats-portal.
 
Originally developed by CERT-EU, then adapted by CERT.at.
 
\u2192 Repository: do-portal
"},{"location":"overview/#stats-portal","title":"Stats Portal","text":"
A Grafana-based statistics portal for the eventdb{.interpreted-text role=\"doc\"}. Can be integrated into do-portal. It uses aggregated data to serve statistical data quickly.
 
 
\u2192 Repository: stats-portal
"},{"location":"overview/#malware-name-mapping","title":"Malware Name Mapping","text":"
A mapping for malware names of different feeds with different names to a common family name.
 
\u2192 Repository: malware_name_mapping
"},{"location":"overview/#intelmq-docker","title":"IntelMQ-Docker","text":"
A repository with tools for IntelMQ docker instance.
 
Developed and maintained by CERT.at.
 
\u2192 Repository: intelmq-docker
"},{"location":"overview/#useful-scripts","title":"Useful scripts","text":"
The list of useful scripts contributed to the IntelMQ universe can be found in the main repository.
 
\u2192 Repository: intelmq/contrib
"},{"location":"security/","title":"Security","text":""},{"location":"security/#found-a-security-issue","title":"Found a security issue?","text":"
In case you find security-relevant bugs in IntelMQ, please contact team@cert.at. More information including the PGP key can be found on CERT.at's website.
"},{"location":"admin/beta-features/","title":"Beta Features","text":""},{"location":"admin/beta-features/#beta-features","title":"Beta Features","text":""},{"location":"admin/beta-features/#using-supervisor-as-a-process-manager","title":"Using Supervisor as a Process Manager","text":"
Warning
 
Do not use it in production environments yet! It has not been tested thoroughly yet.
 
Supervisor is process manager written in Python. The main advantage is that it take care about processes, so if bot process exit with failure (exit code different than 0), supervisor try to run it again. Another advantage is that it not require writing PID files.
 
This was tested on Ubuntu 18.04.
 
Install supervisor. supervisor_twiddler is extension for supervisor, that makes possible to create process dynamically. (Ubuntu supervisor package is currently based on Python 2, so supervisor_twiddler must be installed with Python 2 pip.)
 
apt install supervisor python-pip\npip install supervisor_twiddler\n
 
Create default config /etc/supervisor/conf.d/intelmq.conf and restart supervisor service:
 
[rpcinterface:twiddler]\nsupervisor.rpcinterface_factory=supervisor_twiddler.rpcinterface:make_twiddler_rpcinterface\n\n[group:intelmq]\n
 
Change IntelMQ process manager in the global configuration:
 
process_manager: supervisor\n
 
After this it is possible to manage bots like before with intelmqctl command.
"},{"location":"admin/beta-features/#using-amqp-message-broker","title":"Using AMQP Message Broker","text":"
Starting with IntelMQ 1.2 the AMQP protocol is supported as message queue. To use it, install a broker, for example RabbitMQ. The configuration and the differences are outlined here. Keep in mind that it is slower, but has better monitoring capabilities and is more stable. The AMQP support is considered beta, so small problems might occur. So far, only RabbitMQ as broker has been tested.
 
You can change the broker for single bots (set the parameters in the runtime configuration per bot) or for the whole botnet (using the global configuration).
 
You need to set the parameter source_pipeline_broker/destination_pipeline_broker to amqp. There are more parameters available:
 
Bug
 
This section of the documentation is currently incomplete and will be updated later.
 
destination_pipeline_broker
 
(required, string) \"amqp\"
 
destination_pipeline_host
 
()  (default: '127.0.0.1')
 
destination_pipeline_port
 
() (default: 5672)
 
destination_pipeline_username
 
()
 
destination_pipeline_password
 
()
 
destination_pipeline_socket_timeout
 
() (default: no timeout)
 
destination_pipeline_amqp_exchange
 
() Only change/set this if you know what you do. If set, the destination queues are not declared as queues, but used as routing key. (default: '').
 
destination_pipeline_amqp_virtual_host
 
() (default: '/')
 
source_pipeline_host
 
() (default: '127.0.0.1')
 
source_pipeline_port
 
() (default: 5672)
 
source_pipeline_username
 
()
 
source_pipeline_password
 
()
 
source_pipeline_socket_timeout
 
() (default: no timeout)
 
source_pipeline_amqp_exchange
 
() Only change/set this if you know what you do. If set, the destination queues are not declared as queues, but used as routing key. (default: ['']).
 
source_pipeline_amqp_virtual_host
 
() (default: '/')
 
intelmqctl_rabbitmq_monitoring_url
 
() string, see below (default: \"http://{host}:15672\")
 
For getting the queue sizes, intelmqctl needs to connect to the monitoring interface of RabbitMQ. If the monitoring interface is not available under http://{host}:15672 you can manually set using the parameter intelmqctl_rabbitmq_monitoring_url. In a RabbitMQ's default configuration you might not provide a user account, as by default the administrator (guest:guest) allows full access from localhost. If you create a separate user account, make sure to add the tag \"monitoring\" to it, otherwise IntelMQ can't fetch the queue sizes.
 
 
Setting the statistics (and cache) parameters is necessary when the local redis is running under a non-default host/port. If this is the case, you can set them explicitly:
 
statistics_database
 
() 3
 
statistics_host
 
() \"127.0.0.1\"
 
statistics_password
 
() null
 
statistics_port
 
() 6379
"},{"location":"admin/beta-features/#multithreading","title":"Multithreading","text":"
First of all: Do not use it in production environments yet! There are a few bugs, see below
 
Since IntelMQ 2.0 it is possible to provide the following runtime parameter:
 
instances_threads
 
Set it to a non-zero integer, then this number of worker threads will be spawn. This is useful if bots often wait for system resources or if network-based lookups are a bottleneck.
 
However, there are currently a few cavecats:
 
     
  • This is not possible for all bots, there are some exceptions (collectors and some outputs), see the FAQ for some reasons.
    •  
  • Only use it with the AMQP pipeline, as with Redis, messages may get duplicated because there's only one internal   queue
    •  
  • In the logs, you can see the main thread initializing first, then all of the threads which log with the   name [bot-id].[thread-id].
    •  
"},{"location":"admin/common-problems/","title":"Common Problems","text":""},{"location":"admin/common-problems/#common-problems","title":"Common Problems","text":""},{"location":"admin/common-problems/#intelmq","title":"IntelMQ","text":""},{"location":"admin/common-problems/#permission-denied-when-using-redis-unix-socket","title":"Permission denied when using Redis Unix socket","text":"
If you get an error like this:
 
intelmq.lib.exceptions.PipelineError: pipeline failed - ConnectionError('Error 13 connecting to unix socket: /var/run/redis/redis.sock. Permission denied.',)\n
 
Make sure the intelmq user as sufficient permissions for the socket.
 
In /etc/redis/redis.conf (or wherever your configuration is), check the permissions and set it for example to group-writeable:
 
unixsocketperm 770\n
 
And add the user intelmq to the redis-group:
 
usermod -aG redis intelmq\n
"},{"location":"admin/common-problems/#my-bots-died-on-startup-with-no-errors-logged","title":"My bot(s) died on startup with no errors logged","text":"
Rather than starting your bot(s) with intelmqctl start, try intelmqctl run [bot]. This will provide valuable debug output you might not otherwise see, pointing to issues like system configuration errors.
"},{"location":"admin/common-problems/#orphaned-queues","title":"Orphaned Queues","text":"
This section has been moved to the Management Guide.
"},{"location":"admin/common-problems/#multithreading-is-not-available-for-this-bot","title":"Multithreading is not available for this bot","text":"
Multithreading is not available for some bots and AMQP broker is necessary. Possible reasons why a certain bot or a setup does not support Multithreading include:
 
     
  • Multithreading is only available when using the AMQP broker.
    •  
  • For most collectors, Multithreading is disabled. Otherwise this      would lead to duplicated data, as the data retrieval is not      atomic.
    •  
  • Some bots use libraries which are not thread safe. Look a the      bot's documentation for more information.
    •  
  • Some bots' operations are not thread safe. Look a the bot's      documentation for more information.
    •  
 
If you think this mapping is wrong, please report a bug.
"},{"location":"admin/common-problems/#intelmq-api","title":"IntelMQ API","text":""},{"location":"admin/common-problems/#intelmqctlerror","title":"IntelMQCtlError","text":"
If the command is not configured correctly, you will see exceptions on startup like this:
 
intelmq_manager.runctl.IntelMQCtlError: <ERROR_MESSAGE>\n
 
This means the intelmqctl command could not be executed as a subprocess. The <ERROR_MESSAGE> should indicate why.
"},{"location":"admin/common-problems/#access-denied-authentication-required-please-provide-valid-token-verification-credentials","title":"Access Denied / Authentication Required \"Please provide valid Token verification credentials\"","text":"
If you see the IntelMQ Manager interface and menu, but the API calls to the back-end querying configuration and status of IntelMQ fail with \"Access Denied\" or \"Authentication Required: Please provide valid Token verification credentials\" errors, you are maybe not logged in while the API requires authentication.
 
By default, the API requires authentication. Create user accounts and login with them or - if you have other protection means in place - deactivate the authentication requirement by removing or renaming the session_store parameter in the configuration.
"},{"location":"admin/common-problems/#internal-server-error","title":"Internal Server Error","text":"
There can be various reasons for internal server errors. You need to look at the error log of your web server, for example /var/log/apache2/error.log or /var/log/httpd/error_log for Apache 2. It could be that the sudo-setup is not functional, the configuration file or session database file can not be read or written or other errors in regards to the execution of the API program.
"},{"location":"admin/common-problems/#can-i-just-install-it-from-the-debrpm-packages-while-installing-intelmq-from-a-different-source","title":"Can I just install it from the deb/rpm packages while installing IntelMQ from a different source?","text":"
Yes, you can install the API and the Manager from the deb/rpm repositories, and install your IntelMQ from a somewhere else, e.g. a local repository. However, knowledge about Python and system administration experience is recommended if you do so.
 
The packages install IntelMQ to /usr/lib/python3*/site-packages/intelmq/. Installing with pip results in /usr/local/lib/python3*/site-packages/intelmq/ (and some other accompaning resources) which overrides the installation in /usr/lib/. You probably need to adapt the configuration parameter intelmq_ctl_cmd to the /usr/local/bin/intelmqctl executable and some other tweaks.
"},{"location":"admin/common-problems/#sqlite3operationalerror-attempt-to-write-a-readonly-database","title":"sqlite3.OperationalError: attempt to write a readonly database","text":"
SQLite does not only need write access to the database itself, but also the folder the database file is located in. Please check that the webserver has write permissions to the folder the session file is located in.
"},{"location":"admin/faq/","title":"FAQ","text":""},{"location":"admin/faq/#frequently-asked-questions","title":"Frequently asked questions","text":""},{"location":"admin/faq/#how-can-i-improve-the-speed","title":"How can I improve the speed?","text":"
In most cases the bottlenecks are look-up experts. In these cases you can easily use the integrated load balancing features.
"},{"location":"admin/faq/#multithreading","title":"Multithreading","text":"
When using the AMQP broker, you can make use of Multi-threading. See the multithreading section.
"},{"location":"admin/faq/#classic-load-balancing-multiprocessing","title":"\"Classic\" load-balancing (Multiprocessing)","text":"
Before Multithreading was available in IntelMQ, and in case you use Redis as broker, the only way to do load balancing involves more work. Create multiple instances of the same bot and connect them all to the same source and destination bots. Then set the parameter load_balance to true for the bot which sends the messages to the duplicated bot. Then, the bot sends messages to only one of the destination queues and not to all of them.
 
True Multiprocessing is not available in IntelMQ. See also this discussion on a possible enhanced load balancing <186>.
"},{"location":"admin/faq/#other-options","title":"Other options","text":"
For any bottleneck based on (online) lookups, optimize the lookup itself and if possible use local databases.
 
It is also possible to use multiple servers to spread the workload. To get the messages from one system to the other you can either directly connect to the other's pipeline or use a fast exchange mechanism such as the TCP Collector/Output (make sure to secure the network by other means).
"},{"location":"admin/faq/#removing-raw-data-for-higher-performance-and-less-space-usage","title":"Removing raw data for higher performance and less space usage","text":"
If you do not need the raw data, you can safely remove it. For events (after parsers), it keeps the original data, eg. a line of a CSV file. In reports it keeps the actual data to be parsed, so don't delete the raw field in Reports - between collectors and parsers.
 
The raw data consumes about 50% - 30% of the messages' size. The size of course depends on how many additional data you add to it and how much data the report includes. Dropping it, will improve the speed as less data needs to be transferred and processed at each step.
 
In a bot
 
You can do this for example by using the Field Reducer Expert. The configuration could be:
 
     
  • type: blacklist
    •  
  • keys: raw
    •  
 
Other solutions are the Modify bot and the Sieve bot. The last one is a good choice if you already use it and you only need to add the command:
 
remove raw\n
 
In the database
 
In case you store data in the database and you want to keep its size small, you can (periodically) delete the raw data there.
 
To remove the raw data for a events table of a PostgreSQL database, you can use something like:
 
UPDATE events SET raw = NULL WHERE \"time.source\" < '2018-07-01';\n
 
If the database is big, make sure only update small parts of the database by using an appropriate WHERE clause. If you do not see any negative performance impact, you can increase the size of the chunks, otherwise the events in the output bot may queue up. The id column can also be used instead of the source's time.
 
Another way of reducing the raw-data from the database is described in the EventDB documentation: eventdb_raws_table.
"},{"location":"admin/faq/#how-to-uninstall","title":"How to Uninstall","text":"
If you installed intelmq with native packages: Use the package management tool to remove the package intelmq. These tools do not remove configuration by default.
 
If you installed manually via pip (note that this also deletes all configuration and possibly data):
 
pip3 uninstall intelmq\nrm -r /opt/intelmq\n
"},{"location":"admin/hardware-requirements/","title":"Hardware Requirements","text":""},{"location":"admin/hardware-requirements/#hardware-requirements","title":"Hardware Requirements","text":"
Do you ask yourself how much RAM do you need to give your new IntelMQ virtual machine?
 
The honest answer is simple and pointless: It depends ;)
"},{"location":"admin/hardware-requirements/#intelmq-and-the-messaging-queue-broker","title":"IntelMQ and the messaging queue (broker)","text":"
IntelMQ uses a messaging queue to move the messages between the bots. All bot instances can only process one message at a time, therefore all other messages need to wait in the queue. As not all bots are equally fast, the messages will naturally \"queue up\" before the slower ones. Further, parsers produce many events with just one message (the report) as input.
 
The following estimations assume Redis as messaging broker which is the default for IntelMQ. When RabbitMQ is used, the required resources will differ, and RabbitMQ can handle system overload and therefore a shortage of memory.
 
As Redis stores all data in memory, the data which is processed at any point in time must fit there, including overheads. Please note that IntelMQ does neither store nor cache any input data. These estimates therefore only relate to the processing step, not the storage.
 
For a minimal system, these requirements suffice:
 
     
  • 4 GB of RAM
    •  
  • 2 CPUs
    •  
  • 10 GB disk size
    •  
 
Depending on your data input, you will need the twentiethfold of the input data size as memory for processing.
 
When using Redis persistence, you will additionally need twice as much memory for Redis.
"},{"location":"admin/hardware-requirements/#disk-space","title":"Disk space","text":"
Disk space is only relevant if you save your data to a file, which is not recommended for production setups, and only useful for testing and evaluation.
 
Do not forget to rotate your logs or use syslog, especially if you use the logging level \"DEBUG\". logrotate is in use by default for all installation with deb/rpm packages. When other means of installation are used (pip, manual), configure log rotation manually. See logging configuration.
"},{"location":"admin/hardware-requirements/#background-on-memory","title":"Background on memory","text":"
For experimentation, we used multiple Shadowserver Poodle reports for demonstration purpose, totaling in 120 MB of data. All numbers are estimates and are rounded. In memory, the report data requires 160 MB. After parsing, the memory usage increases to 850 MB in total, as every data line is stored as JSON, with additional information plus the original data encoded in Base 64. The further processing steps depend on the configuration, but you can estimate that caches (for lookups and deduplication) and other added information cause an additional size increase of about 2x. Once a dataset finished processing in IntelMQ, it is no longer stored in memory. Therefore, the memory is only needed to catch high load.
 
The above numbers result in a factor of 14 for input data size vs. memory required by Redis. Assuming some overhead and memory for the bots' processes, a factor of 20 seems sensible.
 
To reduce the amount of required memory and disk size, you can optionally remove the raw data field, see this section in the FAQ.
"},{"location":"admin/hardware-requirements/#additional-components","title":"Additional components","text":"
If some of the optional components are in use, they can add additional hardware requirements.
 
Those components do not add relevant requirements:
 
     
  • IntelMQ API: It is just an API for intelmqctl.
    •  
  • IntelMQ Manager: Only contains static files served by the webserver.
    •  
  • IntelMQ Webinput CSV: Just a webinterface to insert data. Requires     the amount of processed data to fit in memory, see above.
    •  
  • Stats Portal: The aggregation step and Graphana require some     resources, but no exact numbers are known.
    •  
  • Malware Name Mapping
    •  
  • Docker: The docker layer adds only minimal hardware requirements.
    •  
"},{"location":"admin/hardware-requirements/#database","title":"Database","text":"
When storing data in databases (such as MongoDB, PostgreSQL, ElasticSearch), it is recommended to do this on separate machines for operational reasons. Using a different machine results in a separation of stream processing to data storage and allows for a specialized system optimization for both use-cases.
"},{"location":"admin/hardware-requirements/#intelmq-cb-mailgen","title":"IntelMQ cb mailgen","text":"
While the Fody backend and frontend do not have significant requirements, the RIPE import tool of the certbund-contact requires about 8 GB of memory as of March 2021.
"},{"location":"admin/intro/","title":"Intro","text":""},{"location":"admin/intro/#intro","title":"Intro","text":"
This guide provides instructions on how to install, configure and manage IntelMQ and it's components.
 
IntelMQ uses a message broker such as Redis. This is required for IntelMQ to run.
 
IntelMQ doesn't handle long term storage of processed Events beyond writing to a file. However it provides connectors (output bots) for writing events to various database systems and log collectors. It is recommended to configure such system to preserve processed events.
"},{"location":"admin/intro/#base-requirements","title":"Base Requirements","text":"
The following instructions assume the following requirements. Python versions >= 3.7 are supported.
 
Supported and recommended operating systems are:
 
     
  • Debian
    •  
  • openSUSE Tumbleweed/Leap
    •  
  • Ubuntu
    •  
  • For the Docker-installation: Docker Engine: 18.x and higher
    •  
 
Other distributions which are (most probably) supported include AlmaLinux, CentOS, Fedora, FreeBSD 12, RHEL and RockyLinux.
 
A short guide on hardware requirements can be found on the page Hardware Requirements.
"},{"location":"admin/upgrade/","title":"Upgrade","text":""},{"location":"admin/upgrade/#upgrade-instructions","title":"Upgrade instructions","text":"
In order to upgrade your IntelMQ installation it is recommended to follow these five steps:
"},{"location":"admin/upgrade/#1-read-newsmd","title":"1. Read NEWS.md","text":"
Read the NEWS.md file to look for things you need to have a look at.
"},{"location":"admin/upgrade/#2-stop-intelmq-and-create-a-backup","title":"2. Stop IntelMQ and create a backup","text":"
     
  • Make sure that your IntelMQ system is completely stopped: intelmqctl stop
    •  
  • Create a backup of IntelMQ Home directory, which includes all configurations. They are not overwritten, but backups are always nice to have!
    •  
 
sudo cp -R /opt/intelmq /opt/intelmq-backup\n
"},{"location":"admin/upgrade/#3-upgrade-intelmq","title":"3. Upgrade IntelMQ","text":"
Before upgrading, check that your setup is clean and there are no events in the queues:
 
intelmqctl check\nintelmqctl list queues -q\n
 
The upgrade depends on how you installed IntelMQ.
"},{"location":"admin/upgrade/#linux-packages","title":"Linux Packages","text":"
Use your system's package manager.
"},{"location":"admin/upgrade/#pypi","title":"PyPi","text":"
pip install -U --no-deps intelmq\nsudo intelmqsetup\n
 
Using --no-deps will not upgrade dependencies, which would probably overwrite the system's libraries. Remove this option to also upgrade dependencies.
"},{"location":"admin/upgrade/#docker","title":"Docker","text":"
You can check out all current versions on our DockerHub.
 
docker pull certat/intelmq-full:latest\n\ndocker pull certat/intelmq-nginx:latest\n
 
Alternatively you can use docker-compose:
 
docker-compose pull\n
 
You can check the current versions from intelmq & intelmq-manager & intelmq-api via git commit ref.
 
The Version format for each included item is key=value and they are saparated via ,. I. e. IntelMQ=ab12cd34f,IntelMQ-API=xy65z23.
 
docker inspect --format '{{ index .Config.Labels \"org.opencontainers.image.version\" }}' intelmq-full:latest\n
 
Now restart your container, if you're using docker-compose you simply run:
 
docker-compose down\n
 
If you don't use docker-compose, you can restart a single container using:
 
docker ps | grep certat\n\ndocker restart CONTAINER_ID\n
"},{"location":"admin/upgrade/#source-repository","title":"Source repository","text":"
If you have an editable installation, refer to the instructions in the /dev/guide.
 
Update the repository depending on your setup (e.g. [git pull origin master]).
 
And run the installation again:
 
pip install .\nsudo intelmqsetup\n
 
For editable installations (development only), run [pip install -e .] instead.
"},{"location":"admin/upgrade/#4-upgrade-configuration-and-check-the-installation","title":"4. Upgrade configuration and check the installation","text":"
Go through NEWS.md and apply necessary adaptions to your setup. If you have adapted IntelMQ's code, also read the CHANGELOG.md.
 
Check your installation and configuration to detect any problems:
 
intelmqctl upgrade-config\nintelmqctl check\n
 
intelmqctl upgrade-config supports upgrades from one IntelMQ version to the succeeding. If you skip one or more IntelMQ versions, some automatic upgrades may not work and manual intervention may be necessary.
"},{"location":"admin/upgrade/#5-start-intelmq","title":"5. Start IntelMQ","text":"
intelmqctl start\n
"},{"location":"admin/configuration/intelmq-api/","title":"IntelMQ API","text":""},{"location":"admin/configuration/intelmq-api/#configuring-intelmq-api","title":"Configuring IntelMQ API","text":"
Depending on your setup you might have to install sudo to make it possible for the intelmq-api to run the intelmq command as the user-account usually used to run intelmq (which is also often called intelmq).
 
intelmq-api is configured using a configuration file in json format. intelmq-api tries to load the configuration file from /etc/intelmq/api-config.json and ${PREFIX}/etc/intelmq/api-config.json, but you can override the path setting the environment variable INTELMQ_API_CONFIG. (When using Apache, you can do this by modifying the Apache configuration file shipped with intelmq-api, the file contains an example)
 
When running the API using hug, you can set the environment variable like this:
 
INTELMQ_API_CONFIG=/etc/intelmq/api-config.json hug -m intelmq_api.serve\n
 
The default configuration which is shipped with the packages is also listed here for reference:
 
{\n    \"intelmq_ctl_cmd\": [\"sudo\", \"-u\", \"intelmq\", \"intelmqctl\"],\n    \"allowed_path\": \"/opt/intelmq/var/lib/bots/\",\n    \"session_store\": \"/etc/intelmq/api-session.sqlite\",\n    \"session_duration\": 86400,\n    \"allow_origins\": [\"*\"]\n}\n
 
On Debian based systems, the default path for the session_store is /var/lib/dbconfig-common/sqlite3/intelmq-api/intelmqapi, because the Debian package uses the Debian packaging tools to manage the database file.
 
The following configuration options are available:
 
     
  • intelmq_ctl_cmd: Your intelmqctl command. If this is not set in     a configuration file the default is used, which is     [\"sudo\", \"-u\", \"intelmq\", \"/usr/local/bin/intelmqctl\"] The option     \"intelmq_ctl_cmd\" is a list of strings so that we can avoid     shell-injection vulnerabilities because no shell is involved when     running the command. This means that if the command you want to use     needs parameters, they have to be separate strings.
    •  
  • allowed_path: intelmq-api can grant read-only access to     specific files - this setting defines the path those files can     reside in.
    •  
  • session_store: this is an optional path to a sqlite database,     which is used for session storage and authentication. If it is not     set (which is the default), no authentication is used!
    •  
  • session_duration: the maximal duration of a session, its 86400     seconds by default
    •  
  • allow_origins: a list of origins the responses of the API can be     shared with. Allows every origin by default.
    •  
"},{"location":"admin/configuration/intelmq-api/#permissions","title":"Permissions","text":"
intelmq-api tries to write a couple of configuration files in the ${PREFIX}/etc/intelmq directory - this is only possible if you set the permissions accordingly, given that intelmq-api runs under a different user. The user the API run as also needs write access to the folder the session_store is located in, otherwise there will be an error accessing the session data. If you\\'re using the default Apache 2 setup, you might want to set the group of the files to www-data and give it write permissions (chmod -R g+w <directoryname>). In addition to that, the intelmq-manager tries to store the bot positions via the API into the file ${PREFIX}/etc/intelmq/manager/positions.conf. You should therefore create the folder ${PREFIX}/etc/intelmq/manager and the file positions.conf in it.
"},{"location":"admin/configuration/intelmq-api/#adding-a-user","title":"Adding a user","text":"
If you enable the session_store you will have to create user accounts to be able to access the API functionality. You can do this using intelmq-api-adduser:
 
intelmq-api-adduser --user <username> --password <password>\n
"},{"location":"admin/configuration/intelmq-api/#a-note-on-selinux","title":"A note on SELinux","text":"
On systems with SELinux enabled, the API will fail to call intelmqctl. Therefore, SELinux needs to be disabled:
 
setenforce 0\n
 
We welcome contributions to provide SELinux policies.
"},{"location":"admin/configuration/intelmq-manager/","title":"IntelMQ Manager","text":""},{"location":"admin/configuration/intelmq-manager/#configuring-intelmq-manager","title":"Configuring IntelMQ Manager","text":"
In the file /usr/share/intelmq-manager/html/js/vars.js set ROOT to the URL of your intelmq-api installation - by default that's on the same host as intelmq-manager.
"},{"location":"admin/configuration/intelmq-manager/#configuration-paths","title":"Configuration Paths","text":"
The IntelMQ Manager queries the configuration file paths and directory names from intelmqctl and therefore any global environment variables (if set) are effective in the Manager too. The interface for this query is intelmqctl debug --get-paths, the result is also shown in the /about.html page of your IntelMQ Manager installation.
"},{"location":"admin/configuration/intelmq-manager/#csp-headers","title":"CSP Headers","text":"
It is recommended to set these two headers for all requests:
 
Content-Security-Policy: script-src 'self'\nX-Content-Security-Policy: script-src 'self'\n
"},{"location":"admin/configuration/intelmq-manager/#security-considerations","title":"Security considerations","text":"
Never ever run intelmq-manager on a public webserver without SSL and proper authentication!
 
The way the current version is written, anyone can send a POST request and change intelmq's configuration files via sending HTTP POST requests. Intelmq-manager will reject non JSON data but nevertheless, we don't want anyone to be able to reconfigure an intelmq installation.
 
Therefore you will need authentication and SSL. Authentication can be handled by the intelmq-api. Please refer to its documentation on how to enable authentication and setup accounts.
 
Never ever allow unencrypted, unauthenticated access to IntelMQ Manager!
"},{"location":"admin/configuration/intelmq-manager/#docker-security-headers","title":"Docker: Security headers","text":"
If you run our docker image in production, we recommend you to set security headers. You can do this by creating a new file called example_config/nginx/security.conf in the cloned intelmq-docker repository.
 
Write the following inside the configuration file, and change the http(s)://<your-domain> to your domain name.
 
server_tokens off; # turn off server_token, instead of nginx/13.2 now it will only show nginx\nadd_header X-Frame-Options SAMEORIGIN; # https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options\nadd_header X-Content-Type-Options nosniff; # https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Content-Type-Options\nadd_header X-XSS-Protection \"1; mode=block\"; # https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-XSS-Protection\nadd_header Content-Security-Policy \"script-src 'self' 'unsafe-inline' http(s)://<your-domain>; frame-src 'self' http(s)://<your-domain>; object-src 'self' http(s)://<your-domain>\"; # https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP\n
 
After you created the file, edit the docker-compose.yml and mount it to the nginx with
 
volumes:\n  - ./example_config/nginx/security.conf:/etc/nginx/conf.d/security.conf\n
 
IMPORTANT Mount the exact name & not the directory, because otherwise you would overwrite the whole directory and the other files would be gone inside the container.
"},{"location":"admin/configuration/intelmq/","title":"IntelMQ","text":""},{"location":"admin/configuration/intelmq/#configuring-intelmq","title":"Configuring IntelMQ","text":""},{"location":"admin/configuration/intelmq/#directories","title":"Directories","text":""},{"location":"admin/configuration/intelmq/#lsb","title":"LSB","text":"
If you installed the packages, standard Linux paths (LSB paths) are used:
 
     
  • /etc/intelmq/ (configurations)
    •  
  • /var/log/intelmq/ (logs)
    •  
  • /var/lib/intelmq/ (local states)
    •  
  • /var/run/intelmq/ (PID files)
    •  
 
Otherwise, the configuration directory is /opt/intelmq/etc/. Using the environment variable INTELMQ_ROOT_DIR allows setting any arbitrary root directory.
 
You can switch this by setting the environment variables INTELMQ_PATHS_NO_OPT and INTELMQ_PATHS_OPT, respectively.
 
     
  • When installing the Python packages, you can set INTELMQ_PATHS_NO_OPT to something non-empty to use LSB-paths.
    •  
  • When installing the deb/rpm packages, you can set INTELMQ_PATHS_OPT to something non-empty to use /opt/intelmq/ paths, or a path set with INTELMQ_ROOT_DIR.
    •  
 
The environment variable ROOT_DIR is meant to set an alternative root directory instead of /. This is primarily meant for package build environments an analogous to setuptool's --root parameter. Thus it is only used in LSB-mode.
"},{"location":"admin/configuration/intelmq/#environment-variables","title":"Environment Variables","text":"Name Type Description INTELMQ_PATHS_OPT INTELMQ_PATHS_NO_OPT INTELMQ_ROOT_DIR ROOT_DIR"},{"location":"admin/configuration/intelmq/#configuration-files","title":"Configuration Files","text":""},{"location":"admin/configuration/intelmq/#runtimeyaml","title":"runtime.yaml","text":"
This is the main configuration file. It uses YAML format since IntelMQ 3.0. It consists of two parts:
 
     
  • Global Configuration
    •  
  • Individual Bot Configuration
    •  
 
Warning
 
Comments in YAML are currently not preserved by IntelMQ (known bug #2003).
 
Example runtime.yaml configuration file is installed by the tool intelmqsetup. If this is not the case, make sure the program was run. It is shipped preconfigured with 4 collectors and parsers, 6 common experts and one output bot. The default collector and the parser handle data from malware domain list, the file output bot writes all data to one of these files (based on your installation):
 
     
  •  
    /opt/intelmq/var/lib/bots/file-output/events.txt
     
    •  
  •  
    /var/lib/intelmq/bots/file-output/events.txt
     
    •  
 
The runtime.yaml configuration is divided into two sections:
 
     
  • Global configuration which is applied to each bot.
    •  
  • Individual bot configuration which overloads the global configuration and contains bot specific options.
    •  
 
Example configuration snippet:
 
global: # global configuration section\n  # ...\n  http_timeout_max_tries: 3\n  http_timeout_sec: 30\n  http_user_agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36\n  http_verify_cert: true\n\nblocklistde-apache-collector: # individual bot configuration section\n  group: Collector\n  name: Blocklist.de Apache List\n  module: intelmq.bots.collectors.http.collector_http\n  description: Blocklist.de Apache Collector fetches all IP addresses which have been reported within the last 48 hours as having run attacks on the service Apache, Apache-DDOS, RFI-Attacks.\n  parameters:\n    http_url: https://lists.blocklist.de/lists/apache.txt\n    name: Blocklist.de Apache\n    rate_limit: 3600\n    http_verify_cert: false # overriding the global configuration for this particular bot\n
"},{"location":"admin/configuration/intelmq/#global-configuration","title":"Global Configuration","text":"
The global configuration parameters apply to all bots, however they can be overridden in the individual bot configuration.
"},{"location":"admin/configuration/intelmq/#logging","title":"Logging","text":"
The logging can be configured with the following parameters:
 
logging_handler
 
(required, string) Allowed values are file or syslog.
 
logging_level
 
(required, string) Allowed values are CRITICAL, ERROR, WARNING, INFO or DEBUG. Defines the system-wide log level that will be use by all bots and the intelmqctl tool. We recommend logging_level WARNING for production environments and INFO if you want more details. In any case, watch your free disk space!
 
logging_path
 
(required, string) When the logging_handler is file this parameter is used to set the logging directory for all the bots as well as the intelmqctl tool. Defaults to /opt/intelmq/var/log/ or /var/log/intelmq/ respectively.
 
logging_syslog
 
(required, string) When the logging_handler is syslog. Either a list with hostname and UDP port of syslog service, e.g. [\"localhost\", 514] or a device name/path. Defaults to /var/log.
"},{"location":"admin/configuration/intelmq/#log-rotation","title":"Log Rotation","text":"
To rotate the logs, you can use the standard Linux-tool logrotate. An example logrotate configuration is given in contrib/logrotate/ and delivered with all deb/rpm-packages. When not using logrotate, IntelMQ can rotate the logs itself, which is not enabled by default! You need to set both values.
 
logging_max_size
 
(optional, integer) Maximum number of bytes to be stored in one logfile before the file is rotated. Defaults to 0 (log rotation disabled).
 
logging_max_copies
 
(optional, integer) Maximum number of logfiles to keep. Compression is not supported. Default is unset.
 
Some information can as well be found in Python's documentation on the used RotatingFileHandler.
"},{"location":"admin/configuration/intelmq/#error-handling","title":"Error Handling","text":"
error_log_message
 
(required, boolean) Whether to write the message (Event/Report) to the log file in case of an error.
 
error_log_exception
 
(required, boolean) Whether to write an error exception to the log file in case of an error.
 
error_procedure
 
(required, string) Allowed values are stop or pass. In case of an error, this option defines the procedure that the bot will adopt. Use the following values:
 
     
  •  
    stop - stop bot after retrying X times (as defined in error_max_retries) with a delay between retries (as defined in error_retry_delay). If the bot reaches the error_max_retries value, it will remove the message from the pipeline and stop. If the option error_dump_message is also enable, the bot will dump the removed message to its dump file (to be found in var/log).
     
    •  
  •  
    pass - will skip this message and will process the next message after retrying X times, removing the current message from pipeline. If the option error_dump_message is also enable, then the bot will dump the removed message to its dump file. After max retries are reached, the rate limit is applied (e.g. a collector bot fetch an unavailable resource does not try forever).
     
    •  
 
error_max_retries
 
(required, integer) In case of an error, the bot will try to re-start processing the current message X times as   defined by this option.
 
error_retry_delay
 
(required, integer) Defines the number of seconds to wait between subsequent re-tries in case of an error.
 
error_dump_message
 
(required, boolean) Specifies if the bot will write queued up messages to its dump file (use intelmqdump to   re-insert the message).
 
If the path _on_error exists for a bot, the message is also sent to this queue, instead of (only) dumping the file if configured to do so.
"},{"location":"admin/configuration/intelmq/#miscellaneous","title":"Miscellaneous","text":"
load_balance
 
(required, boolean) this option allows you to choose the behavior of the queue. Use the following values:
 
     
  • true - splits the messages into several queues without duplication
    •  
  • false - duplicates the messages into each queue - When using AMQP as message broker, take a look at the multithreading{.interpreted-text role=\"ref\"} section and the instances_threads parameter.
    •  
 
rate_limit
 
(required, integer) time interval (in seconds) between messages processing. int value.
 
ssl_ca_certificate
 
(optional, string) trusted CA certificate for IMAP connections (supported by some bots).
 
source_pipeline_broker
 
(optional, string) Allowed values are redis and amqp. Selects the message broker IntelMQ should use. As this parameter can be overridden by each bot, this allows usage of different broker systems and hosts, as well as switching between them on the same IntelMQ instance. Defaults to redis.
 
     
  • redis - Please note that persistence has to be manually activated.
    •  
  • amqp - Using the AMQP broker is currently beta but there are no known issues. A popular AMQP broker is RabbitMQ.
    •  
 
destination_pipeline_broker
 
(required, string) See source_pipeline_broker.
 
source_pipeline_host
 
(required, string) Hostname or path to Unix socket that the bot will use to connect and receive messages.
 
source_pipeline_port
 
(optional, integer) Broker port that the bot will use to connect and receive messages. Can be empty for Unix socket.
 
source_pipeline_password
 
(optional, string) Broker password that the bot will use to connect and receive messages. Can be null for unprotected broker.
 
source_pipeline_db
 
(required, integer) broker database that the bot will use to connect and receive messages (requirement from   redis broker).
 
destination_pipeline_host
 
(optional, string) broker IP, FQDN or Unix socket that the bot will use to connect and send messages.
 
destination_pipeline_port
 
(optional, integer) broker port that the bot will use to connect and send messages. Can be empty for   Unix socket.
 
destination_pipeline_password
 
(optional, string) broker password that the bot will use to connect and send messages. Can be null   for unprotected broker.
 
destination_pipeline_db
 
(required, integer) broker database that the bot will use to connect and send messages (requirement from   redis broker).
 
http_proxy
 
(optional, string) Proxy to use for HTTP.
 
https_proxy
 
(optional, string) Proxy to use for HTTPS.
 
http_user_agent
 
(optional, string) User-Agent to be used for HTTP requests.
 
http_verify_cert
 
(optional, boolean) Verify the TLS certificate of the server. Defaults to true.
"},{"location":"admin/configuration/intelmq/#individual-bot-configuration","title":"Individual Bot Configuration","text":"
Info
 
For the individual bot configuration please see the Bots document in the User Guide.
"},{"location":"admin/configuration/intelmq/#run-mode","title":"Run Mode","text":"
This sections provides more detailed explanation of the two run modes of the bots.
"},{"location":"admin/configuration/intelmq/#continuous","title":"Continuous","text":"
Most of the cases, bots will need to be configured as continuous run mode (the default) in order to have them always running and processing events. Usually, the types of bots that will require the continuous mode will be Parsers, Experts and Outputs. To do this, set run_mode to continuous in the runtime.yaml for the bot. Check the following example:
 
blocklistde-apache-parser:\n  name: Blocklist.de Parser\n  group: Parser\n  module: intelmq.bots.parsers.blocklistde.parser\n  description: Blocklist.DE Parser is the bot responsible to parse the report and sanitize the information.\n  enabled: false\n  run_mode: continuous\n  parameters: ...\n
 
You can now start the bot using the following command:
 
intelmqctl start blocklistde-apache-parser\n
 
Bots configured as continuous will never exit except if there is an error and the error handling configuration requires the bot to exit. See the Error Handling section for more details.
"},{"location":"admin/configuration/intelmq/#scheduled","title":"Scheduled","text":"
In many cases, it is useful to schedule a bot at a specific time (i.e. via cron(1)), for example to collect information from a website every day at midnight. To do this, set run_mode to scheduled in the runtime.yaml for the bot. Check out the following example:
 
blocklistde-apache-collector:\n  name: Generic URL Fetcher\n  group: Collector\n  module: intelmq.bots.collectors.http.collector_http\n  description: All IP addresses which have been reported within the last 48 hours as having run attacks on the service Apache, Apache-DDOS, RFI-Attacks.\n  enabled: false\n  run_mode: scheduled\n  parameters:\n    feed: Blocklist.de Apache\n    provider: Blocklist.de\n    http_url: https://lists.blocklist.de/lists/apache.txt\n    ssl_client_certificate: null\n
 
You can schedule the bot with a crontab-entry like this:
 
0 0 * * * intelmqctl start blocklistde-apache-collector\n
 
Bots configured as scheduled will exit after the first successful run. Setting enabled to false will cause the bot to not start with intelmqctl start, but only with an explicit start, in this example intelmqctl start blocklistde-apache-collector.
"},{"location":"admin/configuration/intelmq/#additional-runtime-parameters","title":"Additional Runtime Parameters","text":"
Some of the parameters are deliberately skipped from the User Guide because they are configured via graphical user interface provided by the IntelMQ Manager. These parameters have to do with configuring the pipeline: defining how the data is exchanged between the bots. Using the IntelMQ Manager for this have many benefits as it guarantees that the configuration is correct upon saving.
 
However as an administrator you should be also familiar with the manual (and somewhat tedious) configuration. For each bot there are two parameters that need to be set:
 
source_queue
 
(optional, string) The name of the source queue from which the bot is going to processing data. Each bot has maximum one source queue (collector bots don't have any source queue as they fetch data from elsewhere). Defaults to the bot id appended with the string -queue.
 
Example: a bot with id example-bot will have a default source queue named example-bot-queue.
 
destination_queues
 
(optional, object) Bots can have multiple destination queues. Destination queues can also be grouped into named paths. There are two special path names _default and _on_error.  The path _default is used if the path is not is specified by the bot itself (which is the most common case). In case of an error during the processing, the message will be sent to the _on_error path if specified (optional).
 
Only few of the bots (mostly expert bots with filtering capabilities) can take advantage of arbitrarily named paths. Some expert bots are capable of sending messages to paths, this feature is explained in their documentation, e.g. the Filter expert and the Sieve expert.
 
Example:
 
blocklistde-apache-collector:\n  # ...\n  parameters:\n    # ...\n    destination_queues:\n      _default:\n        - <first destination pipeline name>\n        - <second destination pipeline name>\n      _on_error:\n        - <optional first destination pipeline name in case of errors>\n        - <optional second destination pipeline name in case of errors>\n      other-path:\n        - <second destination pipeline name>\n        - <third destination pipeline name>\n
"},{"location":"admin/configuration/intelmq/#harmonizationconf","title":"harmonization.conf","text":"
This configuration is used to specify the fields for all message types. The harmonization library will load this configuration to check, during the message processing, if the values are compliant to the configured harmonization format. Usually, this configuration doesn't need any change. It is mostly maintained by the IntelMQ maintainers.
 
Template:
 
{\n  \"<message type>\": {\n    \"<field 1>\": {\n      \"description\": \"<field 1 description>\",\n      \"type\": \"<field value type>\"\n    },\n    \"<field 2>\": {\n      \"description\": \"<field 2 description>\",\n      \"type\": \"<field value type>\"\n    }\n  }\n}\n
 
Example:
 
{\n  \"event\": {\n    \"destination.asn\": {\n      \"description\": \"The autonomous system number from which originated the connection.\",\n      \"type\": \"Integer\"\n    },\n    \"destination.geolocation.cc\": {\n      \"description\": \"Country-Code according to ISO3166-1 alpha-2 for the destination IP.\",\n      \"regex\": \"^[a-zA-Z0-9]{2}$\",\n      \"type\": \"String\"\n    }\n  }\n}\n
"},{"location":"admin/database/elasticsearch/","title":"Elasticsearch","text":""},{"location":"admin/database/elasticsearch/#using-elasticsearch-as-a-database-for-intelmq","title":"Using Elasticsearch as a database for IntelMQ","text":"
If you wish to run IntelMQ with Elasticsearch or full ELK stack (Elasticsearch, Logstash, Kibana) it is entirely possible. This guide assumes the reader is familiar with basic configuration of ELK and does not aim to cover using ELK in general. It is based on the version 6.8.0 (ELK is a fast moving train therefore things might change). Assuming you have IntelMQ (and Redis) installation in place, lets dive in.
"},{"location":"admin/database/elasticsearch/#configuration-without-logstash","title":"Configuration without Logstash","text":"
This case involves two steps:
 
     
  1.  
    Configure IntelMQ to output data directly into Elasticsearch.
     
    2.  
  2.  
    Configure Elasticsearch for ingesting the inserted data.
     
    3.  
 
Bug
 
This section of the documentation is currently incomplete and will be updated later.
"},{"location":"admin/database/elasticsearch/#configuration-with-logstash","title":"Configuration with Logstash","text":"
This case involves three steps:
 
     
  1.  
    Configuring IntelMQ to output data to Redis.
     
    2.  
  2.  
    Configure Logstash to collect data from Redis and insert them into Elasticsearch.
     
    3.  
  3.  
    Configure Elasticsearch for ingesting the inserted data.
     
    4.  
 
Each step is described in detail in the following sections.
"},{"location":"admin/database/elasticsearch/#configuring-intelmq","title":"Configuring IntelMQ","text":"
In order to pass IntelMQ events to Logstash we will utilize already installed Redis. Add a new Redis Output Bot to your pipeline. As the minimum fill in the following parameters: bot-id, redis_server_ip (can be hostname) , redis_server_port, redis_password (if required, else set to empty!), redis_queue (name for the queue). It is recommended to use a different redis_db parameter than used by the IntelMQ (specified as source_pipeline_db , destination_pipeline_db and statistics_database).
 
Example values:
 
bot-id: redis-output\nredis_server_ip: 10.10.10.10\nredis_server_port: 6379\nredis_db: 4\nredis_queue: logstash-queue\n
 
Warning
 
You will not be able to monitor this redis queue via IntelMQ Manager.
"},{"location":"admin/database/elasticsearch/#configuring-logstash","title":"Configuring Logstash","text":"
Logstash defines pipelines as well. In the pipeline configuration of Logstash you need to specify where it should look for IntelMQ events, what to do with them and where to pass them.
"},{"location":"admin/database/elasticsearch/#input","title":"Input","text":"
This part describes how to receive data from Redis queue. See the example configuration and comments below:
 
input {\n  redis {\n    host => \"10.10.10.10\"\n    port => 6379\n    db => 4\n    data_type => \"list\"\n    key => \"logstash-queue\"\n  }\n}\n
 
     
  • host - same as redis_server_ip from the Redis Output Bot
    •  
  • port - the redis_server_port from the Redis Output Bot
    •  
  • db - the redis_db parameter from the Redis Output Bot
    •  
  • data_type - set to list
    •  
  • key - same as redis_queue from the Redis Output Bot
    •  
 
Tip
 
You can use environment variables for the Logstash configuration, for example host => \"${REDIS_HOST:10.10.10.10}\". The value will be taken from the environment variable $REDIS_HOST. If the environment variable is not set then the default value of 10.10.10.10 will be used instead.
"},{"location":"admin/database/elasticsearch/#filter-optional","title":"Filter (optional)","text":"
Before passing the data to the database you can apply certain changes. This is done with filters. See an example:
 
filter {\n  mutate {\n    lowercase => [\"source.geolocation.city\", \"classification.identifier\"]\n    remove_field => [\"__type\", \"@version\"]\n  }\n  date {\n    match => [\"time.observation\", \"ISO8601\"]\n  }\n}\n
 
Tip
 
It is recommended to use the date filter: generally we have two timestamp fields - time.source (provided by the feed source this can be understood as when the event happened; however it is not always present) and time.observation (when IntelMQ collected this event). Logstash also adds another field @timestamp with time of processing by Logstash. While it can be useful for debugging, I recommend to set the @timestamp to the same value as time.observation.
 
Warning
 
It is not recommended to apply any modifications to the data (within the mutate key) outside of the IntelMQ. All necessary modifications should be done only by appropriate IntelMQ bots. This example only demonstrates the possibility.
"},{"location":"admin/database/elasticsearch/#output","title":"Output","text":"
The pipeline also needs output, where we define our database (Elasticsearch). The simplest way of doing so is defining an output like this:
 
output {\n  elasticsearch {\n    hosts => [\"http://10.10.10.11:9200\", \"http://10.10.10.12:9200\"]\n    index => \"intelmq-%{+YYYY.MM}\"\n  }\n}\n
 
     
  • hosts - Elasticsearch host (or more) with the correct port (9200 by default)
    •  
  • index - name of the index where to insert data
    •  
 
Tip
 
Authors experience, hardware equipment and the amount of events collected led to having a separate index for each month. This might not necessarily suit your needs, but it is a suggested option.
 
Warning
 
By default the ELK stack uses insecure HTTP. It is possible to setup Security for secure connections and basic user management. This is possible with the Basic (free) licence since versions 6.8.0 and 7.1.0.
"},{"location":"admin/database/elasticsearch/#configuring-elasticsearch","title":"Configuring Elasticsearch","text":"
Configuring Elasticsearch is entirely up to you and should be consulted with the official documentation. What you will most likely need is something called index template mappings. IntelMQ provides a tool for generating such mappings. See ElasticMapper Tool.
 
Danger
 
Default installation of Elasticsearch database allows anyone with cURL and connection capability to have administrative access to the database. Make sure you secure your toys!
"},{"location":"admin/database/mssql/","title":"MSSQL","text":""},{"location":"admin/database/mssql/#mssql","title":"MSSQL","text":"
For MSSQL support, the library pymssql>=2.2 is required.
 
To output data to MSSQL use SQL Output Bot with parameter engine set to mssql.
 
For more information see SQL Output Bot documentation page.
"},{"location":"admin/database/postgresql/","title":"PostgreSQL","text":""},{"location":"admin/database/postgresql/#using-postgresql-as-a-database-for-intelmq","title":"Using PostgreSQL as a database for IntelMQ","text":"
The EventDB is a database (usually PostgreSQL) that gets filled with with data from IntelMQ using the SQL Output Bot.
"},{"location":"admin/database/postgresql/#intelmq_psql_initdb","title":"intelmq_psql_initdb","text":"
IntelMQ comes with the intelmq_psql_initdb command line tool designed to help with creating the EventDB. It creates in the first line:
 
     
  • A CREATE TABLE events statement with all valid IntelMQ fields as columns and correct types
    •  
  • Several indexes as examples for a good read & search performance
    •  
 
Having an events table as outlined in the SQL file, IntelMQ's SQL Output Bot can write all received events into this database table.
 
In addition, the script supports some additional features supporting use cases described later in this document:
 
     
  • --partition-key - for generating schema aligned with TimescaleDB or partitioned tables,
    •  
  • --separate-raws - for generating views and triggers needed to eventdb_raws_table (works also together with adjustments for partitioning).
    •  
 
For a full list of supported parameters, call the script help using -h parameter.
 
All elements of the generated SQL file can be adapted and extended before running the SQL file against a database, especially the indexes. Please review the generated script before applying.
 
Be aware that if you create tables using another DB user that is used later by the output bot, you may need to adjust ownership or privileges in the database. If you have problems with database permissions, refer to PostgreSQL documentation <https://www.postgresql.org/docs/current/ddl-priv.html>.
"},{"location":"admin/database/postgresql/#eventdb-utilities","title":"EventDB Utilities","text":"
Some scripts related to the EventDB are located in the contrib/eventdb folder in the IntelMQ git repository.
"},{"location":"admin/database/postgresql/#apply-malware-name-mapping","title":"Apply Malware Name Mapping","text":"
The apply_mapping_eventdb.py script applies the malware name mapping to the EventDB. Source and destination columns can be given, also a local file. If no local file is present, the mapping can be downloaded on demand. It queries the database for all distinct malware names with the taxonomy \"malicious-code\" and sets another column to the malware family name.
"},{"location":"admin/database/postgresql/#apply-domain-suffix","title":"Apply Domain Suffix","text":"
The apply_domain_suffix.py script writes the public domain suffix to the source.domain_suffix / destination.domain_suffix columns, extracted from source.fqdn / destination.fqdn.
"},{"location":"admin/database/postgresql/#usage","title":"Usage","text":"
The Python scripts can connect to a PostgreSQL server with an eventdb database and an events table. The command line arguments interface for both scripts are the same. See --help for more information:
 
apply_mapping_eventdb.py -h\napply_domain_suffix.py -h\n
"},{"location":"admin/database/postgresql/#postgresql-trigger","title":"PostgreSQL trigger","text":"
PostgreSQL trigger is a trigger keeping track of the oldest inserted/updated \"time.source\" data. This can be useful to (re-)generate statistics or aggregation data.
 
The SQL script can be executed in the database directly.
"},{"location":"admin/database/postgresql/#eventdb-statistics","title":"EventDB Statistics","text":"
The EventDB provides a great base for statistical analysis of the data.
 
The eventdb-stats repository contains a Python script that generates an HTML file and includes the Plotly JavaScript Open Source Graphing Library. By modifying the configuration file it is possible to configure various queries that are then displayed using graphs:
 
"},{"location":"admin/database/postgresql/#using-eventdb-with-timescale-db","title":"Using EventDB with Timescale DB","text":"
Timescale DB is a PostgreSQL extension to add time-series support, which is quite handy as you don't have to learn other syntaxes as you already know. You can use the SQL Queries as before, the extension will handle the rest. To see all limitations, please check the Timescale DB Documentation.
"},{"location":"admin/database/postgresql/#what-is-time-series","title":"What is time-series?","text":"
Time-series has been invented as traditional database design like relational or nosql are not made for time-based data. A big benefit of time-series instead of other database designs over a time-based search pattern is the performance. As IntelMQ uses data based upon time, this design is awesome & will give you a performance boost.
"},{"location":"admin/database/postgresql/#how-to-choose-the-time-column","title":"How to choose the time column?","text":"
To utilize the time-series, choose a column containing the right time. This is then used by you for manual queries and graphs, and also by the database itself for organizing the data.
 
An Event has two fields that can be used for this: time.source or time.observation. Depending on your needs (tracking when the event occurred or when it was detected, if different), choose one of them.
 
You can use the :ref:intelmq_psql_initdb tool to generate SQL schema valid for TimescaleDB by passing the partitioning key:
 
intelmq_psql_initdb --partition-key \"time.source\"\n
"},{"location":"admin/database/postgresql/#how-to-setup","title":"How to setup","text":"
Thanks to TimescaleDB its very easy to setup.
 
     
  1. Choose your preferred Timescale DB environment & follow the installation instructions. 2. Now lets create a hypertable, which is the timescale DB time-series structure. SELECT create_hypertable('', 'time.source');. 3. Now our hypertable is setup & timescaleDB takes care of the rest. You can perform queries as usual, for further information please check Timescale DB Documentation.
    2.  
"},{"location":"admin/database/postgresql/#how-to-upgrade-from-my-existing-database","title":"How to upgrade from my existing database?","text":"
To update your existing database to use this awesome time-series feature, just follow the How to setup instruction. You can perform the hypertable command even on already existing databases. BUT there are some limitations from timescaleDB.
"},{"location":"admin/database/postgresql/#separating-raw-values-in-postgresql-using-view-and-trigger","title":"Separating raw values in PostgreSQL using view and trigger","text":"
In order to reduce the row size in the events table, the raw column's data can be separated from the other columns. While the raw-data is about 30-50% of the data row's size, it is not used in most database queries, as it serves only a backup functionality. Other possibilities to reduce or getting rid of this field are described in the FAQ, section faq-remove-raw-data.
 
The steps described here are best performed before the events table is filled with data, but can as well be done with existing data.
 
The approach requires four steps:
 
     
  1. An existing events table, see the first section of     this document.
    2.  
  2. Deleting or renaming the raw column of the     events table.
    3.  
  3. Creating a table raws which holds only the     raw field of the events and linking both tables using     the event_id.
    4.  
  4. Creating the view v_events which joins the tables     events and raws.
    5.  
  5. Creating the function process_v_events_insert and     INSERT trigger tr_events.
    6.  
 
The last steps brings us several advantages:
 
     
  • All INSERT statements can contain all data, including     the raw field.
    •  
  • No code changes are needed in the IntelMQ output bot or your own     scripts. A migration is seamless.
    •  
  • PostgreSQL itself ensures that the data of both tables is consistent     and linked correctly.
    •  
 
The complete SQL script can be generated using the intelmq_psql_initdb. It does not cover step 2 to avoid accidental data loss - you need to do this step manually.
"},{"location":"admin/database/postgresql/#other-docs","title":"Other docs","text":"
You have two basic choices to run PostgreSQL:
 
     
  1. on the same machine as intelmq, then you could use Unix sockets if available on your platform
    2.  
  2. on a different machine. In which case you would need to use a TCP connection and make sure you give the right    connection parameters to each psql or client call.
    3.  
 
Make sure to consult your PostgreSQL documentation about how to allow network connections and authentication in case 2.
 
PostgreSQL Version
 
Any supported version of PostgreSQL should work (v>=9.2 as of Oct 2016) [1].
 
If you use PostgreSQL server v >= 9.4, it gives you the possibility to use the time-zone formatting string \"OF\" for date-times and the GiST index for the CIDR type. This may be useful depending on how you plan to use the events that this bot writes into the database.
 
How to install
 
Use intelmq_psql_initdb to create initial SQL statements from harmonization.conf. The script will create the required table layout and save it as /tmp/initdb.sql
 
You need a PostgreSQL database-user to own the result database. The recommendation is to use the name intelmq . There may already be such a user for the PostgreSQL database-cluster to be used by other bots. (For example from setting up the expert/certbund_contact bot.)
 
Therefore if still necessary: create the database-user as postgresql superuser, which usually is done via the system user postgres:
 
createuser --no-superuser --no-createrole --no-createdb --encrypted --pwprompt intelmq\n
 
Create the new database:
 
createdb --encoding='utf-8' --owner=intelmq intelmq-events\n
 
(The encoding parameter should ensure the right encoding on platform where this is not the default.)
 
Now initialize it as database-user intelmq (in this example a network connection to localhost is used, so you would get to test if the user intelmq can authenticate):
 
psql -h localhost intelmq-events intelmq </tmp/initdb.sql\n
 
PostgreSQL and null characters
 
While null characters (0, not SQL \"NULL\") in TEXT and JSON/JSONB fields are valid, data containing null characters can cause troubles in some combinations of clients, servers and each settings. To prevent unhandled errors and data which can't be inserted into the database, all null characters are escaped (u0000) before insertion.
"},{"location":"admin/database/splunk/","title":"Splunk","text":""},{"location":"admin/database/splunk/#sending-intelmq-events-to-splunk","title":"Sending IntelMQ events to Splunk","text":"
     
  1. Go to Splunk and configure in order to be able to receive     logs (intelmq events) to a TCP port
    2.  
  2. Use TCP output bot and configure accordingly to the Splunk     configuration that you applied.
    3.  
"},{"location":"admin/database/sqlite/","title":"SQLite","text":""},{"location":"admin/database/sqlite/#sqlite","title":"SQLite","text":"
Similarly to PostgreSQL, you can use intelmq_psql_initdb to create initial SQL statements from harmonization.conf. The script will create the required table layout and save it as /tmp/initdb.sql.
 
Create the new database (you can ignore all errors since SQLite doesn't know all SQL features generated for PostgreSQL):
 
sqlite3 your-db.db\nsqlite> .read /tmp/initdb.sql\n
 
Then, set the database parameter to the your-db.db file path.
 
To output data to SQLite use SQL Output Bot with parameter engine set to sqlite. For more information see SQL Output Bot documentation page.
"},{"location":"admin/installation/dockerhub/","title":"DockerHub","text":""},{"location":"admin/installation/dockerhub/#installation-from-dockerhub","title":"Installation from DockerHub","text":"
This guide provides instruction on how to install IntelMQ and it's components using Docker.
 
Warning
 
Docker installation is currently in Beta state and things might break. Consider this if you plan to use IntelMQ as a production level system.
 
Warning
 
Currently you can't manage your botnet via intelmqctl command line tool. You need to use IntelMQ-Manager currently!
 
The latest IntelMQ image is hosted on Docker Hub and the image build instructions are in our intelmq-docker repository.
 
Follow Docker Install and Docker-Compose Install instructions.
 
Before you start using docker-compose or any docker related tools, make sure docker is running:
 
# To start the docker daemon\nsystemctl start docker.service\n# To enable the docker daemon for the future\nsystemctl enable docker.service\n
"},{"location":"admin/installation/dockerhub/#docker-with-docker-compose","title":"Docker with docker-compose","text":"
Now we can download IntelMQ and start the containers. Navigate to your preferred installation directory and run the following commands:
 
git clone https://github.com/certat/intelmq-docker.git --recursive\ncd intelmq-docker\nsudo docker-compose pull\nsudo docker-compose up\n
 
Your installation should be successful now. You're now able to visit http://127.0.0.1:1337/ to access the intelmq-manager. You have to login with the username intelmq and the password intelmq, if you want to change the username or password, you can do this by adding the environment variables INTELMQ_API_USER for the username and INTELMQ_API_PASS for the password.
 
Note
 
If you get an Permission denied error, you should run chown -R $USER:$USER example_config
"},{"location":"admin/installation/dockerhub/#docker-without-docker-compose","title":"Docker without docker-compose","text":"
If not already installed, please install Docker.
 
Navigate to your preferred installation directory and run git clone https://github.com/certat/intelmq-docker.git --recursive.
 
You need to prepare some volumes & configs. Edit the left-side after -v, to change paths.
 
Change redis_host to a running redis-instance. Docker will resolve it automatically. All containers are connected using Docker Networks.
 
In order to work with your current infrastructure, you need to specify some environment variables
 
sudo docker pull redis:latest\n\nsudo docker pull certat/intelmq-full:latest\n\nsudo docker pull certat/intelmq-nginx:latest\n\nsudo docker network create intelmq-internal\n\nsudo docker run -v ~/intelmq/example_config/redis/redis.conf:/redis.conf \\\n                --network intelmq-internal \\\n                --name redis \\\n                redis:latest\n\nsudo docker run --network intelmq-internal \\\n                --name nginx \\\n                certat/intelmq-nginx:latest\n\nsudo docker run -e INTELMQ_IS_DOCKER=\"true\" \\\n                -e INTELMQ_SOURCE_PIPELINE_BROKER: \"redis\" \\\n                -e INTELMQ_PIPELINE_BROKER: \"redis\" \\\n                -e INTELMQ_DESTIONATION_PIPELINE_BROKER: \"redis\" \\\n                -e INTELMQ_PIPELINE_HOST: redis \\\n                -e INTELMQ_SOURCE_PIPELINE_HOST: redis \\\n                -e INTELMQ_DESTINATION_PIPELINE_HOST: redis \\\n                -e INTELMQ_REDIS_CACHE_HOST: redis \\\n                -v $(pwd)/example_config/intelmq/etc/:/etc/intelmq/etc/ \\\n                -v $(pwd)/example_config/intelmq-api/config.json:/etc/intelmq/api-config.json \\\n                -v $(pwd)/intelmq_logs:/etc/intelmq/var/log \\\n                -v $(pwd)/intelmq_output:/etc/intelmq/var/lib/bots \\\n                -v ~/intelmq/lib:/etc/intelmq/var/lib \\\n                --network intelmq-internal \\\n                --name intelmq \\\n                certat/intelmq-full:latest\n
 
If you want to use another username and password for the intelmq-manager / api login, additionally add two new environment variables.
 
-e INTELMQ_API_USER: \"your username\"\n-e INTELMQ_API_PASS: \"your password\"\n
"},{"location":"admin/installation/linux-packages/","title":"Linux Package","text":""},{"location":"admin/installation/linux-packages/#installation-as-linux-package","title":"Installation as Linux package","text":"
This guide provides instructions on how to install IntelMQ and it's components from Linux distribution's package repository.
 
Note
 
Some bots may have additional dependencies which are mentioned in their own documentation.
"},{"location":"admin/installation/linux-packages/#supported-os","title":"Supported OS","text":"
Native packages are currently provided for the following Linux distributions:
 
     
  • Debian 11 (bullseye)
    •  
  • Debian 12 (bookworm)
    •  
  • openSUSE Tumbleweed
    •  
  • Ubuntu 20.04 (focal fossa)
    •  
  • Ubuntu 22.04 (jammy jellyfish)
    •  
"},{"location":"admin/installation/linux-packages/#debian-11-and-12","title":"Debian 11 and 12","text":"
Add the repository to the package manager and install IntelMQ (packages intelmq-api and intelmq-manager are optional):
 
echo \"deb http://download.opensuse.org/repositories/home:/sebix:/intelmq/Debian_$(lsb_release -rs)/ /\" | sudo tee /etc/apt/sources.list.d/intelmq.list\ncurl -fsSL \"https://download.opensuse.org/repositories/home:sebix:intelmq/Debian_$(lsb_release -rs)/Release.key\" | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/intelmq.gpg > /dev/null\nsudo apt update\nsudo apt install intelmq intelmq-api intelmq-manager\n
"},{"location":"admin/installation/linux-packages/#opensuse-tumbleweed","title":"openSUSE Tumbleweed","text":"
Add the repository to the package manager and install IntelMQ (packages intelmq-api and intelmq-manager are optional):
 
zypper addrepo https://download.opensuse.org/repositories/home:sebix:intelmq/openSUSE_Tumbleweed/home:sebix:intelmq.repo\nzypper refresh\nzypper install intelmq intelmq-api intelmq-manager\n
"},{"location":"admin/installation/linux-packages/#ubuntu-2004-and-2204","title":"Ubuntu 20.04 and 22.04","text":"
For Ubuntu you must enable the Universe repository which provides community-maintained free and open-source software.
 
Add the repository to the package manager and install IntelMQ (packages intelmq-api and intelmq-manager are optional):
 
     
  1.  
    Open the file /etc/apt/sources.list in an editor of your choice. Use sudo or the root user.
     
    2.  
  2.  
    Append universe to this line: 
    deb http://[...].archive.ubuntu.com/ubuntu/ focal main universe\n
     
    3.  
  3.  
    Next, add the IntelMQ APT Repository for Ubuntu: 
    echo \"deb http://download.opensuse.org/repositories/home:/sebix:/intelmq/xUbuntu_$(lsb_release -rs)/ /\" | sudo tee /etc/apt/sources.list.d/intelmq.list\ncurl -fsSL \"https://download.opensuse.org/repositories/home:sebix:intelmq/xUbuntu_$(lsb_release -rs)/Release.key\" | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/intelmq.gpg > /dev/null\n
     
    4.  
  4.  
    Now update the list of available packages and install the IntelMQ packages: 
    sudo apt update\nsudo apt install intelmq intelmq-api intelmq-manager\n
     
    5.  
"},{"location":"admin/installation/pypi/","title":"PyPI","text":""},{"location":"admin/installation/pypi/#installation-from-pypi","title":"Installation from PyPI","text":"
This guide provides instruction on how to install IntelMQ and it's components using the Python Package Index (PyPI) repository.
 
Note
 
Some bots may have additional dependencies which are mentioned in their own documentation.
"},{"location":"admin/installation/pypi/#installing-intelmq","title":"Installing IntelMQ","text":""},{"location":"admin/installation/pypi/#requirements","title":"Requirements","text":""},{"location":"admin/installation/pypi/#ubuntu-debian","title":"Ubuntu / Debian","text":"
apt install python3-pip python3-dnspython python3-psutil python3-redis python3-requests python3-termstyle python3-tz python3-dateutil redis-server bash-completion jq\n# optional dependencies\napt install python3-pymongo python3-psycopg2\n
"},{"location":"admin/installation/pypi/#opensuse","title":"openSUSE:","text":"
zypper install python3-dateutil python3-dnspython python3-psutil python3-redis python3-requests python3-python-termstyle redis bash-completion jq\n# optional dependencies\nzypper in python3-psycopg2 python3-pymongo\n
"},{"location":"admin/installation/pypi/#centos-8","title":"CentOS 8:","text":"
dnf install epel-release\ndnf install python3-dateutil python3-dns python3-pip python3-psutil python3-redis python3-requests redis bash-completion jq\n# optional dependencies\ndnf install python3-psycopg2 python3-pymongo\n
"},{"location":"admin/installation/pypi/#centos-7-rhel-7","title":"CentOS 7 / RHEL 7:","text":"
Warning
 
We no longer support already end-of-life Python 3.6, which is the last Python version officially packaged for CentOS 7. You can either use alternative Python source, or stay on the IntelMQ 3.0.2.
 
yum install epel-release\nyum install python36 python36-dns python36-requests python3-setuptools redis bash-completion jq\nyum install gcc gcc-c++ python36-devel\n# optional dependencies\nyum install python3-psycopg2\n
"},{"location":"admin/installation/pypi/#installation","title":"Installation","text":"
The default installation directory is /opt/intelmq/.
 
If you prefer to use Linux Standard Base (LSB) paths, set the following environment variable:
 
export INTELMQ_PATHS_NO_OPT=1\n
 
If you want to use custom installation directory, set the following environment variable:
 
export INTELMQ_ROOT_DIR=/my-installation-directory-path\n
 
Run the following commands to install IntelMQ. The provided tool intelmqsetup will create all the necessary directories and installs a default configuration for new setups. If you are using the LSB paths installation, change the --home-dir parameter to /var/lib/intelmq
 
sudo --preserve-env=INTELMQ_PATHS_NO_OPT,INTELMQ_ROOT_DIR -i\npip3 install intelmq\n[[ ! -z \"$INTELMQ_PATHS_NO_OPT\" ]] && export HOME_DIR=/var/lib/intelmq || export HOME_DIR=${INTELMQ_ROOT_DIR:-/opt/intelmq}\nuseradd --system --user-group --home-dir $HOME_DIR --shell /bin/bash intelmq\nintelmqsetup\n
"},{"location":"admin/installation/pypi/#installation-to-python-virtual-environment","title":"Installation to Python virtual environment","text":"
sudo mkdir -m 755 /opt/intelmq\nsudo useradd --system --user-group --home-dir /opt/intelmq --shell /bin/bash intelmq\nsudo chown intelmq:intelmq /opt/intelmq/\nsudo -u intelmq python3 -m venv /opt/intelmq/venv\nsudo -u intelmq /opt/intelmq/venv/bin/pip install intelmq intelmq-api intelmq-manager\nsudo /opt/intelmq/venv/bin/intelmqsetup\n
"},{"location":"admin/installation/pypi/#installing-intelmq-api-optional","title":"Installing IntelMQ API (optional)","text":"
The intelmq-api packages ships:
 
     
  • api configuration file in ${PREFIX}/etc/intelmq/api-config.json
    •  
  • positions configuration for the intelmq-manager in {PREFIX}/etc/intelmq/manager/positions.conf
    •  
  • virtualhost configuration file for Apache 2 in ${PREFIX}/etc/intelmq/api-apache.conf
    •  
  • sudoers configuration file in ${PREFIX}/etc/intelmq/api-sudoers.conf
    •  
 
The value of ${PREFIX} depends on your environment and is something like /usr/local/lib/pythonX.Y/dist-packages/ (where X.Y is your Python version).
 
The virtualhost configuration file needs to be placed in the correct directory for your Apache 2 installation.
 
     
  • On Debian or Ubuntu, move the file to /etc/apache2/conf-available.d/ directory and then execute a2enconf api-apache.
    •  
  • On CentOS, RHEL or Fedora, move the file to /etc/httpd/conf.d/ directory.
    •  
  • On openSUSE, move the file to /etc/apache2/conf.d/ directory.
    •  
 
Don't forget to reload your webserver afterwards.
 
The api configuration file and the positions configuration file need to be placed in one of the following directories (based on your IntelMQ installation directory):
 
     
  • /etc/intelmq/
    •  
  • /opt/intelmq/etc/
    •  
  • [my-installation-directory-path]/etc/
    •  
 
The sudoers configuration file should be placed in the /etc/sudoers.d/ directory and adapt the webserver username in this file. Set the file permissions to 0o440.
 
Afterwards continue with the section Permissions below.
 
IntelMQ 2.3.1 comes with a tool intelmqsetup which performs these set-up steps automatically. Please note that the tool is very new and may not detect all situations correctly. Please report us any bugs you are observing. The tools is idempotent, you can execute it multiple times.
"},{"location":"admin/installation/pypi/#installing-intelmq-manager-optional","title":"Installing IntelMQ Manager (optional)","text":"
To use the IntelMQ Manager web interface, it is required to have a working IntelMQ and IntelMQ API installation.
 
For installation via pip, the situation is more complex. The intelmq-manager package does not contain ready-to-use files, they need to be built locally. First, lets install the Manager itself:
 
pip3 install intelmq-manager\n
 
If your system uses wheel-packages, not the source distribution, you can use the intelmqsetup tool. intelmqsetup which performs these set-up steps automatically but it may not detect all situations correctly. If it finds intelmq-manager installed, calls its build routine is called. The files are placed in /usr/share/intelmq_manager/html, where the default Apache configuration expect it.
 
If your system used the dist-package or if you are using a local source, the tool may not do all required steps. To call the build routine manually, use intelmq-manager-build --output-dir your/preferred/output/directory/.
 
intelmq-manager ships with a default configuration for the Apache webserver (manager-apache.conf):
 
Alias /intelmq-manager /usr/share/intelmq_manager/html/\n\n<Directory /usr/share/intelmq_manager/html>\n    <IfModule mod_headers.c>\n    Header set Content-Security-Policy \"script-src 'self'\"\n    Header set X-Content-Security-Policy \"script-src 'self'\"\n    </IfModule>\n</Directory>\n
 
This file needs to be placed in the correct place for your Apache 2 installation.
 
     
  • On Debian and Ubuntu, the file needs to be placed at /etc/apache2/conf-available.d/manager-apache.conf and then execute   a2enconf manager-apache.
    •  
  • On CentOS, RHEL and Fedora, the file needs to be placed at /etc/httpd/conf.d/ and reload the webserver.
    •  
  • On openSUSE, the file needs to be placed at /etc/apache2/conf.d/ and reload the webserver.
    •  
"},{"location":"admin/integrations/cifv3/","title":"CIFv3","text":""},{"location":"admin/integrations/cifv3/#cifv3-integrations-in-intelmq","title":"CIFv3 integrations in IntelMQ","text":"
CIF creates an accessible indicator store. A REST API is exposed to interact with the store and quickly process/share indicators. CIFv3 can correlate indicators via the UUID attribute.
"},{"location":"admin/integrations/cifv3/#cif3-api-output","title":"CIF3 API Output","text":"
Can be used to submit indicators to a CIFv3 instance by using the CIFv3 API.
 
Look at the CIFv3 API Output Bot for more information.
"},{"location":"admin/integrations/misp/","title":"MISP","text":""},{"location":"admin/integrations/misp/#misp-integrations-in-intelmq","title":"MISP integrations in IntelMQ","text":"
While MISP and IntelMQ seem to solve similar problems in the first hindsight, their intentions and strengths differ significantly.
 
In a nutshell, MISP stores manually curated indicators (called attributes) grouped in events. An event can have an arbitrary number of attributes. MISP correlates these indicators with each other and can synchronize the data between multiple MISP instances.
 
On the other side, IntelMQ in it's essence (not considering the EventDB <eventdb>) has no state or database, but is stream-oriented. IntelMQ acts as a toolbox which can be configured as needed to automate processes of mass data with little or no human interaction At the end of the processing the data may land in some database or be sent to other systems.
 
Both systems do not intend to replace each other or do compete. They integrate seamless and combine each other enabling more use-cases and
"},{"location":"admin/integrations/misp/#misp-api-collector","title":"MISP API Collector","text":"
The MISP API Collector fetches data from MISP via the MISP API .
 
Look at the Bots documentation page for more information.
"},{"location":"admin/integrations/misp/#misp-expert","title":"MISP Expert","text":"
The MISP Expert searches MISP by using the MISP API for attributes/events matching the source.ip of the event. The MISP Attribute UUID and MISP Event ID of the newest attribute are added to the event.
 
Look at the Bots documentation page for more information.
"},{"location":"admin/integrations/misp/#misp-feed-output","title":"MISP Feed Output","text":"
This bot creates a complete MISP feed ready to be configured in MISP as incoming data source.
 
Look at the Bots documentation page for more information.
"},{"location":"admin/integrations/misp/#misp-api-output","title":"MISP API Output","text":"
Can be used to directly create MISP events in a MISP instance by using the MISP API.
 
Look at the Bots documentation page for more information.
"},{"location":"admin/integrations/n6/","title":"N6","text":""},{"location":"admin/integrations/n6/#intelmq-n6-integration","title":"IntelMQ - n6 Integration","text":"
n6 is an Open Source Tool with very similar aims as IntelMQ: processing and distributing IoC data. The use-cases, architecture and features differ and both tools have non-overlapping strengths. n6 is maintained and developed by CERT.pl.
 
Information about n6 can be found here:
 
     
  • Website: cert.pl/en/n6
    •  
  • Source Code:   github.com/CERT-Polska/n6
    •  
  • n6 documentation: n6.readthedocs.io
    •  
 
 
"},{"location":"admin/integrations/n6/#data-format","title":"Data format","text":"
The internal data representation differs between IntelMQ and n6, so any data exchange between the systems requires a format conversion. For example, in n6 one message can contain multiple IP addresses, but IntelMQ is intentionally restricted to one IP address per message. Therefore, one n6 event results in one or more IntelMQ events. Because of this, and some other naming differences and ambiguities, the format conversion is not bidirectional.
"},{"location":"admin/integrations/n6/#data-exchange-interface","title":"Data exchange interface","text":"
n6 offers a STOMP interface via the RabbitMQ broker, which can be used for both sending and receiving data. IntelMQ offers both a STOMP collector bot for receiving data from n6, as well as a STOMP output bot for sending data to n6 instances.
 
     
  • Stomp Collector Bot
    •  
  • N6 Parser Bot
    •  
  • Stomp Output Bot
    •  
"},{"location":"admin/integrations/n6/#data-conversion","title":"Data conversion","text":"
IntelMQ can parse n6 data using the n6 parser and n6 can parse IntelMQ data using the Intelmq n6 parser.
 
     
  • N6 Parser Bot
    •  
"},{"location":"admin/integrations/n6/#complete-example","title":"Complete example","text":""},{"location":"admin/integrations/n6/#data-flow-n6-to-intelmq","title":"Data flow n6 to IntelMQ","text":""},{"location":"admin/integrations/n6/#data-flow-intelmq-to-n6","title":"Data flow IntelMQ to n6","text":""},{"location":"admin/integrations/n6/#certpl-data-feed","title":"CERT.pl Data feed","text":"
CERT.pl offers data feed available to their partners through the STOMP interface. Our feeds documentation contains details how it can be enabled in IntelMQ: CERT.pl n6 STOMP stream
"},{"location":"admin/integrations/n6/#webinput-csv","title":"Webinput CSV","text":"
The IntelMQ Webinput CSV software can also be used together with n6. The documentation on this component can be found in the software's repository: https://github.com/certat/intelmq-webinput-csv/blob/master/docs/webinput-n6.md
"},{"location":"admin/management/intelmq-api/","title":"IntelMQ API","text":""},{"location":"admin/management/intelmq-api/#managing-intelmq-api","title":"Managing IntelMQ API","text":""},{"location":"admin/management/intelmq-api/#running","title":"Running","text":"
For development purposes and testing you can run directly using hug:
 
hug -m intelmq_api.serve\n
"},{"location":"admin/management/intelmq/","title":"IntelMQ","text":""},{"location":"admin/management/intelmq/#managing-intelmq","title":"Managing IntelMQ","text":""},{"location":"admin/management/intelmq/#required-services","title":"Required services","text":"
You need to enable and start Redis if not already done. Using systemd it can be done with:
 
systemctl enable redis.service\nsystemctl start redis.service\n
"},{"location":"admin/management/intelmq/#introduction","title":"Introduction","text":"
intelmqctl is the main tool to handle a intelmq installation. It handles the bots themselves and has some tools to handle the installation.
 
Should you get lost any time, just use the --help after any argument for further explanation.
 
> intelmqctl run file-output --help\n
"},{"location":"admin/management/intelmq/#manage-the-botnet","title":"Manage the botnet","text":"
In IntelMQ, the botnet is the set of all currently configured and enabled bots. All configured bots have their configuration in runtime.yaml. By default, all bots are enabled.
 
If no bot id is given, the command applies to all bots / the botnet. All commands except the start action are applied to all bots. But only enabled bots are started.
 
In the examples below, a very minimal botnet is used.
"},{"location":"admin/management/intelmq/#start","title":"start","text":"
The start action applies to all bots which are enabled.
 
> intelmqctl start\nStarting abusech-domain-parser...\nabusech-domain-parser is running.\nStarting abusech-feodo-domains-collector...\nabusech-feodo-domains-collector is running.\nStarting deduplicator-expert...\ndeduplicator-expert is running.\nfile-output is disabled.\nBotnet is running.\n
 
As we can file-output is disabled and thus has not been started. You can always explicitly start disabled bots.
"},{"location":"admin/management/intelmq/#stop","title":"stop","text":"
The stop action applies to all bots. Assume that all bots have been running:
 
> intelmqctl stop\nStopping Botnet...\nStopping abusech-domain-parser...\nabusech-domain-parser is stopped.\nStopping abusech-feodo-domains-collector...\nabusech-feodo-domains-collector is stopped.\nStopping deduplicator-expert...\ndeduplicator-expert is stopped.\nStopping file-output...\nfile-output is stopped.\nBotnet is stopped.\n
"},{"location":"admin/management/intelmq/#status","title":"status","text":"
With this command we can see the status of all configured bots. Here, the botnet was started beforehand:
 
> intelmqctl status\nabusech-domain-parser is running.\nabusech-feodo-domains-collector is running.\ndeduplicator-expert is running.\nfile-output is disabled.\n
 
And if the disabled bot has also been started:
 
> intelmqctl status\nabusech-domain-parser is running.\nabusech-feodo-domains-collector is running.\ndeduplicator-expert is running.\nfile-output is running.\n
 
If the botnet is stopped, the output looks like this:
 
> intelmqctl status\nabusech-domain-parser is stopped.\nabusech-feodo-domains-collector is stopped.\ndeduplicator-expert is stopped.\nfile-output is disabled.\n
"},{"location":"admin/management/intelmq/#restart","title":"restart","text":"
The same as start and stop consecutively.
"},{"location":"admin/management/intelmq/#reload","title":"reload","text":"
The same as reload of every bot.
"},{"location":"admin/management/intelmq/#enable-disable","title":"enable / disable","text":"
The sub commands enable and disable set the corresponding flags in runtime.yaml.
 
> intelmqctl status\nfile-output is stopped.\nmalware-domain-list-collector is stopped.\nmalware-domain-list-parser is stopped.\n> intelmqctl disable file-output\n> intelmqctl status\nfile-output is disabled.\nmalware-domain-list-collector is stopped.\nmalware-domain-list-parser is stopped.\n> intelmqctl enable file-output\n> intelmqctl status\nfile-output is stopped.\nmalware-domain-list-collector is stopped.\nmalware-domain-list-parser is stopped.\n
"},{"location":"admin/management/intelmq/#manage-individual-bots","title":"Manage individual bots","text":"
As all init systems, intelmqctl has the methods start, stop, restart, reload and status.
"},{"location":"admin/management/intelmq/#start_1","title":"start","text":"
This will start the bot with the ID file-output. A file with it's PID will be created in /opt/intelmq/var/run/[bot-id].pid.
 
> intelmqctl start file-output\nStarting file-output...\nfile-output is running.\n
 
If the bot is already running, it won't be started again:
 
> intelmqctl start file-output\nfile-output is running.\n
"},{"location":"admin/management/intelmq/#stop_1","title":"stop","text":"
If the PID file does exist, a SIGINT will be sent to the process. After 0.25s we check if the process is running. If not, the PID file will be removed.
 
> intelmqctl stop file-output\nStopping file-output...\nfile-output is stopped.\n
 
If there's no running bot, there's nothing to do.
 
> intelmqctl stop file-output\nfile-output was NOT RUNNING.\n
 
If the bot did not stop in 0.25s, intelmqctl will say it's still running:
 
> intelmqctl stop file-output\nfile-output is still running\n
"},{"location":"admin/management/intelmq/#status_1","title":"status","text":"
Checks for the PID file and if the process with the given PID is alive. If the PID file exists, but the process does not exist, it will be removed.
 
> intelmqctl status file-output\nfile-output is stopped.\n> intelmqctl start file-output\nStarting file-output...\nfile-output is running.\n> intelmqctl status file-output\nfile-output is running.\n
"},{"location":"admin/management/intelmq/#restart_1","title":"restart","text":"
The same as stop and start consecutively.
 
> intelmqctl restart file-output\nStopping file-output...\nfile-output is stopped.\nStarting file-output...\nfile-output is running.\n
"},{"location":"admin/management/intelmq/#reload_1","title":"reload","text":"
Sends a SIGHUP to the bot, which will then reload the configuration.
 
> intelmqctl reload file-output\nReloading file-output ...\nfile-output is running.\n
 
If the bot is not running, we can't reload it:
 
> intelmqctl reload file-output\nfile-output was NOT RUNNING.\n
"},{"location":"admin/management/intelmq/#run","title":"run","text":"
This command is used for debugging purposes.
 
If launched with no arguments, the bot will call its init method and start processing messages as usual -- but you see everything happens.
 
> intelmqctl run file-output\nfile-output: RestAPIOutputBot initialized with id file-output and version 3.5.2 as process 12345.\nfile-output: Bot is starting.\nfile-output: Loading source pipeline and queue 'file-output-queue'.\nfile-output: Connected to source queue.\nfile-output: No destination queues to load.\nfile-output: Bot initialization completed.\nfile-output: Waiting for incoming message.\n
 
Note that if another instance of the bot is running, only warning will be displayed.
 
> intelmqctl run file-output\nMain instance of the bot is running in the background. You may want to launch: intelmqctl stop file-output\n
 
You can set the log level with the -l flag, e.g. -l DEBUG. For the 'console' subcommand, 'DEBUG' is the default.
"},{"location":"admin/management/intelmq/#console","title":"console","text":"
This command is used for debugging purposes.
 
If launched with console argument, you get a pdb live console; or ipdb or pudb consoles if they were previously installed (I.E. pip3 install ipdb --user).
 
> intelmqctl run file-output console\n*** Using console ipdb. Please use 'self' to access to the bot instance properties. ***\nipdb> self. ...\n
 
You may specify the desired console in the next argument.
 
> intelmqctl run file-output console pudb\n
"},{"location":"admin/management/intelmq/#message","title":"message","text":"
Operate directly with the input / output pipelines.
 
If get is the parameter, you see the message that waits in the input (source or internal) queue. If the argument is pop, the message gets popped as well.
 
> intelmqctl run file-output message get\nfile-output: Waiting for a message to get...\n{\n    \"classification.type\": \"c&c\",\n    \"feed.url\": \"https://example.com\",\n    \"raw\": \"1233\",\n    \"source.ip\": \"1.2.3.4\",\n    \"time.observation\": \"2017-05-17T22:00:33+00:00\",\n    \"time.source\": \"2017-05-17T22:00:32+00:00\"\n}\n
 
To send directly to the bot's output queue, just as it was sent by self.send_message() in bot's process() method, use the send argument. In our case of file-output, it has no destination queue so that nothing happens.
 
> intelmqctl run file-output message send '{\"time.observation\": \"2017-05-17T22:00:33+00:00\", \"time.source\": \"2017-05-17T22:00:32+00:00\"}'\nfile-output: Bot has no destination queues.\n
 
Note, if you would like to know possible parameters of the message, put a wrong one -- you will be prompted if you want to list all the current bot harmonization.
"},{"location":"admin/management/intelmq/#process","title":"process","text":"
With no other arguments, bot's process() method will be run one time.
 
> intelmqctl run file-output process\nfile-output: Bot is starting.\nfile-output: Bot initialization completed.\nfile-output: Processing...\nfile-output: Waiting for incoming message.\nfile-output: Received message {'raw': '1234'}.\n
 
If run with --dryrun|-d flag, the message gets never really popped out from the source or internal pipeline, nor sent to the output pipeline. Plus, you receive a note about the exact moment the message would get sent, or acknowledged. If the message would be sent to a non-default path, the name of this path is printed on the console.
 
> intelmqctl run file-output process -d\nfile-output:  * Dryrun only, no message will be really sent through.\n...\nfile-output: DRYRUN: Message would be acknowledged now!\n
 
You may trick the bot to process a JSON instead of the Message in its pipeline with --msg|-m flag.
 
> intelmqctl run file-output process -m '{\"source.ip\":\"1.2.3.4\"}'\nfile-output:  * Message from cli will be used when processing.\n...\n
 
If you wish to display the processed message as well, you the --show-sent|-s flag. Then, if sent through (either with --dryrun or without), the message gets displayed as well.
"},{"location":"admin/management/intelmq/#disable","title":"disable","text":"
Sets the enabled flag in the runtime configuration of the bot to false. By default, all bots are enabled.
 
Example output:
 
> intelmqctl status file-output\nfile-output is stopped.\n> intelmqctl disable file-output\n> intelmqctl status file-output\nfile-output is disabled.\n
"},{"location":"admin/management/intelmq/#enable","title":"enable","text":"
Sets the enabled flag in the runtime configuration of the bot to true.
 
Example output:
 
> intelmqctl status file-output\nfile-output is disabled.\n> intelmqctl enable file-output\n> intelmqctl status file-output\nfile-output is stopped.\n
"},{"location":"admin/management/intelmq/#list-bots","title":"List bots","text":"
intelmqctl list bots does list all configured bots and their description.
"},{"location":"admin/management/intelmq/#list-queues","title":"List queues","text":"
intelmqctl list queues shows all queues which are currently in use according to the configuration and how much events are in it:
 
> intelmqctl list queues\nabusech-domain-parser-queue - 0\nabusech-domain-parser-queue-internal - 0\ndeduplicator-expert-queue - 0\ndeduplicator-expert-queue-internal - 0\nfile-output-queue - 234\nfile-output-queue-internal - 0\n
 
Use the -q or --quiet flag to only show non-empty queues:
 
> intelmqctl list queues -q\nfile-output-queue - 234\n
 
The --sum or --count flag will show the sum of events on all queues:
 
> intelmqctl list queues --sum\n42\n
"},{"location":"admin/management/intelmq/#logging","title":"Logging","text":"
intelmqctl can show the last log lines for a bot, filtered by the log level.
 
Logs are stored in /opt/intelmq/var/log/ or /var/log/intelmq/ directory. In case of failures, messages are dumped to the same directory with the file extension .dump.
 
See the help page for more information.
"},{"location":"admin/management/intelmq/#check","title":"Check","text":"
This command will do various sanity checks on the installation and especially the configuration.
"},{"location":"admin/management/intelmq/#orphaned-queues","title":"Orphaned Queues","text":"
The intelmqctl check tool can search for orphaned queues. \"Orphaned queues\" are queues that have been used in the past and are no longer in use. For example you had a bot which you removed or renamed afterwards, but there were still messages in it's source queue. The source queue won't be renamed automatically and is now disconnected. As this queue is no longer configured, it won't show up in the list of IntelMQ's queues too. In case you are using redis as message broker, you can use the redis-cli tool to examine or remove these queues:
 
redis-cli -n 2\nkeys * # lists all existing non-empty queues\nllen [queue-name] # shows the length of the queue [queue-name]\nlindex [queue-name] [index] # show the [index]'s message of the queue [queue-name]\ndel [queue-name] # remove the queue [queue-name]\n
 
To ignore certain queues in this check, you can set the parameter intelmqctl_check_orphaned_queues_ignore in the defaults configuration file. For example:
 
\"intelmqctl_check_orphaned_queues_ignore\": [\"Taichung-Parser\"]\n
"},{"location":"admin/management/intelmq/#configuration-upgrade","title":"Configuration upgrade","text":"
The intelmqctl upgrade-config function upgrade, upgrade the configuration from previous versions to the current one. It keeps track of previously installed versions and the result of all \"upgrade functions\" in the \"state file\", locate in the $var_state_path/state.json /opt/intelmq/var/lib/state.json or /var/lib/intelmq/state.json).
 
This function has been introduced in version 2.0.1.
 
It makes backups itself for all changed files before every run. Backups are overridden if they already exists. So make sure to always have a backup of your configuration just in case.
"},{"location":"admin/management/intelmq/#output-type","title":"Output type","text":"
intelmqctl can be used as command line tool, as library and as tool by other programs. If called directly, it will print all output to the console (stderr). If used as python library, the python types themselves are returned. The third option is to use machine-readable JSON as output (used by other managing tools).
"},{"location":"admin/management/intelmq/#exit-code","title":"Exit code","text":"
In case of errors, unsuccessful operations, the exit code is higher than 0. For example, when running intelmqctl start and one enabled bot is not running, the exit code is 1. The same is valid for e.g. intelmqctl status, which can be used for monitoring, and all other operations.
"},{"location":"admin/management/intelmq/#error-handling","title":"Error Handling","text":"
When bots are failing due to bad input data or programming errors, they can dump the problematic message to a file along with a traceback, if configured accordingly. These dumps are saved at in the logging directory as [botid].dump as JSON files. IntelMQ comes with an inspection and reinjection tool, called intelmqdump. It is an interactive tool to show all dumped files and the number of dumps per file. Choose a file by bot-id or listed numeric id. You can then choose to delete single entries from the file with e 1,3,4, show a message in more readable format with s 1 (prints the raw-message, can be long!), recover some messages and put them back in the pipeline for the bot by a or r 0,4,5. Or delete the file with all dumped messages using d.
 
intelmqdump -h\nusage:\n    intelmqdump [botid]\n    intelmqdump [-h|--help]\n\nintelmqdump can inspect dumped messages, show, delete or reinject them into\nthe pipeline. It's an interactive tool, directly start it to get a list of\navailable dumps or call it with a known bot id as parameter.\n\npositional arguments:\n  botid       botid to inspect dumps of\n\noptional arguments:\n  -h, --help  show this help message and exit\n  --truncate TRUNCATE, -t TRUNCATE\n                        Truncate raw-data with more characters than given. 0 for no truncating. Default: 1000.\n\nInteractive actions after a file has been selected:\n- r, Recover by IDs\n  > r id{,id} [queue name]\n  > r 3,4,6\n  > r 3,7,90 modify-expert-queue\n  The messages identified by a consecutive numbering will be stored in the\n  original queue or the given one and removed from the file.\n- a, Recover all\n  > a [queue name]\n  > a\n  > a modify-expert-queue\n  All messages in the opened file will be recovered to the stored or given\n  queue and removed from the file.\n- d, Delete entries by IDs\n  > d id{,id}\n  > d 3,5\n  The entries will be deleted from the dump file.\n- d, Delete file\n  > d\n  Delete the opened file as a whole.\n- s, Show by IDs\n  > s id{,id}\n  > s 0,4,5\n  Show the selected IP in a readable format. It's still a raw format from\n  repr, but with newlines for message and traceback.\n- e, Edit by ID\n  > e id\n  > e 0\n  > e 1,2\n  Opens an editor (by calling `sensible-editor`) on the message. The modified message is then saved in the dump.\n- q, Quit\n  > q\n\n$ intelmqdump\n id: name (bot id)                    content\n  0: alienvault-otx-parser            1 dumps\n  1: cymru-whois-expert               8 dumps\n  2: deduplicator-expert              2 dumps\n  3: dragon-research-group-ssh-parser 2 dumps\n  4: file-output2                     1 dumps\n  5: fraunhofer-dga-parser            1 dumps\n  6: spamhaus-cert-parser             4 dumps\n  7: test-bot                         2 dumps\nWhich dump file to process (id or name)? 3\nProcessing dragon-research-group-ssh-parser: 2 dumps\n  0: 2015-09-03T13:13:22.159014 InvalidValue: invalid value u'NA' (<type 'unicode'>) for key u'source.asn'\n  1: 2015-09-01T14:40:20.973743 InvalidValue: invalid value u'NA' (<type 'unicode'>) for key u'source.asn'\n(r)ecover by ids, recover (a)ll, delete (e)ntries, (d)elete file, (s)how by ids, (q)uit, edit id (v)? d\nDeleted file /opt/intelmq/var/log/dragon-research-group-ssh-parser.dump\n
 
Bots and the intelmqdump tool use file locks to prevent writing to already opened files. Bots are trying to lock the file for up to 60 seconds if the dump file is locked already by another process (intelmqdump) and then give up. Intelmqdump does not wait and instead only shows an error message.
 
By default, the show command truncates the raw field of messages at 1000 characters to change this limit or disable truncating at all (value 0), use the --truncate parameter.
"},{"location":"admin/management/intelmq/#known-issues","title":"Known issues","text":"
The currently implemented process managing using PID files is very erroneous.
"},{"location":"admin/utilities/bash-completion/","title":"Bash Completion","text":""},{"location":"admin/utilities/bash-completion/#bash-completion","title":"Bash Completion","text":"
To enable bash completion on intelmqctl and intelmqdump in order to help you run the commands in an easy manner, follow the installation process here.
 
Bug
 
This section of the documentation is currently incomplete and will be added later.
"},{"location":"dev/adding-feeds/","title":"Adding Feeds","text":""},{"location":"dev/adding-feeds/#adding-feeds","title":"Adding Feeds","text":"
Adding a feed doesn't necessarily require any programming experience. There are several collector and parser bots intended for general use. Depending on the data source you are trying to add as a feed, it might be only a matter of creating a working combination of collector bot (such as URL Fetcher) configuration and a parser bot (such as CSV parser) configuration. When you are satisfied with the configurations, add it to the intelmq/etc/feeds.yaml file using the following template and open a pull request!
 
<NAME OF THE FEED PROVIDER>:\n    <NAME OF THE FEED>:\n      description: <DESCRIPTION OF WHAT KIND OF DATA THE FEED PROVIDES>\n      additional_information: <ANY ADDITIONAL INFORMATION>\n      documentation: <FEED HOMEPAGE/DOCUMENTATION URL>\n      revision: <DATE WHEN YOU ADDED THIS FEED>\n      public: <TRUE/FALSE IF THE DATA SOURCE IS PUBLICLY AVAILABLE>\n      bots:\n        collector:\n          module: <MODULE USED FOR THE COLLECTOR BOT>\n          parameters:\n            name: __FEED__ # KEEP AS IT IS\n            provider: __PROVIDER__  # KEEP AS IT IS\n            <ADDITIONAL COLLECTOR BOT PARAMETERS>\n        parser:\n          module: <MODULE USED FOR THE PARSER BOT>\n          parameters:\n            <ADDITIONAL PARSER BOT PARAMETERS>\n
 
If the data source utilizes some unusual way of distribution or uses a custom format for the data it might be necessary to develop specialized bot(s) for this particular data source. Always try to use existing bots before you start developing your own. Please also consider extending an existing bot if your use-case is close enough to it's features. If you are unsure which way to take, start an issue and you will receive guidance.
"},{"location":"dev/adding-feeds/#feeds-wishlist","title":"Feeds Wishlist","text":"
This is a list with potentially interesting data sources, which are either currently not supported or the usage is not clearly documented in IntelMQ. If you want to contribute new feeds to IntelMQ, this is a great place to start!
 
Note
 
Some of the following data sources might better serve as an expert bot for enriching processed events.
 
     
  • Lists of feeds:
       
    • threatfeeds.io
      •  
    • TheCyberThreat
      •  
    • sbilly: Awesome Security
      •  
    • pannoniait:Backlists
      •  
    • hslatman:awesome-threat-intelligence
      •  
    • Zeek Intelligence Feeds
      •  
    • imuledx OSING feeds
      •  
     
    •  
  • Some third party intelmq bots: NRDCS IntelMQ fork
    •  
  • List of potentially interesting data sources:
       
    • Abuse.ch SSL Blacklists
      •  
    • AbuseIPDB
      •  
    • Adblock Plus
      •  
    • apivoid IP Reputation API
      •  
    • Anomali Limo Free Intel Feed
      •  
    • APWG's ecrimex
      •  
    • Avast Threat Intel IoCs of dark matter repository
      •  
    • Berkeley
      •  
    • Binary Defense
      •  
    • Bot Invaders Realtime tracker
      •  
    • Botherder Targetedthreats
      •  
    • Botscout Last Caught
      •  
    • botvrij
      •  
    • Carbon Black Feeds
      •  
    • CERT.pl Phishing Warning List
      •  
    • Chaos Reigns
      •  
    • Critical Stack
      •  
    • Cruzit
      •  
    • Cyber Crime Tracker
      •  
    • drb-ra C2IntelFeeds
      •  
    • DNS DB API
      •  
    • ESET Malware Indicators of Compromise
      •  
    • Facebook Threat Exchange
      •  
    • FilterLists
      •  
    • Firehol IPLists
      •  
    • Google Webmaster Alerts
      •  
    • GPF Comics DNS Blacklist
      •  
    • Greensnow
      •  
    • Greynoise
      •  
    • HP Feeds
      •  
    • IBM X-Force Exchange
      •  
    • ImproWare AntiSpam
      •  
    • ISightPartners
      •  
    • James Brine
      •  
    • Joewein
      •  
    • Maltrail:
         
      • Malware
        •  
      • Suspicious
        •  
      • Mass Scanners   (for whitelisting)
        •  
       
      •  
    • Malshare
      •  
    • MalSilo Malware URLs
      •  
    • Malware Config
      •  
    • Malware DB (cert.pl)
      •  
    • MalwareInt
      •  
    • Malware Must Die
      •  
    • Manity Spam IP addresses
      •  
    • Marc Blanchard DGA Domains
      •  
    • MaxMind Proxies
      •  
    • mIRC Servers
      •  
    • MISP Warning Lists
      •  
    • Monzymerza
      •  
    • Multiproxy
      •  
    • Neo23x0 signature-base
      •  
    • OpenBugBounty
      •  
    • Phishing Army
      •  
    • Phishstats (offers JSON API and CSV download)
      •  
    • Project Honeypot (#284)
      •  
    • RST Threat Feed (offers a free and a commercial feed)
      •  
    • SANS ISC
      •  
    • ShadowServer Sandbox API
      •  
    • Shodan search API
      •  
    • Snort
      •  
    • stopforumspam Toxic IP addresses and domains
      •  
    • Spamhaus Botnet Controller List
      •  
    • SteveBlack Hosts File
      •  
    • The Haleys
      •  
    • Threat Crowd
      •  
    • Threat Grid
      •  
    • Threatstream
      •  
    • TotalHash
      •  
    • UCE Protect
      •  
    • Unit 42 Public Report IOCs
      •  
    • URI BL
      •  
    • urlscan.io
      •  
    • Virustotal
      •  
    • virustream
      •  
    • VoIP Blacklist
      •  
    • YourCMC
      •  
     
    •  
"},{"location":"dev/bot-development/","title":"Bot Development","text":""},{"location":"dev/bot-development/#bot-development","title":"Bot Development","text":"
Here you should find everything you need to develop a new bot.
"},{"location":"dev/bot-development/#steps","title":"Steps","text":"
     
  1. Create appropriately placed and named python file.
    2.  
  2. Use correct parent class.
    3.  
  3. Code the functionality you want (with mixins, inheritance, etc).
    4.  
  4. Create appropriately placed test file.
    5.  
  5. Prepare code for testing your bot.
    6.  
  6. Add documentation for your bot.
    7.  
  7. Add changelog and news info.
    8.  
"},{"location":"dev/bot-development/#layout-rules","title":"Layout Rules","text":"
intelmq/\n  lib/\n    bot.py\n    cache.py\n    message.py\n    pipeline.py\n    utils.py\n  bots/\n    collector/\n      <bot name>/\n            collector.py\n    parser/\n      <bot name>/\n            parser.py\n    expert/\n      <bot name>/\n            expert.py\n    output/\n      <bot name>/\n            output.py\n  etc/\n    runtime.yaml\n
 
Assuming you want to create a bot for a new 'Abuse.ch' feed. It turns out that here it is necessary to create different parsers for the respective kind of events (e.g. malicious URLs). Therefore, the usual hierarchy intelmq/bots/parser/<FEED>/parser.py would not be suitable because it is necessary to have more parsers for each Abuse.ch Feed. The solution is to use the same hierarchy with an additional \"description\" in the file name, separated by underscore. Also see the section Directories and Files naming.
 
Example (including the current ones):
 
/intelmq/bots/parser/abusech/parser_domain.py\n/intelmq/bots/parser/abusech/parser_ip.py\n/intelmq/bots/parser/abusech/parser_ransomware.py\n/intelmq/bots/parser/abusech/parser_malicious_url.py\n
"},{"location":"dev/bot-development/#directories-hierarchy-on-default-installation","title":"Directories Hierarchy on Default Installation","text":"
     
  • Configuration Files Path: /opt/intelmq/etc/
    •  
  • PID Files Path: /opt/intelmq/var/run/
    •  
  • Logs Files and dumps Path: /opt/intelmq/var/log/
    •  
  • Additional Bot Files Path, e.g. templates or databases:   /opt/intelmq/var/lib/bots/[bot-name]/
    •  
"},{"location":"dev/bot-development/#directories-and-files-naming","title":"Directories and Files naming","text":"
Any directory and file of IntelMQ has to follow the Directories and Files naming. Any file name or folder name has to:
 
     
  • be represented with lowercase and in case of the name has multiple words, the spaces between them must be removed or replaced by underscores
    •  
  • be self-explaining what the content contains.
    •  
 
In the bot directories name, the name must correspond to the feed provider. If necessary and applicable the feed name can and should be used as postfix for the filename.
 
Examples:
 
intelmq/bots/parser/taichung/parser.py\nintelmq/bots/parser/cymru/parser_full_bogons.py\nintelmq/bots/parser/abusech/parser_ransomware.py\n
"},{"location":"dev/bot-development/#guide","title":"Guide","text":""},{"location":"dev/bot-development/#naming-your-bot-class","title":"Naming your bot class","text":"
Class name of the bot (ex: PhishTank Parser) must correspond to the type of the bot (ex: Parser) e.g. PhishTankParserBot
"},{"location":"dev/bot-development/#choosing-the-parent-class","title":"Choosing the parent class","text":"
Please use the correct bot type as parent class for your bot. The intelmq.lib.bot module contains the following classes:
 
     
  • CollectorBot
    •  
  • ParserBot
    •  
  • ExpertBot
    •  
  • OutputBot
    •  
"},{"location":"dev/bot-development/#template","title":"Template","text":"
Please adjust the doc strings accordingly and remove the in-line comments (#).
 
\"\"\"\nSPDX-FileCopyrightText: 2021 Your Name\nSPDX-License-Identifier: AGPL-3.0-or-later\n\nParse data from example.com, be a nice ExampleParserBot.\n\nDocument possible necessary configurations.\n\"\"\"\nimport sys\n\n# imports for additional libraries and intelmq\nfrom intelmq.lib.bot import ParserBot\n\n\nclass ExampleParserBot(ParserBot):\n    option1: str = \"defaultvalue\"\n    option2: bool = False\n\n    def process(self):\n        report = self.receive_message()\n\n        event = self.new_event(report)  # copies feed.name, time.observation\n        ...  # implement the logic here\n        event.add('source.ip', '127.0.0.1')\n        event.add('extra', {\"os.name\": \"Linux\"})\n        if self.option2:\n            event.add('extra', {\"customvalue\": self.option1})\n\n        self.send_message(event)\n        self.acknowledge_message()\n\n\nBOT = ExampleParserBot\n
 
Any attributes of the bot that are not private can be set by the user using the IntelMQ configuration settings.
 
There are some names with special meaning. These can be used i.e. called:
 
     
  • stop: Shuts the bot down.
    •  
  • receive_message
    •  
  • send_message
    •  
  • acknowledge_message: see next section
    •  
  • start: internal method to run the bot
    •  
 
These can be defined:
 
     
  • init: called at startup, use it to set up the bot (initializing classes, loading files etc)
    •  
  • process: processes the messages
    •  
  • shutdown: To Gracefully stop the bot, e.g. terminate connections
    •  
 
All other names can be used freely.
"},{"location":"dev/bot-development/#mixins","title":"Mixins","text":"
For common settings and methods you can use mixins from intelmq.lib.mixins. To use the mixins, just let your bot inherit from the Mixin class (in addition to the inheritance from the Bot class). For example:
 
class HTTPCollectorBot(CollectorBot, HttpMixin):\n
 
The following mixins are available:
 
     
  • HttpMixin
    •  
  • SqlMixin
    •  
  • CacheMixin
    •  
 
The HttpMixin provides the HTTP attributes described in common-parameters and the following methods:
 
     
  • http_get takes an URL as argument. Any other arguments get passed to the request.Session.get method. http_get   returns a   requests.Response.
    •  
  • http_session can be used if you ever want to work with the session object directly. It takes no arguments and   returns the bots request.Session.
    •  
 
The SqlMixin provides methods to connect to SQL servers. Inherit this Mixin so that it handles DB connection for you. You do not have to bother:
 
     
  • connecting database in the self.init() method, self.cur will be set in the __init__()
    •  
  • catching exceptions, just call self.execute() instead of   self.cur.execute()
    •  
  • self.format_char will be set to '%s' in PostgreSQL and to '?' in SQLite
    •  
 
The CacheMixin provides methods to cache values for bots in a Redis database. It uses the following attributes:
 
     
  • redis_cache_host: str = \"127.0.0.1\"
    •  
  • redis_cache_port: int = 6379
    •  
  • redis_cache_db: int = 9
    •  
  • redis_cache_ttl: int = 15
    •  
  • redis_cache_password: Optional[str] = None
    •  
 
and provides the methods:
 
     
  • cache_exists
    •  
  • cache_get
    •  
  • cache_set
    •  
  • cache_flush
    •  
  • cache_get_redis_instance
    •  
"},{"location":"dev/bot-development/#pipeline-interactions","title":"Pipeline Interactions","text":"
We can call three methods related to the pipeline:
 
     
  • self.receive_message(): The pipeline handler pops one message     from the internal queue if possible. Otherwise one message from     the sources list is popped, and added it to an internal queue. In     case of errors in process handling, the message can still be found     in the internal queue and is not lost. The bot class unravels the     message a creates an instance of the Event or Report class.
    •  
  • self.send_message(event, path=\"_default\"): Processed     message is sent to destination queues. It is possible to change     the destination queues by optional path parameter.
    •  
  • self.acknowledge_message(): Message formerly received by     receive_message is removed from the internal     queue. This should always be done after processing and after the     sending of the new message. In case of errors, this function is     not called and the message will stay in the internal queue waiting     to be processed again.
    •  
"},{"location":"dev/bot-development/#logging","title":"Logging","text":""},{"location":"dev/bot-development/#log-messages-format","title":"Log Messages Format","text":"
Log messages have to be clear and well formatted. The format is the following:
 
Format:
 
<timestamp> - <bot id> - <log level> - <log message>\n
 
Rules:
 
     
  • the Log message MUST follow the common rules of a sentence, beginning with uppercase and ending with period.
    •  
  • the sentence MUST describe the problem or has useful information to give to an inexperienced user a context. Pure stack traces without any further explanation are not helpful.
    •  
 
When the logger instance is created, the bot id must be given as parameter anyway. The function call defines the log level, see below.
"},{"location":"dev/bot-development/#log-levels","title":"Log Levels","text":"
     
  • debug: Debugging information includes retrieved and sent messages, detailed status information. Can include   sensitive information like passwords and amount can be huge.
    •  
  • info: Logs include loaded databases, fetched reports or waiting messages.
    •  
  • warning: Unexpected, but handled behavior.
    •  
  • error: Errors and Exceptions.
    •  
  • critical Program is failing.
    •  
"},{"location":"dev/bot-development/#what-to-log","title":"What to Log","text":"
     
  • Try to keep a balance between obscuring the source code file with hundreds of log messages and having too little log   messages.
    •  
  • In general, a bot MUST report error conditions.
    •  
"},{"location":"dev/bot-development/#how-to-log","title":"How to Log","text":"
The Bot class creates a logger with that should be used by bots. Other components won't log anyway currently. Examples:
 
self.logger.info('Bot start processing.')\nself.logger.error('Pipeline failed.')\nself.logger.exception('Pipeline failed.')\n
 
The exception method automatically appends an exception traceback. The logger instance writes by default to the file /opt/intelmq/var/log/[bot-id].log and to stderr.
"},{"location":"dev/bot-development/#string-formatting-in-logs","title":"String formatting in Logs","text":"
Parameters for string formatting are better passed as argument to the log function, see https://docs.python.org/3/library/logging.html#logging.Logger.debug In case of formatting problems, the error messages will be better. For example:
 
self.logger.debug('Connecting to %r.', host)\n
"},{"location":"dev/bot-development/#error-handling","title":"Error handling","text":"
The bot class itself has error handling implemented. The bot itself is allowed to throw exceptions and intended to fail! The bot should fail in case of malicious messages, and in case of unavailable but necessary resources. The bot class handles the exception and will restart until the maximum number of tries is reached and fail then. Additionally, the message in question is dumped to the file /opt/intelmq/var/log/[bot-id].dump and removed from the queue.
"},{"location":"dev/bot-development/#initialization","title":"Initialization","text":"
Maybe it is necessary so setup a Cache instance or load a file into memory. Use the init function for this purpose:
 
class ExampleParserBot(Bot):\n    def init(self):\n        try:\n            self.database = pyasn.pyasn(self.database)\n        except IOError:\n            self.logger.error(\"pyasn data file does not exist or could not be \"\n                              \"accessed in '%s'.\" % self.database)\n            self.logger.error(\"Read 'bots/experts/asn_lookup/README.md' and \"\n                              \"follow the procedure.\")\n            self.stop()\n
"},{"location":"dev/bot-development/#custom-configuration-checks","title":"Custom configuration checks","text":"
Every bot can define a static method check(parameters) which will be called by intelmqctl check. For example the check function of the ASNLookupExpert:
 
@staticmethod\ndef check(parameters):\n    if not os.path.exists(parameters.get('database', '')):\n        return [[\"error\", \"File given as parameter 'database' does not exist.\"]]\n    try:\n        pyasn.pyasn(parameters['database'])\n    except Exception as exc:\n        return [[\"error\", \"Error reading database: %r.\" % exc]]\n
"},{"location":"dev/bot-development/#running","title":"Running","text":"
You can always start any bot directly from command line by calling the executable. The executable will be created during installation a directory for binaries. After adding new bots to the code, install IntelMQ to get the files created. Don't forget to give an bot id as first argument. Also, running bots with other users than intelmq will raise permission errors.
 
$ sudo -i intelmq\n$ intelmqctl run file-output  # if configured\n$ intelmq.bots.outputs.file.output file-output\n
 
You will get all logging outputs directly on stderr as well as in the log file.
"},{"location":"dev/bot-development/#examples","title":"Examples","text":"
     
  • Check Expert Bots
    •  
  • Check Parser Bots
    •  
"},{"location":"dev/bot-development/#parsers","title":"Parsers","text":"
Parsers can use a different, specialized Bot-class. It allows to work on individual elements of a report, splitting the functionality of the parser into multiple functions:
 
     
  • process: getting and sending data, handling of failures etc.
    •  
  • parse: Parses the report and splits it into single elements (e.g. lines). Can be overridden.
    •  
  • parse_line: Parses elements, returns an Event. Can be overridden.
    •  
  • recover_line: In case of failures and for the field raw, this function recovers a fully functional report containing only one element. Can be overridden.
    •  
 
For common cases, like CSV, existing function can be used, reducing the amount of code to implement. In the best case, only parse_line needs to be coded, as only this part interprets the data.
 
You can have a look at the implementation intelmq/lib/bot.py or at examples, e.g. the DummyBot in intelmq/tests/lib/test_parser_bot.py. This is a stub for creating a new Parser, showing the parameters and possible code:
 
class MyParserBot(ParserBot):\n\n    def parse(self, report):\n        \"\"\"A generator yielding the single elements of the data.\n\n        Comments, headers etc. can be processed here. Data needed by\n        `self.parse_line` can be saved in `self.tempdata` (list).\n\n        Default parser yields stripped lines.\n        Override for your use or use an existing parser, e.g.:\n            parse = ParserBot.parse_csv\n        \"\"\"\n        for line in utils.base64_decode(report.get(\"raw\")).splitlines():\n            yield line.strip()\n\n    def parse_line(self, line, report):\n        \"\"\"A generator which can yield one or more messages contained in line.\n\n        Report has the full message, thus you can access some metadata.\n        Override for your use.\n        \"\"\"\n        raise NotImplementedError\n\n    def process(self):\n        self.tempdata = []  # temporary data for parse, parse_line and recover_line\n        self.__failed = []\n        report = self.receive_message()\n\n        for line in self.parse(report):\n            if not line:\n                continue\n            try:\n                # filter out None\n                events = list(filter(bool, self.parse_line(line, report)))\n            except Exception as exc:\n                self.logger.exception('Failed to parse line.')\n                self.__failed.append((exc, line))\n            else:\n                self.send_message(*events)\n\n        for exc, line in self.__failed:\n            self._dump_message(exc, self.recover_line(line))\n\n        self.acknowledge_message()\n\n    def recover_line(self, line):\n        \"\"\"Reverse of parse for single lines.\n\n        Recovers a fully functional report with only the problematic line.\n        \"\"\"\n        return 'n'.join(self.tempdata + [line])\n\n\nBOT = MyParserBot\n
"},{"location":"dev/bot-development/#parse_line","title":"parse_line","text":"
One line can lead to multiple events, thus parse_line can't just return one Event. Thus, this function is a generator, which allows to easily return multiple values. Use yield event for valid Events and return in case of a void result (not parsable line, invalid data etc.).
"},{"location":"dev/bot-development/#tests","title":"Tests","text":"
In order to do automated tests on the bot, it is necessary to write tests including sample data. Have a look at some existing tests:
 
     
  • The DummyParserBot in intelmq/tests/lib/test_parser_bot.py. This test has the example data (report and event) inside the file, defined as dictionary.
    •  
  • The parser for malwaregroup at intelmq/tests/bots/parsers/malwaregroup/test_parser_*.py. The latter loads a sample HTML file from the same directory, which is the raw report.
    •  
  • The test for ASNLookupExpertBot has two event tests, one is an expected fail (IPv6).
    •  
 
Ideally an example contains not only the ideal case which should succeed, but also a case where should fail instead. (TODO: Implement assertEventNotEqual or assertEventNotcontainsSubset or similar) Most existing bots are only tested with one message. For newly written test it is appreciable to have tests including more then one message, e.g. a parser fed with an report consisting of multiple events.
 
import unittest\n\nimport intelmq.lib.test as test\nfrom intelmq.bots.parsers.exampleparser.parser import ExampleParserBot  # adjust bot class name and module\n\n\nclass TestExampleParserBot(test.BotTestCase, unittest.TestCase):  # adjust test class name\n    \"\"\"A TestCase for ExampleParserBot.\"\"\"\n\n    @classmethod\n    def set_bot(cls):\n        cls.bot_reference = ExampleParserBot  # adjust bot class name\n        cls.default_input_message = EXAMPLE_EVENT  # adjust source of the example event (dict), by default an empty event or report (depending on bot type)\n\n    # This is an example how to test the log output\n    def test_log_test_line(self):\n        \"\"\"Test if bot does log example message.\"\"\"\n        self.run_bot()\n        self.assertRegexpMatches(self.loglines_buffer,\n                                 \"INFO - Lorem ipsum dolor sit amet\")\n\n    def test_event(self):\n        \"\"\"Test if correct Event has been produced.\"\"\"\n        self.run_bot()\n        self.assertMessageEqual(0, EXAMPLE_REPORT)\n\n\nif __name__ == '__main__':  # pragma: no cover\n    unittest.main()\n
 
When calling the file directly, only the tests in this file for the bot will be expected. Some default tests are always executed (via the test.BotTestCase class), such as pipeline and message checks, logging, bot naming or empty message handling.
 
See the testing section about how to run the tests.
"},{"location":"dev/bot-development/#cache","title":"Cache","text":"
Bots can use a Redis database as cache instance. Use the intelmq.lib.utils.Cache class to set this up and/or look at existing bots, like the cymru_whois expert how the cache can be used. Bots must set a TTL for all keys that are cached to avoid caches growing endless over time. Bots must use the Redis databases >= 10, but not those already used by other bots. Look at find intelmq -type f -name '*.py' -exec grep -r 'redis_cache_db' {} + to see which databases are already used.
 
The databases < 10 are reserved for the IntelMQ core:
 
     
  • 2: pipeline
    •  
  • 3: statistics
    •  
  • 4: tests
    •  
"},{"location":"dev/bot-development/#documentation","title":"Documentation","text":"
Please document your added/modified code.
 
For doc strings, we are using the sphinx-napoleon-google-type-annotation.
 
Additionally, Python's type hints/annotations are used, see PEP484.
"},{"location":"dev/bot-development/#testing-pre-releases","title":"Testing Pre-releases","text":""},{"location":"dev/bot-development/#installation","title":"Installation","text":"
The installation procedures need to be adapted only a little bit.
 
For native packages, you can find the unstable packages of the next version here: Installation Unstable Native Packages . The unstable only has a limited set of packages, so enabling the stable repository can be activated in parallel. For CentOS 8 unstable, the stable repository is required.
 
For the installation with pip, use the --pre parameter as shown here following command:
 
pip3 install --pre intelmq\n
 
All other steps are not different. Please report any issues you find in our Issue Tracker.
"},{"location":"dev/data-format/","title":"Data Format","text":""},{"location":"dev/data-format/#data-format","title":"Data Format","text":"
Data passed between bots is called a Message. There are two types of Messages: Report and Event. Report is produced by collector bots and consists of collected raw data (CSV, JSON, HTML, etc) and feed metadata. It is passed to a parser bot which parses Report into a single or multiple Events. Expert bots and output bots handle only Events.
 
All Messages (Reports and Events) are Python dictionaries (or JSONs). The key names and according types are defined by the IntelMQ Data Format.
 
The source code for the Data Format can be found in the Python module intelmq.lib.harmonization and the configuration is present inside the harmonization.conf file. (The term Harmonization is used for historical reasons.)
"},{"location":"dev/data-format/#rules-for-keys","title":"Rules for keys","text":"
The keys are grouped together in sub-fields, e.g. source.ip or source.geolocation.latitude.
 
Only the lower-case alphabet, numbers and the underscore are allowed. Further, the field name must not begin with a number. Thus, keys must match ^[a-z_][a-z_0-9]+(\\.[a-z_0-9]+)*$. These rules also apply for the otherwise unregulated extra. namespace.
"},{"location":"dev/data-format/#data-types","title":"Data Types","text":"
This document describes the IntelMQ data types used for individual events with a description of each allowed field.
"},{"location":"dev/data-format/#asn","title":"ASN","text":"
ASN type. Derived from Integer with forbidden values.
 
Only valid are: 0 < ASN <= 4294967295
 
See https://en.wikipedia.org/wiki/Autonomous_system_(Internet)
 
The first and last ASNs of the original 16-bit integers, namely 0 and 65,535, and the last ASN of the 32-bit numbers, namely 4,294,967,295 are reserved and should not be used by operators.
"},{"location":"dev/data-format/#accuracy","title":"Accuracy","text":"
Accuracy type. A Float between 0 and 100.
"},{"location":"dev/data-format/#base64","title":"Base64","text":"
Base64 type. Always gives unicode strings.
 
Sanitation encodes to base64 and accepts binary and unicode strings.
"},{"location":"dev/data-format/#boolean","title":"Boolean","text":"
Boolean type. Without sanitation only python bool is accepted.
 
Sanitation accepts string 'true' and 'false' and integers 0 and 1.
"},{"location":"dev/data-format/#classificationtaxonomy","title":"ClassificationTaxonomy","text":"
classification.taxonomy type.
 
The mapping follows Reference Security Incident Taxonomy Working Group \u2013 RSIT WG: https://github.com/enisaeu/Reference-Security-Incident-Taxonomy-Task-Force/
 
These old values are automatically mapped to the new ones:
 
     
  • 'abusive content' -> 'abusive-content'
    •  
  • 'information gathering' -> 'information-gathering'
    •  
  • 'intrusion attempts' -> 'intrusion-attempts'
    •  
  • 'malicious code' -> 'malicious-code'
    •  
 
Allowed values are:
 
     
  • abusive-content
    •  
  • availability
    •  
  • fraud
    •  
  • information-content-security
    •  
  • information-gathering
    •  
  • intrusion-attempts
    •  
  • intrusions
    •  
  • malicious-code
    •  
  • other
    •  
  • test
    •  
  • vulnerable
    •  
"},{"location":"dev/data-format/#classificationtype","title":"ClassificationType","text":"
classification.type type.
 
The mapping extends Reference Security Incident Taxonomy Working Group \u2013 RSIT WG:
 
https://github.com/enisaeu/Reference-Security-Incident-Taxonomy-Task-Force/
 
These old values are automatically mapped to the new ones:
 
     
  • 'botnet drone' -> 'infected-system'
    •  
  • 'ids alert' -> 'ids-alert'
    •  
  • 'c&c' -> 'c2-server'
    •  
  • 'c2server' -> 'c2-server'
    •  
  • 'infected system' -> 'infected-system'
    •  
  • 'malware configuration' -> 'malware-configuration'
    •  
  • 'Unauthorised-information-access' -> 'unauthorised-information-access'
    •  
  • 'leak' -> 'data-leak'
    •  
  • 'vulnerable client' -> 'vulnerable-system'
    •  
  • 'vulnerable service' -> 'vulnerable-system'
    •  
  • 'ransomware' -> 'infected-system'
    •  
  • 'unknown' -> 'undetermined'
    •  
 
These values changed their taxonomy: 'malware': In terms of the taxonomy 'malicious-code' they can be either 'infected-system' or 'malware-distribution' but in terms of malware actually, it is now taxonomy 'other'
 
Allowed values are:
 
     
  • application-compromise
    •  
  • blacklist
    •  
  • brute-force
    •  
  • burglary
    •  
  • c2-server
    •  
  • copyright
    •  
  • data-leak
    •  
  • data-loss
    •  
  • ddos
    •  
  • ddos-amplifier
    •  
  • dga-domain
    •  
  • dos
    •  
  • exploit
    •  
  • harmful-speech
    •  
  • ids-alert
    •  
  • infected-system
    •  
  • information-disclosure
    •  
  • malware
    •  
  • malware-configuration
    •  
  • malware-distribution
    •  
  • masquerade
    •  
  • misconfiguration
    •  
  • other
    •  
  • outage
    •  
  • phishing
    •  
  • potentially-unwanted-accessible
    •  
  • privileged-account-compromise
    •  
  • proxy
    •  
  • sabotage
    •  
  • scanner
    •  
  • sniffing
    •  
  • social-engineering
    •  
  • spam
    •  
  • system-compromise
    •  
  • test
    •  
  • tor
    •  
  • unauthorised-information-access
    •  
  • unauthorised-information-modification
    •  
  • unauthorized-use-of-resources
    •  
  • undetermined
    •  
  • unprivileged-account-compromise
    •  
  • violence
    •  
  • vulnerable-system
    •  
  • weak-crypto
    •  
"},{"location":"dev/data-format/#datetime","title":"DateTime","text":"
Date and time type for timestamps.
 
Valid values are timestamps with time zone and in the format '%Y-%m-%dT%H:%M:%S+00:00'. Invalid are missing times and missing timezone information (UTC). Microseconds are also allowed.
 
Sanitation normalizes the timezone to UTC, which is the only allowed timezone.
 
The following additional conversions are available with the convert function:
 
     
  • timestamp
    •  
  • windows_nt: From Windows NT / AD / LDAP
    •  
  • epoch_millis: From Milliseconds since Epoch
    •  
  • from_format: From a given format, eg. 'from_format|%H %M %S %m %d %Y %Z'
    •  
  • from_format_midnight: Date from a given format and assume midnight, e.g. 'from_format_midnight|%d-%m-%Y'
    •  
  • utc_isoformat: Parse date generated by datetime.isoformat()
    •  
  • fuzzy (or None): Use dateutils' fuzzy parser, default if no specific parser is given
    •  
"},{"location":"dev/data-format/#fqdn","title":"FQDN","text":"
Fully qualified domain name type.
 
All valid lowercase domains are accepted, no IP addresses or URLs. Trailing dot is not allowed.
 
To prevent values like '10.0.0.1:8080' (#1235), we check for the non-existence of ':'.
"},{"location":"dev/data-format/#float","title":"Float","text":"
Float type. Without sanitation only python float/integer/long is accepted. Boolean is explicitly denied.
 
Sanitation accepts strings and everything float() accepts.
"},{"location":"dev/data-format/#ipaddress","title":"IPAddress","text":"
Type for IP addresses, all families. Uses the ipaddress module.
 
Sanitation accepts integers, strings and objects of ipaddress.IPv4Address and ipaddress.IPv6Address.
 
Valid values are only strings. 0.0.0.0 is explicitly not allowed.
"},{"location":"dev/data-format/#ipnetwork","title":"IPNetwork","text":"
Type for IP networks, all families. Uses the ipaddress module.
 
Sanitation accepts strings and objects of ipaddress.IPv4Network and ipaddress.IPv6Network. If host bits in strings are set, they will be ignored (e.g 127.0.0.1/32).
 
Valid values are only strings.
"},{"location":"dev/data-format/#integer","title":"Integer","text":"
Integer type. Without sanitation only python integer/long is accepted. Bool is explicitly denied.
 
Sanitation accepts strings and everything int() accepts.
"},{"location":"dev/data-format/#json","title":"JSON","text":"
JSON type.
 
Sanitation accepts any valid JSON objects.
 
Valid values are only unicode strings with JSON objects.
"},{"location":"dev/data-format/#jsondict","title":"JSONDict","text":"
JSONDict type.
 
Sanitation accepts pythons dictionaries and JSON strings.
 
Valid values are only unicode strings with JSON dictionaries.
"},{"location":"dev/data-format/#lowercasestring","title":"LowercaseString","text":"
Like string, but only allows lower case characters.
 
Sanitation lowers all characters.
"},{"location":"dev/data-format/#registry","title":"Registry","text":"
Registry type. Derived from UppercaseString.
 
Only valid values: AFRINIC, APNIC, ARIN, LACNIC, RIPE. RIPE-NCC and RIPENCC are normalized to RIPE.
"},{"location":"dev/data-format/#string","title":"String","text":"
Any non-empty string without leading or trailing whitespace.
"},{"location":"dev/data-format/#tlp","title":"TLP","text":"
TLP level type. Derived from UppercaseString.
 
Only valid values: WHITE, GREEN, AMBER, RED.
 
Accepted for sanitation are different cases and the prefix 'tlp:'.
"},{"location":"dev/data-format/#url","title":"URL","text":"
URI type. Local and remote.
 
Sanitation converts hxxp and hxxps to http and https. For local URIs (file) a missing host is replaced by localhost.
 
Valid values must have the host (network location part).
"},{"location":"dev/data-format/#uppercasestring","title":"UppercaseString","text":"
Like string, but only allows upper case characters.
 
Sanitation uppers all characters.
"},{"location":"dev/documentation/","title":"Documentation","text":""},{"location":"dev/documentation/#documentation","title":"Documentation","text":"
The documentation is automatically published to https://docs.intelmq.org at every push to the develop branch of the repository.
 
To build the documentation you need additional packages:
 
pip3 install .[development]\n
 
Then use the Makefile to build the documentation using mkdocs:
 
make docs\n
 
Some parts of the documentation are automatically generated using dedicated scripts. You can find them in the Makefile.
"},{"location":"dev/environment/","title":"Environment","text":""},{"location":"dev/environment/#development-environment","title":"Development Environment","text":""},{"location":"dev/environment/#directories","title":"Directories","text":"
For development purposes, you need two directories:
 
     
  • directory with the local source code repository
    •  
  • root directory of the IntelMQ installation
    •  
 
The default root directory of the IntelMQ installation is /opt/intelmq. This directory is used for configurations (/opt/intelmq/etc), local states (/opt/intelmq/var/lib) and logs (/opt/intelmq/var/log). If you want to change it, please set the INTELMQ_ROOT_DIR environment variable with a desired location.
 
For repository directory, you can use any path that is accessible by users you use to run IntelMQ. For globally installed IntelMQ, the directory has to be readable by other unprivileged users (e.g. home directories on Fedora can't be read by other users by default).
 
To keep commands in the guide universal, we will use environmental variables for repository and installation paths. You can set them with following commands:
 
# Adjust paths if you want to use non-standard directories\nexport INTELMQ_REPO=/opt/dev_intelmq\nexport INTELMQ_ROOT_DIR=/opt/intelmq\n
 
Note
 
If using non-default installation directory, remember to keep the root directory variable set for every run of IntelMQ commands. If you don't, then the default location /opt/intelmq will be used.
"},{"location":"dev/environment/#installation","title":"Installation","text":"
Developers can create a fork repository of IntelMQ in order to commit the new code to this repository and then be able to do pull requests to the main repository. Otherwise you can just use the 'certtools' as username below.
 
The following instructions will use pip3 -e, which gives you a so called editable installation. No code is copied in the libraries directories, there's just a link to your code. However, configuration files still required to be moved to /opt/intelmq as the instructions show.
 
The traditional way to work with IntelMQ is to install it globally and have a separated user for running it. If you wish to separate your machine Python's libraries, e.g. for development purposes, you could alternatively use a Python virtual environment and your local user to run IntelMQ. Please use your preferred way from instructions below.
"},{"location":"dev/environment/#using-globally-installed-intelmq","title":"Using globally installed IntelMQ","text":"
sudo -s\n\ngit clone https://github.com/<your username>/intelmq.git $INTELMQ_REPO\ncd $INTELMQ_REPO\n\npip3 install -e .\n\nuseradd -d $INTELMQ_ROOT_DIR -U -s /bin/bash intelmq\n\nintelmqsetup\n
"},{"location":"dev/environment/#using-virtual-environment","title":"Using virtual environment","text":"
git clone https://github.com/<your username>/intelmq.git $INTELMQ_REPO\ncd $INTELMQ_REPO\n\npython -m venv .venv\nsource .venv/bin/activate\n\npip install -e .\n\n# If you use a non-local directory as INTELMQ_ROOT_DIR, use following\n# command to create it and change the ownership.\nsudo install -g `whoami` -o `whoami` -d $INTELMQ_ROOT_DIR\n# For local directory, just create it with mkdir:\nmkdir $INTELMQ_ROOT_DIR\n\nintelmqsetup --skip-ownership\n
 
Note
 
Please do not forget that configuration files, log files will be available on $INTELMQ_ROOT_DIR. However, if your development is somehow related to any shipped configuration file, you need to apply the changes in your repository $INTELMQ_REPO/intelmq/etc/.
"},{"location":"dev/environment/#additional-services","title":"Additional services","text":"
Some features require additional services, like message queue or database. The commonly used services are gained for development purposes in the Docker Compose file in contrib/development-tools/docker-compose-common-services.yaml in the repository. You can use them to run services on your machine in a docker containers, or decide to configure them in an another way. To run them using Docker Compose, use following command from the main repository directory:
 
# For older Docker versions, you may need to use `docker-compose` command\ndocker compose -f contrib/development-tools/docker-compose-common-services.yaml up -d\n
 
This will start in the background containers with Redis, RabbitMQ, PostgreSQL and MongoDB.
"},{"location":"dev/environment/#how-to-develop","title":"How to develop","text":"
After you successfully setup your IntelMQ development environment, you can perform any development on any .py file on $INTELMQ_REPO. After you change, you can use the normal procedure to run the bots:
 
su - intelmq # Use for global installation\nsource .venv/bin/activate # Use for virtual environment installation\n\nintelmqctl start spamhaus-drop-collector\n\ntail -f $INTELMQ_ROOT_DIR/var/log/spamhaus-drop-collector.log\n
 
You can also add new bots, creating the new .py file on the proper directory inside cd $INTELMQ_REPO/intelmq. However, your IntelMQ installation with pip3 needs to be updated. Please check the following section.
"},{"location":"dev/environment/#update","title":"Update","text":"
In case you developed a new bot, you need to update your current development installation. In order to do that, please follow this procedure:
 
     
  1. Make sure that you have your new bot in the right place.
    2.  
  2.  
    Update pip metadata and new executables: 
    sudo -s # Use for global installation\nsource .venv/bin/activate # Use for virtual environment installation\n\ncd /opt/dev_intelmq\npip3 install -e .\n
     
    3.  
  3.  
    If you're using the global installation, an additional step of changing permissions and ownership is necessary: 
    find $INTELMQ_ROOT_DIR/ -type d -exec chmod 0770 {} \\+\nfind $INTELMQ_ROOT_DIR/ -type f -exec chmod 0660 {} \\+\nchown -R intelmq.intelmq $INTELMQ_ROOT_DIR\n## if you use the intelmq manager (adapt the webservers' group if needed):\nchown intelmq.www-data $INTELMQ_ROOT_DIR/etc/*.conf\n
     
    4.  
 
Now you can test run your new bot following this procedure:
 
su - intelmq              # Use for global installation\nsource .venv/bin/activate # Use for virtual environment installation\n\nintelmqctl start <bot_id>\n
"},{"location":"dev/extensions-packages/","title":"Extensions Packages","text":""},{"location":"dev/extensions-packages/#creating-extensions-packages","title":"Creating extensions packages","text":"
IntelMQ supports adding additional bots using your own independent packages. You can use this to add a new integration that is special to you, or cannot be integrated into the main IntelMQ repository for some reason.
"},{"location":"dev/extensions-packages/#building-an-extension-package","title":"Building an extension package","text":"
A simple example of the package can be found in contrib/example-extension-package. To make your custom bots work with IntelMQ, you need to ensure that
 
     
  • your bot's module exposes a BOT object of the class inherited from intelmq.lib.bot.Bot    or its subclasses,
    •  
  • your package registers an entry point    in the console_scripts group with a name starting with intelmq.bots. followed by    the name of the group (collectors, experts, outputs, parsers), and then your original name.    The entry point must point to the BOT.run method,
    •  
  • the module in which the bot resides must be importable by IntelMQ (e.g. installed in the same    virtualenv, if you use them).
    •  
 
Apart from these requirements, your package can use any of the usual package features. We strongly recommend following the same principles and main guidelines as the official bots. This will ensure the same experience when using official and additional bots.
"},{"location":"dev/extensions-packages/#naming-convention","title":"Naming convention","text":"
Building your own extensions gives you a lot of freedom, but it's important to know that if your bot's entry point uses the same name as another bot, it may not be possible to use it, or to determine which one is being used. For this reason, we recommend that you start the name of your bot with an with an organization identifier and then the bot name.
 
For example, if I create a collector bot for feed source Special and run it on behalf of the organization Awesome, the suggested entry point might be intelmq.bots.collectors.awesome.special. Note that the structure of your package doesn't matter, as long as it can be imported properly.
 
For example, I could create a package called awesome-bots with the following file structure
 
   awesome_bots\n   \u251c\u2500\u2500 pyproject.toml\n   \u2514\u2500\u2500 awesome_bots\n        \u251c\u2500\u2500 __init__.py\n        \u2514\u2500\u2500 special.py\n
 
The pyproject.toml file would then have the following section:
 
   [project.scripts]\n   intelmq.bots.collectors.awesome.special = \"awesome_bots.special:BOT.run\"\n
 
Once you have installed your package, you can run intelmqctl list bots to check if your bot was properly registered.
"},{"location":"dev/guidelines/","title":"Guidelines","text":""},{"location":"dev/guidelines/#development-guidelines","title":"Development Guidelines","text":""},{"location":"dev/guidelines/#coding-rules","title":"Coding-Rules","text":"
Most important: KEEP IT SIMPLE! This can not be over-estimated. Feature creep can destroy any good software project. But if new folks can not understand what you wrote in 10-15 minutes, it is not good. It's not about the performance, etc. It's about readability.
 
In general, we follow PEP8. We recommend reading it before committing code.
 
There are some exceptions: sometimes it does not make sense to check for every PEP8 error (such as whitespace indentation when you want to make a dict=() assignment look pretty. Therefore, we do have some exceptions defined in the setup.cfg file.
 
We support Python 3 only.
"},{"location":"dev/guidelines/#unicode","title":"Unicode","text":"
     
  • Each internal object in IntelMQ (Event, Report, etc) that has strings, their strings MUST be in UTF-8 Unicode format.
    •  
  • Any data received from external sources MUST be transformed into UTF-8 Unicode format before add it to IntelMQ   objects.
    •  
"},{"location":"dev/guidelines/#back-end-independence-and-compatibility","title":"Back-end independence and Compatibility","text":"
Any component of the IntelMQ MUST be independent of the message queue technology (Redis, RabbitMQ, etc...).
"},{"location":"dev/guidelines/#license-header","title":"License Header","text":"
Please add a license and copyright header to your bots. There is a Github action that tests for reuse compliance of your code files.
"},{"location":"dev/guidelines/#intelmq-data-format-rules","title":"IntelMQ Data Format Rules","text":"
Any component of IntelMQ MUST respect the IntelMQ Data Format.
"},{"location":"dev/guidelines/#code-submission-rules","title":"Code Submission Rules","text":""},{"location":"dev/guidelines/#releases-repositories-and-branches","title":"Releases, Repositories and Branches","text":"
     
  • The main repository is in github.com/certtools/intelmq.
    •  
  • We use semantic versioning.
    •  
  • If you contribute something, please fork the repository, create a separate branch and use this for pull requests, see section below.
    •  
  • There are a couple of forks which might be regularly merged into the main repository. They are independent and can have incompatible changes and can deviate from the upstream repository.
    •  
"},{"location":"dev/guidelines/#branching-model","title":"Branching model","text":"
     
  • \"master\" is the stable branch. It hold the latest stable release. Non-developers should only work on this branch. The recommended log level is WARNING. Code is only added by merges from the maintenance branches.
    •  
  • \"maintenance/a.b.x\" branches accumulate (cherry-picked) patches for a maintenance release (a.b.x). Recommended for   experienced users which deploy intelmq themselves. No new features will be added to these branches.
    •  
  • \"develop\" is the development branch for the next stable release   (a.x). New features must go there. Developers may want to work on this branch. This branch also holds all patches from   maintenance releases if applicable. The recommended log level is DEBUG.
    •  
  • Separate branches to develop features or bug fixes may be used by any contributor.
    •  
"},{"location":"dev/guidelines/#how-to-contribute","title":"How to Contribute","text":"
     
  • Make separate pull requests / branches on GitHub for changes. This allows us to discuss things via GitHub.
    •  
  • We prefer one Pull Request per feature or change. If you have a bunch of small fixes, please don't create one PR per fix :)
    •  
  • Only very small and changes (docs, ...) might be committed directly to development branches without Pull Request by the core-team.
    •  
  • Keep the balance between atomic commits and keeping the amount of commits per PR small. You can use interactive   rebasing to squash multiple small commits into one (rebase -i [base-branch]). Only do rebasing if the code you are rebasing is yet not used by others or is already merged - because then others may need to run into conflicts.
    •  
  • Make sure your PR is merge able in the develop branch and all tests are successful.
    •  
  • If possible sign your commits with GPG.
    •  
"},{"location":"dev/guidelines/#workflow","title":"Workflow","text":"
We assume here, that origin is your own fork. We first add the upstream repository:
 
 git remote add upstream https://github.com/certtools/intelmq.git\n
 
Syncing develop:
 
 git checkout develop\n git pull upstream develop\n git push origin develop\n
 
You can do the same with the branches master and maintenance.
 
Create a separate feature-branch to work on, sync develop with upstream. Create working branch from develop:
 
 git checkout develop\n git checkout -b bugfix\n# your work\n git commit\n
 
Or, for bugfixes create a separate bugfix-branch to work on, sync maintenance with upstream. Create working branch from maintenance:
 
git checkout maintenance\ngit checkout -b new-feature\n# your work\ngit commit\n
 
Getting upstream's changes for master or any other branch:
 
git checkout develop\ngit pull upstream develop\ngit push origin develop\n
 
There are 2 possibilities to get upstream's commits into your branch. Rebasing and Merging. Using rebasing, your history is rewritten, putting your changes on top of all other commits. You can use this if your changes are not published yet (or only in your fork).
 
git checkout bugfix\ngit rebase develop\n
 
Using the -i flag for rebase enables interactive rebasing. You can then remove, reorder and squash commits, rewrite commit messages, beginning with the given branch, e.g. develop.
 
Or using merging. This doesn't break the history. It's considered more , but also pollutes the history with merge commits.
 
git checkout bugfix\ngit merge develop\n
 
You can then create a PR with your branch bugfix to our upstream repository, using GitHub's web interface.
"},{"location":"dev/guidelines/#commit-messages","title":"Commit Messages","text":"
If it fixes an existing issue, please use GitHub syntax, e.g.: fixes certtools/intelmq#<IssueID>
"},{"location":"dev/guidelines/#prepare-for-discussion-in-github","title":"Prepare for Discussion in GitHub","text":"
If we don't discuss it, it's probably not tested.
"},{"location":"dev/guidelines/#license-and-author-files","title":"License and Author files","text":"
License and Authors files can be found at the root of repository.
 
     
  • License file MUST NOT be modified except by the explicit written permission by CNCS/CERT.PT or CERT.at
    •  
  • Credit to the authors file must be always retained. When a new contributor (person and/or organization) improves in   some way the repository content (code or documentation), he or she might add his name to the list of contributors.
    •  
 
License and authors must be only listed in an external file but not inside the code files.
"},{"location":"dev/intro/","title":"Intro","text":""},{"location":"dev/intro/#intro","title":"Intro","text":"
This guide is for developers of IntelMQ. It explains the code architecture, coding guidelines as well as ways you can contribute code or documentation. If you have not done so, please read the User Guide and the Administrator Guide first. Once you feel comfortable running IntelMQ with open source bots and you feel adventurous enough to contribute to the project, this guide is for you. It does not matter if you are an experienced Python programmer or just a beginner. There is a lot of examples to help you out.
 
However, before we go into the details, it is important to observe and internalize some overall project goals.
"},{"location":"dev/intro/#goals","title":"Goals","text":"
It is important, that all developers agree and stick to these meta-guidelines. IntelMQ tries to:
 
     
  • Be well tested. For developers this means, we expect you to write unit tests for bots. Every time.
    •  
  • Reduce the complexity of system administration.
    •  
  • Reduce the complexity of writing new bots for new data feeds.
    •  
  • Make your code easily and pleasantly readable.
    •  
  • Reduce the probability of events lost in all process with persistence functionality (even system crash).
    •  
  • Strictly adhere to the existing format for keys and values in events.
    •  
  • Always use JSON format for all messages internally.
    •  
  • Help and support the interconnection between IntelMQ and existing tools like AbuseHelper, CIF, etc. or new tools (in other words: we will not accept data-silos!).
    •  
  • Provide an easy way to store data into log collectors such as ElasticSearch or Splunk.
    •  
  • Provide an easy way to create your own black-lists.
    •  
  • Provide easy to understand interfaces with other systems via HTTP RESTFUL API.
    •  
 
The main take away point from the list above is: things MUST stay intuitive and easy. How do you ultimately test if things are still easy? Let them new programmers test-drive your features and if it is not understandable in 15 minutes, go back to the drawing board.
 
Similarly, if code does not get accepted upstream by the main developers, it is usually only because of the ease-of-use argument. Do not give up, go back to the drawing board, and re-submit again.
"},{"location":"dev/intro/#mailing-list","title":"Mailing list","text":"
There is a separate mailing list for developers to discuss development topics: The IntelMQ-DevArchive is public as well.
"},{"location":"dev/intro/#github","title":"GitHub","text":"
The ideal way to propose changes and additions to IntelMQ is to open a Pull Request on GitHub.
"},{"location":"dev/library/","title":"Use as Library","text":""},{"location":"dev/library/#running-intelmq-as-library","title":"Running IntelMQ as Library","text":""},{"location":"dev/library/#introduction","title":"Introduction","text":"
The feature is specified in IEP007.
"},{"location":"dev/library/#quickstart","title":"Quickstart","text":"
First, import the Python module and a helper. More about the BotLibSettings later.
 
from intelmq.lib.bot import BotLibSettings\nfrom intelmq.bots.experts.domain_suffix.expert import DomainSuffixExpertBot\n
 
Then we need to initialize the bot's instance. We pass two parameters:
 
     
  • bot_id: The id of the bot
    •  
  • settings: A Python dictionary of runtime configuration parameters, see runtime-configuration. The bot first loads the runtime configuration file if it exists. Then we update them with the BotLibSettings which are some accumulated settings disabling the logging to files and configure the pipeline so that we can send and receive messages directly to/from the bot. Last by not least, the actual bot parameters, taking the highest priority.
    •  
 
domain_suffix = DomainSuffixExpertBot('domain-suffix',  # bot id\nsettings=BotLibSettings | {\n'field': 'fqdn',\n'suffix_file': '/usr/share/publicsuffix/public_suffix_list.dat'}\n
 
As the bot is not fully initialized, we can process messages now. Inserting a message as dictionary:
 
queues = domain_suffix.process_message({'source.fqdn': 'www.example.com'})\n
 
The return value is a dictionary of queues, e.g. the output queue and the error queue. More details below.
 
The methods accepts multiple messages as positional argument:
 
domain_suffix.process_message(\n    {'source.fqdn': 'www.example.com'},\n    {'source.fqdn': 'www.example.net'}\n)\ndomain_suffix.process_message(*[\n    {'source.fqdn': 'www.example.com'},\n    {'source.fqdn': 'www.example.net'}\n])\n
 
Select the output queue (as defined in destination_queues), first message, access the field source.domain_suffix: 
>>> output['output'][0]['source.domain_suffix']\n'com'\n
"},{"location":"dev/library/#configuration","title":"Configuration","text":"
Configuration files are not required to run IntelMQ as library. Contrary to IntelMQ normal behavior, if the files runtime.yaml and harmonization.conf do not exist, IntelMQ won't raise any errors. For the harmonization configuration, internal defaults are loaded.
"},{"location":"dev/release/","title":"Release","text":""},{"location":"dev/release/#release-procedure","title":"Release procedure","text":"
General assumption: You are working on branch maintenance, the next version is a bug fix release. For feature releases it is slightly different.
"},{"location":"dev/release/#check-before","title":"Check before","text":"
     
  • Make sure the current state is really final ;) You can test most of     the steps described here locally before doing it real.
    •  
  • Check the upgrade functions in intelmq/lib/upgrades.py.
    •  
  • Close the milestone on GitHub and move any open issues to the next     one.
    •  
  • docs/admin/installation/linux-packages.md: Update supported operating systems.
    •  
"},{"location":"dev/release/#documentation","title":"Documentation","text":"
These apply to all projects:
 
     
  • CHANGELOG.MD and NEWS.MD: Update the latest header, fix the order, remove empty sections and (re)group the entries if necessary.
    •  
  • debian/changelog: Insert a new section for the new version with the tool dch or update the version of the existing last item if yet unreleased. Don't forget the revision after the version number!
    •  
"},{"location":"dev/release/#intelmq","title":"IntelMQ","text":"
     
  • intelmq/version.py: Update the version.
    •  
 
Eventually adapt the default log levels if necessary. Should be INFO for stable releases.
"},{"location":"dev/release/#intelmq-api","title":"IntelMQ API","text":"
     
  • intelmq_api/version.py: Update the version.
    •  
"},{"location":"dev/release/#intelmq-manager","title":"IntelMQ Manager","text":"
     
  • intelmq_manager/version.py: Update the version.
    •  
  • intelmq_manager/static/js/about.js: Update the version.
    •  
"},{"location":"dev/release/#commit-push-review-and-merge","title":"Commit, push, review and merge","text":"
Commit your changes in a separate branch, the final commit message should start with REL:. Push and create a pull request to the develop branch. Someone else should review the changes. Eventually fix them, make sure the REL: is the last commit, you can also push that one at last, after the reviews.
 
Why a separate branch? Because if problems show up, you can still force-push to that one, keeping the release commit the latest one.
"},{"location":"dev/release/#tag-and-release","title":"Tag and release","text":"
Tag the commit with git tag -s version HEAD, merge it into develop, push the branches and the tag. The tag is just a.b.c, not prefixed with v (that was necessary only with SVN a long time ago...).
 
Go to https://github.com/certtools/intelmq/tags and enter the release notes (from the CHANGELOG) for the new tag, then it's considered a release by GitHub.
"},{"location":"dev/release/#tarballs-and-pypi","title":"Tarballs and PyPI","text":"
     
  • Build the source and binary (wheel) distribution:
    •  
 
rm -r build/\npython3 setup.py sdist bdist_wheel\n
 
     
  • Upload the files including signatures to PyPI with e.g. twine: twine upload -u __token__ -p $APITOKEN dist/intelmq... (or set the API Token in .pypirc).
    •  
"},{"location":"dev/release/#documentation_1","title":"Documentation","text":"
Since using mkdocs (see https://docs.intelmq.org) nothing needs to be done anymore.
"},{"location":"dev/release/#packages","title":"Packages","text":"
We are currently using the public Open Build Service instance of openSUSE: http://build.opensuse.org/project/show/home:sebix:intelmq
 
First, test all the steps first with the unstable-repository and check that at least installations succeed.
 
     
  • Create the tarballs with the script create-archives.sh.
    •  
  • Update the dsc and spec files for new filenames and versions.
    •  
  • Update the .changes file
    •  
  • Build locally for all distributions.
    •  
  • Commit.
    •  
"},{"location":"dev/release/#docker-image","title":"Docker Image","text":"
Releasing a new Docker image is very easy.
 
     
  • Clone IntelMQ Docker Repository with git clone https://github.com/certat/intelmq-docker.git --recursive as this repository contains submodules
    •  
  • If the intelmq-docker repository is not updated yet, use git pull --recurse-submodules to pull the latest changes from their respective repository.
    •  
  • Run ./build.sh, check your console if the build was successful.
    •  
  • Run ./test.sh - It will run nosetests3 with the exotic flag. All     errors/warnings will be displayed.
    •  
  • Change the build_version in publish.sh to the new version you     want to release.
    •  
  • Change the namespace variable in publish.sh.
    •  
  • If no error/warning was shown, you can release with ./publish.sh.
    •  
  • Update the DockerHub ReadMe and add the latest version.
    •  
  • Commit and push the updates to the intelmq-docker repository
    •  
"},{"location":"dev/release/#announcements","title":"Announcements","text":"
Announce the new version at the mailinglists intelmq-users, intelmq-dev. For bigger releases, probably also at IHAP, Twitter, etc. Ask your favorite social media consultant.
"},{"location":"dev/release/#prepare-new-version","title":"Prepare new version","text":"
Increase the version in intelmq/version.py and declare it as alpha version. Add the new version in intelmq/lib/upgrades.py. Add a new entry in debian/changelog with dch -v [version] -c debian/changelog.
 
Add new entries to CHANGELOG.md and NEWS.md.
"},{"location":"dev/release/#intelmq_1","title":"IntelMQ","text":"
For CHANGELOG.md:
 
### Configuration\n\n### Core\n\n### Development\n\n### Data Format\n\n### Bots\n#### Collectors\n\n#### Parsers\n\n#### Experts\n\n#### Outputs\n\n### Documentation\n\n### Packaging\n\n### Tests\n\n### Tools\n\n### Contrib\n\n### Known issues\n
 
And for NEWS.md:
 
### Requirements\n\n### Tools\n\n### Data Format\n\n### Configuration\n\n### Libraries\n\n### Postgres databases\n
"},{"location":"dev/release/#intelmq-api_1","title":"IntelMQ API","text":"
An empty section of CHANGELOG.rst.
"},{"location":"dev/release/#intelmq-manager_1","title":"IntelMQ Manager","text":"
For CHANGELOG.md:
 
### Pages\n\n#### Landing page\n\n#### Configuration\n\n#### Management\n\n#### Monitor\n\n#### Check\n\n### Documentation\n\n### Third-party libraries\n\n### Packaging\n\n### Known issues\n
 
And an empty section in the NEWS.md file.
"},{"location":"dev/structure/","title":"Structure","text":""},{"location":"dev/structure/#system-overview","title":"System Overview","text":"
In the intelmq/lib/ directory you can find some libraries:
 
     
  • Bots: Defines base structure for bots and handling of startup, stop,     messages etc.
    •  
  • Cache: For some expert bots it does make sense to cache external     lookup results. Redis is used here.
    •  
  • Harmonization: For defined types, checks and sanitation methods are     implemented.
    •  
  • Message: Defines Events and Reports classes, uses harmonization to     check validity of keys and values according to config.
    •  
  • Pipeline: Writes messages to message queues. Implemented for     productions use is only Redis, AMQP is beta.
    •  
  • Test: Base class for bot tests with predefined test and assert     methods.
    •  
  • Utils: Utility functions used by system components.
    •  
"},{"location":"dev/structure/#code-architecture","title":"Code Architecture","text":""},{"location":"dev/testing/","title":"Testing","text":""},{"location":"dev/testing/#testing","title":"Testing","text":""},{"location":"dev/testing/#additional-test-requirements","title":"Additional test requirements","text":"
Libraries required for tests are listed in the setup.py file. You can install them with pip:
 
pip3 install -e .[development]\n
 
or the package management of your operating system.
"},{"location":"dev/testing/#run-the-tests","title":"Run the tests","text":"
All changes have to be tested and new contributions should be accompanied by according unit tests. Please do not run the tests as root just like any other IntelMQ component for security reasons. Any other unprivileged user is possible.
 
You can run the tests by changing to the directory with IntelMQ repository and running either unittest or pytest. For virtual environment installation, please activate it and omit the sudo -u from examples below:
 
cd $INTELMQ_REPO\nsudo -u intelmq python3 -m unittest {discover|filename}  # or\nsudo -u intelmq pytest [filename]\nsudo -u intelmq python3 setup.py test  # uses a build environment (no external dependencies)\n
 
Some bots need local databases to succeed. If you only want to test one explicit test file, give the file path as argument.
 
There are multiple GitHub Action Workflows setup for automatic testing, which are triggered on pull requests. You can also easily activate them for your forks.
"},{"location":"dev/testing/#environment-variables","title":"Environment variables","text":"
There are a bunch of environment variables which switch on/off some tests:
 Environment\u00a0Variable\u00a0Name Description INTELMQ_TEST_DATABASES databases such as postgres, elasticsearch, mongodb are not tested by default. Set this environment variable to 1 to test those bots. These tests need preparation, e.g. running databases with users and certain passwords etc. Have a look at the .github/workflows/unittests.yml and the corresponding .github/workflows/scripts/setup-full.sh in IntelMQ's repository for steps to set databases up. INTELMQ_SKIP_INTERNET tests requiring internet connection will be skipped if this is set to 1. INTELMQ_SKIP_REDIS redis-related tests are ran by default, set this to 1 to skip those. INTELMQ_TEST_EXOTIC some bots and tests require libraries which may not be available, those are skipped by default. To run them, set this to 1. INTELMQ_TEST_REDIS_PASSWORD Set this value to the password for the local redis database if needed. INTELMQ_LOOKYLOO_TEST Set this value to run the lookyloo tests. Public lookyloo instance will be used as default. INTELMQ_TEST_INSTALLATION Set this value to run tests which require a local IntelMQ installation, such as for testing the command lines tools relying on configuration files, dump files etc. 
For example, to run all tests you can use:
 
INTELMQ_TEST_DATABASES=1 INTELMQ_TEST_EXOTIC=1 pytest intelmq/tests/\n
"},{"location":"dev/testing/#configuration-test-files","title":"Configuration test files","text":"
The tests use the configuration files in your working directory, not those installed in /opt/intelmq/etc/ or /etc/. You can run the tests for a locally changed intelmq without affecting an installation or requiring root to run them.
"},{"location":"tutorials/intelmq-manager/","title":"Using IntelMQ Manager","text":""},{"location":"tutorials/intelmq-manager/#tutorial-on-using-intelmq-manager","title":"Tutorial on using IntelMQ Manager","text":"
Bug
 
This section of the documentation is currently incomplete and will be updated later.
"},{"location":"unsorted/botnet-concept/","title":"Botnet concept","text":""},{"location":"unsorted/botnet-concept/#botnet-concept","title":"Botnet Concept","text":"
The \\\"botnet\\\" represents all currently configured bots which are explicitly enabled. It is, in essence, the graph of the bots which are connected together via their input source queues and destination queues.
 
To get an overview which bots are running, use intelmqctl status or use the IntelMQ Manager. Set \"enabled\": true in the runtime configuration to add a bot to the botnet. By default, bots will be configured as \"enabled\": true. See bots{.interpreted-text role=\"doc\"} for more details on configuration.
 
Disabled bots can still be started explicitly using intelmqctl start <bot_id>, but will remain in the state disabled if stopped (and not be implicitly enabled by the start command). They are not started by intelmqctl start in analogy to the behavior of widely used initialization systems.
"},{"location":"unsorted/intelmq-3.0-architecture/","title":"Intelmq 3.0 architecture","text":""},{"location":"unsorted/intelmq-3.0-architecture/#idea-list-and-architecture-of-intelmq-30","title":"Idea list and architecture of IntelMQ 3.0","text":"
Authors: Aaron Kaplan kaplan@cert.at, Sebastian Wagner wagner@cert.at
"},{"location":"unsorted/intelmq-3.0-architecture/#use-cases","title":"Use-cases","text":"
XXX fill in a complete list of use cases XXX
"},{"location":"unsorted/intelmq-3.0-architecture/#certs","title":"CERTs","text":"
No direct access to networks in constituency.
"},{"location":"unsorted/intelmq-3.0-architecture/#data-collection","title":"Data collection","text":""},{"location":"unsorted/intelmq-3.0-architecture/#distribution-of-information","title":"Distribution of information","text":""},{"location":"unsorted/intelmq-3.0-architecture/#national-cert","title":"National CERT","text":"
Work is based heavily on Geolocation
"},{"location":"unsorted/intelmq-3.0-architecture/#sector-cert","title":"Sector CERT","text":"
Work is based on known constituents, sector information, lists of IP address ranges and domains, company & organisation names.
"},{"location":"unsorted/intelmq-3.0-architecture/#socs-and-nocs","title":"SOCs and NOCs","text":"
Goal is the protection of internal known networks only. Direct access to the networks.
 
Involves collecting information from internal infrastructure, matching IoCs to internal infrastructure, using IoCs for active protection.
"},{"location":"unsorted/intelmq-3.0-architecture/#data-science-and-research","title":"Data science and research","text":""},{"location":"unsorted/intelmq-3.0-architecture/#users","title":"Users","text":"
XXX fill in a complete list of use cases XXX
"},{"location":"unsorted/intelmq-3.0-architecture/#restful-api","title":"RESTful API","text":"
For automation purposes, we will need a typical RESTful API to manage, control, monitor the IntelMQ \"botnet\" and read and set configs. See #1424
"},{"location":"unsorted/intelmq-3.0-architecture/#ux","title":"UX","text":""},{"location":"unsorted/intelmq-3.0-architecture/#devops-sysadmin-perspective","title":"Devops/ Sysadmin perspective","text":""},{"location":"unsorted/intelmq-3.0-architecture/#docker","title":"Docker","text":"
Task: create a setup where each bot MAY run in a docker container
 
Background: It might make sense to  be able to run each bot in a docker container since it fits with a lot of new paradigms in orchestration. With a proper template, each bot running in a docker container could send its logs to some central logger (for example splunk or similar) and the sysadmin/devops teams which are already using these systems for monitoring alerts can properly fit the IntelMQ logs and alerts to their regular daily routine. Docker also allows the sysadmin/devops folks to centrally manage the system.
 
Think about: how do we integrate the pipeline graph?
 
Category: this feature should be OPTIONAL.
"},{"location":"unsorted/intelmq-3.0-architecture/#tutorials-and-vms-dockers","title":"Tutorials and VMs / dockers","text":"
Task: create tutorials with VMs/docker images.
 
Background: We are missing good tutorials (\"playbooks\") on how to run certain workflows via IntelMQ. Ideally, we would offer ready-made VMs/docker images where people who want to try out IntelMQ (and consequently adapt the setup to their own needs). This also helps teachers/presenters who want to demo IntelMQ.
 
Specifically we would like to have:   * how to process shadowserver feeds   * how to process shodan data   * how to process n6 data
 
Think about: shadowserver already created some training material. Build on this.
 
Category: OPTIONAL component, but highly needed.
"},{"location":"unsorted/intelmq-3.0-architecture/#architecture","title":"Architecture","text":""},{"location":"unsorted/intelmq-3.0-architecture/#message-queue","title":"Message queue","text":"
Task: Create a Kafka MQ backend: add Kafka as a replaceable MQ for IntelMQ 3.0
 
Background: IntelMQ 2.0 supports AMQP (RabbitMQ) next to redis as a message queue. Many organisations use Kafka internally. Support connecting to their other work flows.
 
Think about: Using Apache Pulsar
 
Category: SHOULD
"},{"location":"unsorted/intelmq-3.0-architecture/#notification-settings","title":"Notification settings","text":"
Task: Keep notification settings per event: Where to (destination mail/host address), how (protocol, authentication (SSL client certificate), etc), how often/time information (intervals etc.)
 
Background: CERTs (and potentially other groups of users) need to specify where the events should be sent to, how often etc. Currently only destination email addresses can be saved (source.abuse_contact), which is not enough for most use-cases. There exist some custom solutions (e.g. notify boolean at cert.at (to be changed), extra.processing dictionary at BSI), but no least common denominator.
 
See also https://github.com/certtools/intelmq/issues/758
 
Category: this feature should be OPTIONAL but is NEEDED by several users.
"},{"location":"unsorted/intelmq-3.0-architecture/#configuration-parameter-handling-in-bots-and-a-bots-unified-documentation","title":"Configuration parameter handling in Bots and a bot's unified documentation","text":"
Task: Handle bots' configuration parameters by the core, providing type sanitation, checks, default values and documentation.
 
Background: Currently every bot needs to handle these issues itself, but many of these checks could be done centrally in a generic way. At upgrades, new configuration might get introduced and the bots need to provide defaults values although they are available in BOTS. Error handling on parameters must be done for every bot on itself. Documentation is not available to the Bots, not available in BOTS and the Manager. There are 3 places for parameters where the available information is spread: BOTS, Bots.md and the bots' code.
"},{"location":"unsorted/intelmq-3.0-architecture/#automatic-monitoring-management-handling-full-load-situations","title":"Automatic Monitoring & Management: Handling full load situations","text":"
Task: Create a solution to prevent system over-loading (only for Redis).
 
Background: If too much data is ingested, collected or enriched, the system can easily run out of memory. This quickly causes major operation troubles and data loss, needing manual intervention.
 
See also: https://github.com/certtools/intelmq/issues/709
"},{"location":"unsorted/intelmq-3.0-architecture/#making-intelmq-plug-able-and-getting-rid-of-bots","title":"Making intelmq plug-able and getting rid of BOTS","text":"
Task: Allow installation of IntelMQ bots, meaning the deprecation of the centralized BOTS file and a generated documentation.
 
Background: Adapting IntelMQ to specific needs also means the development of specific bots which might not part of the public repository. Adding them to an existing IntelMQ installation is currently only possible by cloning the repository and adding the code there, not by just providing/installing the required code (because of BOTS and central documentation).
 
See also https://github.com/certtools/intelmq/issues/972
"},{"location":"unsorted/intelmq-3.0-architecture/#exposing-a-plug-in-or-hooking-api","title":"Exposing a plug-in or hooking API","text":"
Task: Provide an hooking API for the core classes.
 
Background: Adapting IntelMQ to specific can require adaptions in the Core classes' code. Instead of making the changes/extensions in the core itself, we can provide a hook system allowing to call (or replace?) functions at specific steps. For example custom monitoring.
"},{"location":"unsorted/intelmq-3.0-architecture/#grouping-of-events","title":"Grouping of events","text":"
Task: Provide possibilities to assign an event to a group of events.
 
Background: Several IoCs part of one MISP Event. Grouping of similar events to one group for outputs (e.g. one CSV file per Network).
 
See also: https://github.com/certtools/intelmq/issues/751
"},{"location":"unsorted/intelmq-3.0-architecture/#data-format-multiple-values","title":"Data Format: Multiple values","text":"
Task: Allow multiple values for (some) fields in the data format.
 
Background: In some cases one value per field is not enough, for example for Domain -> IP address lookups. Other formats like IDEA and n6 support this.
 
See also: https://github.com/certtools/intelmq/issues/543 https://github.com/certtools/intelmq/issues/373
"},{"location":"unsorted/intelmqctl-more/","title":"Intelmqctl more","text":""},{"location":"unsorted/intelmqctl-more/#command-line-interface-intelmqctl","title":"Command-line interface: intelmqctl","text":"
Syntax see intelmqctl -h
 
     
  • Starting a bot: intelmqctl start bot-id
    •  
  • Stopping a bot: intelmqctl stop bot-id
    •  
  • Reloading a bot: intelmqctl reload bot-id
    •  
  • Restarting a bot: intelmqctl restart bot-id
    •  
  • Get status of a bot: intelmqctl status bot-id
    •  
  • Run a bot directly for debugging purpose and temporarily leverage the logging level to DEBUG: intelmqctl run bot-id
    •  
  • Get a pdb (or ipdb if installed) live console.   intelmqctl run bot-id console
    •  
  • See the message that waits in the input queue.   intelmqctl run bot-id message get
    •  
  • See additional help for further explanation.   intelmqctl run bot-id --help
    •  
  • Starting the botnet (all bots): intelmqctl start
    •  
  • Starting a group of bots: intelmqctl start --group experts
    •  
  • Get a list of all configured bots: intelmqctl list bots
    •  
  • Get a list of all queues: intelmqctl list queues If -q is given, only queues with more than one item are listed.
    •  
  • Get a list of all queues and status of the bots:   intelmqctl list queues-and-status
    •  
  • Clear a queue: intelmqctl clear queue-id
    •  
  • Get logs of a bot: intelmqctl log bot-id number-of-lines log-level   Reads the last lines from bot log. Log level should be one of DEBUG, INFO, ERROR or CRITICAL. Default is INFO. Number   of lines defaults to 10, -1 gives all. Result can be longer due to our logging format!
    •  
  • Upgrade from a previous version: intelmqctl upgrade-config Make a backup of your configuration first, also including   bot's configuration files.
    •  
"},{"location":"unsorted/intelmqctl-more/#reloading","title":"Reloading","text":"
Whilst restart is a mere stop & start, performing intelmqctl reload <bot_id> will not stop the bot, permitting it to keep the state: the same common behavior as for ( Linux) daemons. It will initialize again (including reading all configuration again) after the current action is finished. Also, the rate limit/sleep is continued (with the new time) and not interrupted like with the restart command. So if you have a collector with a rate limit of 24 h, the reload does not trigger a new fetching of the source at the time of the reload, but just 24 h after the last run -- with the new configuration. Which state the bots are keeping depends on the bots of course.
"},{"location":"unsorted/intelmqctl-more/#forcing-reset-pipeline-and-cache-be-careful","title":"Forcing reset pipeline and cache (be careful)","text":"
If you are using the default broker (Redis), in some test situations you may need to quickly clear all pipelines and caches. Use the following procedure:
 
redis-cli FLUSHDB\nredis-cli FLUSHALL\n
"},{"location":"unsorted/intelmqctl-more/#management","title":"Management","text":"
IntelMQ has a modular structure consisting of bots. There are four types of bots:
 
     
  • collector bots retrieve data from internal or external sources, the output are *   reports* consisting of many individual data sets / log lines.
    •  
  • parser bots parse the (report) data by splitting it into individual events (log lines) and giving them a defined   structure, see also /dev/data-format for the list of fields an event may be split up into.
    •  
  • expert bots enrich the existing events by e.g. lookup up information such as DNS reverse records, geographic   location information (country code) or abuse contacts for an IP address or domain name.
    •  
  • output bots write events to files, databases, (REST)-APIs or any other data sink that you might want to write to.
    •  
 
Each bot has one source queue (except collectors) and can have multiple destination queues (except outputs). But multiple bots can write to the same pipeline (queue), resulting in multiple inputs for the next bot.
 
Every bot runs in a separate process. A bot is identifiable by a bot id.
 
Currently only one instance (i.e. with the same bot id) of a bot can run at the same time. Concepts for multiprocessing are being discussed, see this issue: Multiprocessing per queue is not supported #186 <186>. Currently you can run multiple processes of the same bot ( with different bot ids) in parallel.
 
Example: multiple gethostbyname bots (with different bot ids) may run in parallel, with the same input queue and sending to the same output queue. Note that the bot providing the input queue must have the load_balance option set to true.
"},{"location":"user/abuse-contacts/","title":"Abuse Contacts","text":""},{"location":"user/abuse-contacts/#abuse-contact-look-ups","title":"Abuse-contact look-ups","text":"
The right decision whom to contact about a specific incident is vital to get the incident resolved as quick as possible. Different types of events may required different abuse-contact to be selected. For example, issues about a device, e.g. a vulnerability in the operating system or an application, is better sent to the hoster which can inform the server administrator. For website-related issues, like defacements or phishing, the domain owner (maintaining the content of the website) could be the better and more direct contact. Additionally, different CERT's have different approaches and different contact databases. Multiple information sources have different information, and some sources are more accurate than others. IntelMQ can query multiple sources of abuse-contacts and combine them. Internal databases, like a Constituency Portal provide high-quality and first-hand contact information. The RIPE document Sources of Abuse Contact Information for Abuse Handlers contains a good summary of the complex of themes.
"},{"location":"user/abuse-contacts/#sources-for-abuse-contacts","title":"Sources for abuse-contacts","text":"
All these bots add the queried contacts to the IntelMQ events in the field source.abuse_contact if not state otherwise in the documentation.
"},{"location":"user/abuse-contacts/#sources-for-domain-based-abuse-contacts","title":"Sources for domain-based abuse-contacts","text":"
These bots are suitable for domain-based abuse-contact look-ups.
 
     
  • intelmq.bots.experts.rdap.expert expert queries private and public RDAP servers for source.fqdn and add the contact information to the event as source.abuse_contact.
    •  
  • intelmq.bots.experts.trusted_introducer_lookup.expert expert queries a locally cached Trusted Introducer team directory for the TLD or domain (first match) of source.fqdn.
    •  
"},{"location":"user/abuse-contacts/#sources-for-ip-address-based-abuse-contacts","title":"Sources for IP address-based abuse-contacts","text":"
These bots are suitable for IP address and ASN based abuse-contact look-ups.
 
     
  • intelmq.bots.experts.abusix.expert expert queries the online Abusix service.
    •  
  • intelmq.bots.experts.do_portal.expert expert queries an instance of the do-portal software (deprecated).
    •  
  • intelmq.bots.experts.tuency.expert expert queries an instance of the tuency Constituency Portal for the IP address. The Portal also takes into account any notification rules, which are saved   additionally in the event.
    •  
  • intelmq.bots.experts.ripe.expert expert queries the online RIPE database for IP-Address and AS contacts.
    •  
  • intelmq.bots.experts.trusted_introducer_lookup.expert expert queries a locally   cached Trusted Introducer team directory   for the Autonomous system source.asn.
    •  
"},{"location":"user/abuse-contacts/#generic-sources-for-abuse-contacts","title":"Generic sources for abuse-contacts","text":"
     
  • intelmq.bots.experts.generic_db_lookup.expert expert for local data sources, like   database tables mapping ASNs to abuse-contact or Country Codes to abuse-contact.
    •  
  • intelmq.bots.experts.uwhoisd.expert expert for fetching whois-data, not extracting   abuse-contact information
    •  
"},{"location":"user/abuse-contacts/#helpful-other-bots-for-pre-processing","title":"Helpful other bots for pre-processing","text":"
     
  • intelmq.bots.experts.asn_lookup.expert queries locally cached database to lookup ASN.
    •  
  • intelmq.bots.experts.cymru_whois.expert to lookup ASN, Geolocation, and BGP prefix   for *.ip.
    •  
  • intelmq.bots.experts.domain_suffix.expert to lookup the public suffix of the domain   in *.fqdn.
    •  
  • intelmq.bots.experts.format_field.expert
    •  
  • intelmq.bots.experts.gethostbyname.expert resolve *.ip from *.fqdn.
    •  
  • intelmq.bots.experts.maxmind_geoip.expert to lookup Geolocation information for *.ip   .
    •  
  • intelmq.bots.experts.reverse_dns.expert to resolve *.reverse_dns from *.ip.
    •  
  • intelmq.bots.experts.ripe.expert to lookup *.asn and Geolocation information   for *.ip.
    •  
  • intelmq.bots.experts.tor_nodes.expert for filtering out TOR nodes.
    •  
  • intelmq.bots.experts.url2fqdn.expert to extract *.fqdn/*.ip from *.url.
    •  
"},{"location":"user/abuse-contacts/#combining-the-lookup-approaches","title":"Combining the lookup approaches","text":"
In order to get the best contact, it may be necessary to combine multiple abuse-contact sources. IntelMQ's modularity provides methods to arrange and configure the bots as needed. Among others, the following bots can help in getting the best result:
 
     
  • intelmq.bots.experts.filter.expert Your lookup process may be different for different types of data. E.g. website-related issues may be better addressed at the domain owner and device-related issues may be better addressed to the hosting provider.
    •  
  • intelmq.bots.experts.modify.expert Allows you to set values based on filter and also format values based on the value of other fields.
    •  
  • intelmq.bots.experts.sieve.expert Very powerful expert which allows filtering, routing (to different subsequent bots) based on if-expressions . It support set-operations (field value is in list) as well as sub-network operations for IP address networks in CIDR notation for the expression-part. You can as well set the abuse-contact directly.
    •  
"},{"location":"user/api/","title":"API","text":""},{"location":"user/api/#using-intelmq-api","title":"Using IntelMQ API","text":"
Bug
 
This section of the documentation is currently incomplete and will be added later.
"},{"location":"user/api/#usage-from-programs","title":"Usage from programs","text":"
The IntelMQ API can also be used from programs, not just browsers. To do so, first send a POST-Request with JSON-formatted data to http://localhost/intelmq/v1/api/login/
 
{\n    \"username\": \"$your_username\",\n    \"password\": \"$your_password\"\n}\n
 
With valid credentials, the JSON-formatted response contains the login_token. This token can be used like an API key in the Authorization header for the next API calls:
 
Authorization: $login_token\n
 
Here is a full example using curl:
 
     
  1.  
    Authentication step:    
    curl --location --request POST \"http://localhost/intelmq/v1/api/login/\" \\\n     --header \"Content-Type: application/x-www-form-urlencoded\" \\\n     --data-urlencode \"username=$username\"\\\n     --data-urlencode \"password=$password\"\n
     
    {\"login_token\":\"68b329da9893e34099c7d8ad5cb9c940\",\"username\":\"$username\"}\n
     
    2.  
  2.  
    Using the login token to fetch data:    
    curl --location \"http://localhost/intelmq/v1/api/version\" \\\n     --header \"Authorization: 68b329da9893e34099c7d8ad5cb9c940\"\n
     
    {\"intelmq\":\"3.0.0rc1\",\"intelmq-manager\":\"2.3.1\"}\n
     
    3.  
 
The same approach also works for Ansible, as you can see here:
 
     
  1. https://github.com/schacht-certat/intelmq-vagrant/blob/7082719609c0aafc9324942a8775cf2f8813703d/ansible/tasks/api/00_registerauth.yml#L1-L9
    2.  
  2. https://github.com/schacht-certat/intelmq-vagrant/blob/7082719609c0aafc9324942a8775cf2f8813703d/ansible/tasks/api/02_queuestatus.yml#L1-L5
    3.  
"},{"location":"user/bots/","title":"Bots","text":""},{"location":"user/bots/#bots-inventory","title":"Bots Inventory","text":"
This document contains complete reference of bots implemented by IntelMQ and how to configure them from the users perspective (meaning via IntelMQ Manager). Some of the bots are intended for general use and some of them are for processing particular data sources.
"},{"location":"user/bots/#individual-bot-configuration","title":"Individual Bot Configuration","text":"
Each bot has it's own configuration. The configuration consists of two types of parameters:
 
     
  •  
    Generic parameters that are common to all the bots and need to be set for each bot.
     
    •  
  •  
    Runtime parameters are needed by the bot itself during runtime. Some of these parameters can be inherited from the global configuration (which is applied to all the bots), but can be overridden in the individual bot configuration.
     
    •  
"},{"location":"user/bots/#generic-parameters","title":"Generic Parameters","text":"
These parameters must be set for each bot (at least the required ones).
"},{"location":"user/bots/#id","title":"id","text":"
(required, string) This must be a unique identifier. Commonly it looks something like this: abusech-feodo-tracker-collector. It is safer to avoid using spaces.
"},{"location":"user/bots/#name","title":"name","text":"
(required, string) Human readable name of the bot.
"},{"location":"user/bots/#description","title":"description","text":"
(required, string) The description of the bot.
"},{"location":"user/bots/#module","title":"module","text":"
(required, string) The executable (should be in PATH environment variable) which will be started.
"},{"location":"user/bots/#group","title":"group","text":"
(optional, string) The group of the bot. Can be Collector, Parser, Expert or Output. Only used for visualization by other tools.
"},{"location":"user/bots/#enabled","title":"enabled","text":"
(optional, boolean) Whether the bot will start when the whole botnet is started. You can still start a disabled bot explicitly. Defaults to true.
"},{"location":"user/bots/#run_mode","title":"run_mode","text":"
(optional, string) There are two run modes, continuous or scheduled. In the first case, the bot will be running forever until stopped or exits because of errors (depending on the configuration). In the latter case, the bot will stop after one successful run. This is especially useful when scheduling bots via cron or systemd. Check Configuration section for more details. Defaults to continuous.
"},{"location":"user/bots/#http-parameters","title":"HTTP Parameters","text":"
Common HTTP runtime parameters used in multiple bots.
"},{"location":"user/bots/#http_timeout_sec","title":"http_timeout_sec","text":"
(optional, float) A tuple of floats or only one float describing the timeout (seconds) of the HTTP connection. Can be a tuple of two floats (read and connect timeout) or just one float (applies for both timeouts). See also https://requests.readthedocs.io/en/master/user/advanced/#timeouts. Defaults to 30.
"},{"location":"user/bots/#http_timeout_max_tries","title":"http_timeout_max_tries","text":"
(optional, integer) An integer depicting how many times a connection is retried, when a timeout occurred. Defaults to 3.
"},{"location":"user/bots/#http_username","title":"http_username","text":"
(optional, string) Username for basic HTTP authentication.
"},{"location":"user/bots/#http_password","title":"http_password","text":"
(optional, string) Password for basic HTTP authentication.
"},{"location":"user/bots/#http_proxy","title":"http_proxy","text":"
(optional, string) Proxy to use for HTTP.
"},{"location":"user/bots/#https_proxy","title":"https_proxy","text":"
(optional, string) Proxy to use for HTTPS.
"},{"location":"user/bots/#http_user_agent","title":"http_user_agent","text":"
(optional, string) User-Agent to be used for HTTP requests.
"},{"location":"user/bots/#http_verify_cert","title":"http_verify_cert","text":"
(optional, boolean/string) Path to trusted CA bundle or directory, false to ignore verifying SSL certificates, or true to verify SSL certificates. Defaults to true.
"},{"location":"user/bots/#ssl_client_certificate","title":"ssl_client_certificate","text":"
(optional, string) Path to client certificate to use for TLS connections.
"},{"location":"user/bots/#ssl_ca_certificate","title":"ssl_ca_certificate","text":"
(optional, string) Path to trusted CA certificate. Only used by some bots.
"},{"location":"user/bots/#cache-parameters","title":"Cache Parameters","text":"
Common Redis cache runtime parameters used in multiple bots (mainly lookup experts).
"},{"location":"user/bots/#redis_cache_host","title":"redis_cache_host","text":"
(required, string) Hostname of the Redis database.
"},{"location":"user/bots/#redis_cache_port","title":"redis_cache_port","text":"
(required, string) Port of the Redis database.
"},{"location":"user/bots/#redis_cache_db","title":"redis_cache_db","text":"
(required, integer) Database number.
"},{"location":"user/bots/#redis_cache_ttl","title":"redis_cache_ttl","text":"
(required, integer) TTL used for caching.
"},{"location":"user/bots/#redis_cache_password","title":"redis_cache_password","text":"
(optional, string) Password for the Redis database.
"},{"location":"user/bots/#collector-bots","title":"Collector Bots","text":"
Multihreading is disabled for all Collectors, as this would lead to duplicated data.
"},{"location":"user/bots/#feed-parameters","title":"Feed Parameters","text":"
These runtime parameters must be set for each collector bot (at least the required ones).
"},{"location":"user/bots/#name_1","title":"name","text":"
(required, string) Name of the feed (feed.name).
"},{"location":"user/bots/#accuracy","title":"accuracy","text":"
(optional, float) Accuracy of the data from the feed (feed.accuracy).
"},{"location":"user/bots/#code","title":"code","text":"
(optional, string) Code for the feed (feed.code).
"},{"location":"user/bots/#documentation","title":"documentation","text":"
(optional, string) Link to documentation for the feed (feed.documentation).
"},{"location":"user/bots/#provider","title":"provider","text":"
(optional, string) Name of the provider of the feed (feed.provider).
"},{"location":"user/bots/#rate_limit","title":"rate_limit","text":"
(optional, integer) Time interval (in seconds) between fetching data if applicable. Defaults to 0.
"},{"location":"user/bots/#alien-vault-otx","title":"Alien Vault OTX","text":"
Collects report messages from Alien Vault OTX.
 
Module: intelmq.bots.collectors.alienvault_otx.collector
 
Requirements
 
Install the library from GitHub, as there is no package in PyPi:
 
pip3 install -r intelmq/bots/collectors/alienvault_otx/REQUIREMENTS.txt\n
 
Parameters (also expects feed parameters):
 
api_key
 
(required, string) API Key
 
modified_pulses_only
 
(optional, boolean) Whether to get only modified pulses instead of all. Defaults to false.
 
interval
 
(optional, integer) When modified_pulses_only is set, define the time in hours (integer value) to get modified pulses since then. Defaults to 24 (hours).
"},{"location":"user/bots/#amqp","title":"AMQP","text":"
This bot collects data from (remote) AMQP servers, for both IntelMQ as well as external data. Currently only fetching from a queue is supported can be extended in the future. Messages will be acknowledge at AMQP after it is sent to the pipeline. Requires the pika library, minimum version 1.0.0.
 
Module: intelmq.bots.collectors.amqp.collector_amqp
 
Parameters (also expects feed parameters):
 
connection_host
 
(optional, string) Hostname of the AMQP server. Defaults to 127.0.0.1.
 
connection_port
 
(optional, integer) Port of the AMQP server. Defaults to 5672.
 
connection_attempts
 
(optional, integer) The number of connection attempts to the defined server. Defaults to 3.
 
connection_heartbeat
 
(optional, integer) Heartbeat to server (seconds). Defaults to 3600.
 
connection_vhost
 
(optional, string) Virtual host to connect, on an HTTP(S) connection would be . 
expect_intelmq_message
 
(optional, boolean) This parameter denotes whether the the data is from IntelMQ or not. If true, then the data can be any Report or Event and will be passed to the next bot as is. Otherwise a new Report is created with the raw data. Defaults to false.
 
queue_name
 
(optional, string) The name of the queue to fetch the data from.
 
username
 
(optional, string) Username for authentication to the AMQP server.
 
password
 
(optional, string) Password for authentication to the AMQP server.
 
use_ssl
 
(optional, boolean) Use of TLS for the connection. Make sure to also set the correct port. Defaults to false.
"},{"location":"user/bots/#api","title":"API","text":"
This bot collects data from HTTP or Socket REST API. The API is available at /intelmq/push when the HTTP interface is used. Requires the tornado library.
 
Module: intelmq.bots.collectors.api.collector
 
Parameters (also expects feed parameters):
 
port
 
(optional, integer) The local port at which the API is available. Defaults to 5000.
 
use_socket
 
(optional, boolean) If true, the socket will be opened at the location given with socket_path. Defaults to false.
 
socket_path
 
(optional, string) Location of the socket. Defaults to /tmp/imq_api_default_socket.
 
socket_perms
 
(optional, octal integer) Unix permissions to grant to the socket file. Default: 600
 
socket_group
 
(optional, string) Name of group to change group ownership of socket file to.
"},{"location":"user/bots/#generic-url-fetcher","title":"Generic URL Fetcher","text":"
This bot collects data from remote hosts using HTTP protocol. If the HTTP response' status code is not 2xx, this is treated as error. In Debug logging level, the request's and response's headers and body are logged for further inspection.
 
Module: intelmq.bots.collectors.http.collector_http
 
Parameters (also expects feed parameters and HTTP parameters):
 
http_url
 
(required, string) Location of the resource to download.
 
http_url_formatting
 
(optional, boolean/object) When true, {time[format]} will be replaced by the current time in local timezone formatted by the given format. E.g. if the URL is http://localhost/{time[%Y]}, then the resulting URL is http://localhost/2019 for the year 2019. ( Python's Format Specification Mini-Language is used for this.). You may use a JSON specifying time-delta parameters to shift the current time accordingly. For example use days: -1 for the yesterday's date; the URL http://localhost/{time[%Y-%m-%d]} will get translated to http://localhost/2018-12-31 for the 1st Jan of 2019. Defaults to false.
 
extract_files
 
(optional, boolean/array of strings) If true, the retrieved (compressed) file or archived will be uncompressed/unpacked and the files are extracted. If the parameter is a list of strings, only the files matching the filenames are extracted. Extraction handles gzipped files and both compressed and uncompressed tar-archives as well as zip archives. For extracted files, every extracted file is sent in it's own report. Every report has a field named extra.file_name with the file name in the archive the content was extracted from. Defaults to false.
 
verify_pgp_signatures
 
(optional, boolean) When true, signature file is downloaded and report file is checked. On error (missing signature, mismatch, ...), the error is logged and the report is not processed. Public key has to be imported in local keyring. This requires the python-gnupg library. Defaults to false.
 
signature_url
 
(optional, string) Location of the signature file for the downloaded content.
 
signature_url_formatting
 
(optional, boolean/object) Same as http_url_formatting. Defaults to false.
 
gpg_keyring
 
(optional, string) If specified, the string represents path to keyring file. Otherwise the PGP keyring file of the current intelmq user is used.
"},{"location":"user/bots/#generic-url-stream-fetcher","title":"Generic URL Stream Fetcher","text":"
Opens a streaming connection to the URL and collects the received lines.
 
If the stream is interrupted, the connection will be aborted using the timeout parameter. No error will be logged if the number of consecutive connection fails does not reach the parameter error_max_retries. Instead of errors, an INFO message is logged. This is a measurement against too frequent ERROR logging messages. The consecutive connection fails are reset if a data line has been successfully transferred. If the consecutive connection fails reaches the parameter error_max_retries, an exception will be thrown and rate_limit applies, if not null.
 
Module: intelmq.bots.collectors.http.collector_http_stream
 
Parameters (also expects feed parameters and HTTP parameters):
 
Uses the same parameters as Generic URL Fetcher. The parameter http_timeout_max_tries is of no use in this collector.
 
strip_lines
 
(optional, boolean) Whether the single lines should be stripped (removing whitespace from the beginning and the end of the line) or not. Defaults to true.
"},{"location":"user/bots/#generic-mail-url-fetcher","title":"Generic Mail URL Fetcher","text":"
Extracts URLs from e-mail messages and downloads the content from the URLs. It uses the imbox library.
 
The resulting reports contain the following special fields:
 
     
  • feed.url: The URL the data was downloaded from.
    •  
  • extra.email_date: The content of the email's Date header.
    •  
  • extra.email_subject: The subject of the email.
    •  
  • extra.email_from: The email's from address.
    •  
  • extra.email_message_id: The email's message ID.
    •  
  • extra.file_name: The file name of the downloaded file (extracted from the HTTP Response Headers if possible).
    •  
 
The fields can be used by parsers to identify the feed and are not automatically passed on to events.
 
Chunking
 
For line-based inputs the bot can split up large reports into smaller chunks. This is particularly important for setups that use Redis as a message queue which has a per-message size limitation of 512 MB. To configure chunking, set chunk_size to a value in bytes. chunk_replicate_header determines whether the header line should be repeated for each chunk that is passed on to a parser bot. Specifically, to configure a large file input to work around Redis size limitation set chunk_size to something like 384000000 (~384 MB).
 
Module: intelmq.bots.collectors.mail.collector_mail_url
 
Parameters (also expects feed parameters and HTTP parameters):
 
mail_host
 
(required, string) Hostname of the mail server.
 
mail_port
 
(optional, integer) IMAP server port: 143 without TLS, 993 with TLS. Defaults to 143.
 
mail_user
 
(required, string) Username of the email account.
 
mail_password
 
(required, string) Password associated with the user account.
 
mail_ssl
 
(optional, boolean) Whether the mail server uses TLS or not. Defaults to true.
 
mail_starttls
 
(optional, boolean) Whether the mail server uses STARTTLS or not. Defaults to false.
 
folder
 
(optional, string) Folder in which to look for e-mail messages. Defaults to INBOX.
 
subject_regex
 
(optional, string) Regular expression to look for in the e-mail subject.
 
url_regex
 
(optional, string) Regular expression of the feed URL to look for in the e-mail body.
 
sent_from
 
(optional, string) Filter messages by the sender.
 
sent_to
 
(optional, string) Filter messages by the recipient.
 
ssl_ca_certificate
 
(optional, string) Path to trusted CA certificate. Applies only to IMAP connections, not HTTP. If the provided certificate is not found, the IMAP connection will fail on handshake. Defaults to no certificate.
"},{"location":"user/bots/#generic-mail-attachment-fetcher","title":"Generic Mail Attachment Fetcher","text":"
This bot collects messages from mailboxes and downloads the attachments. It uses the imbox library.
 
The resulting reports contains the following special fields:
 
     
  • extra.email_date: The content of the email's Date header
    •  
  • extra.email_subject: The subject of the email
    •  
  • extra.email_from: The email's from address
    •  
  • extra.email_message_id: The email's message ID
    •  
  • extra.file_name: The file name of the attachment or the file name in the attached archive if attachment is to   uncompress.
    •  
 
The fields can be used by parsers to identify the feed and are not automatically passed on to events.
 
Module: intelmq.bots.collectors.mail.collector_mail_attach
 
Parameters (also expects feed parameters):
 
mail_host
 
(required, string) Hostname of the mail server.
 
mail_port
 
(optional, integer) IMAP server port: 143 without TLS, 993 with TLS. Default depends on SSL setting.
 
mail_user
 
(required, string) Username of the email account.
 
mail_password
 
(required, string) Password associated with the user account.
 
mail_ssl
 
(optional, boolean) Whether the mail server uses TLS or not. Defaults to true.
 
mail_starttls
 
(optional, boolean) Whether to use STARTTLS before authenticating to the server. Defaults to false.
 
folder
 
(optional, string) Folder in which to look for e-mail messages. Defaults to INBOX.
 
subject_regex
 
(optional, string) Regular expression to look for in the e-mail subject.
 
attach_regex
 
(optional, string) All attachments which match this regular expression will be processed. Defaults to csv.zip.
 
extract_files
 
(optional, boolean) Whether to extract compress files from the attachment. Defaults to true.
 
sent_from
 
(optional, string) Only process messages sent from this address. Defaults to null (any sender).
 
sent_to
 
(optional, string) Only process messages sent to this address. Defaults to null (any recipient).
 
ssl_ca_certificate
 
(optional, string) Path to trusted CA certificate. Applies only to IMAP connections, not HTTP. If the provided certificate is not found, the IMAP connection will fail on handshake. By default, no certificate is used.
"},{"location":"user/bots/#generic-mail-body-fetcher","title":"Generic Mail Body Fetcher","text":"
This bot collect messages from mailboxes, forwards the bodies as reports. Each non-empty body with the matching content type is sent as individual report.
 
The resulting reports contains the following special fields:
 
     
  • extra.email_date: The content of the email's Date header
    •  
  • extra.email_subject: The subject of the email
    •  
  • extra.email_from: The email's from address
    •  
  • extra.email_message_id: The email's message ID
    •  
 
Module: intelmq.bots.collectors.mail.collector_mail_body
 
Parameters (also expects feed parameters):
 
mail_host
 
(required, string) Hostname of the mail server.
 
mail_port
 
(optional, integer) IMAP server port: 143 without TLS, 993 with TLS. Defaults to 143.
 
mail_user
 
(required, string) Username of the email account.
 
mail_password
 
(required, string) Password associated with the user account.
 
mail_ssl
 
(optional, boolean) Whether the mail server uses TLS or not. Defaults to true.
 
folder
 
(optional, string) Folder in which to look for e-mail messages. Defaults to INBOX.
 
subject_regex
 
(optional, string) Regular expression to look for in the e-mail subject.
 
url_regex
 
(optional, string) Regular expression of the feed URL to look for in the e-mail body.
 
sent_from
 
(optional, string) Filter messages by the sender.
 
sent_to
 
(optional, string) Filter messages by the recipient.
 
ssl_ca_certificate
 
(optional, string) Path to trusted CA certificate. Applies only to IMAP connections, not HTTP. If the provided certificate is not found, the IMAP connection will fail on handshake. Defaults to no certificate.
 
content_types
 
(optional, boolean/array of strings) Which bodies to use based on the content_type. Defaults to true (same as ['html', 'plain']) for all:
 
     
  • string with comma separated values, e.g. ['html', 'plain']
    •  
  • true, false, null: Same as default value - string, e.g. plain
    •  
"},{"location":"user/bots/#github-api","title":"Github API","text":"
Collects files matched by regular expression from GitHub repository via the GitHub API. Optionally with GitHub credentials, which are used as the Basic HTTP authentication.
 
Workflow
 
The optional authentication parameters provide a high limit of the GitHub API requests. With the git hub user authentication, the requests are rate limited to 5000 per hour, otherwise to 60 requests per hour.
 
The collector recursively searches for regex-defined files in the provided repository. Additionally it adds extra file metadata defined by the extra_fields.
 
The bot always sets the url, from which downloaded the file, as feed.url.
 
Module: intelmq.bots.collectors.github_api.collector_github_contents_api
 
Parameters (also expects feed parameters):
 
personal_access_token
 
(required, string) GitHub account personal access token GitHub documentation: Creating a personal access token
 
repository
 
(required, string) GitHub target repository (<USER>/<REPOSITORY>)
 
regex
 
(optional, string) Valid regular expression of target files within the repository. Defaults to .*.json.
 
extra_fields
 
(optional, array of strings) Comma-separated list of extra fields from GitHub contents API.
"},{"location":"user/bots/#file","title":"File","text":"
This bot is capable of reading files from the local file-system. This is handy for testing purposes, or when you need to react to spontaneous events. In combination with the Generic CSV parser this should work great.
 
The resulting reports contains the following special fields:
 
     
  • feed.url: The URI using the file:// scheme and localhost, with the full path to the processed file.
    •  
  • extra.file_name: The file name (without path) of the processed file.
    •  
 
Chunking
 
Additionally, for line-based inputs the bot can split up large reports into smaller chunks.
 
This is particularly important for setups that use Redis as a message queue which has a per-message size limitation of 512 MB.
 
To configure chunking, set chunk_size to a value in bytes. chunk_replicate_header determines whether the header line should be repeated for each chunk that is passed on to a parser bot.
 
Specifically, to configure a large file input to work around Redis' size limitation set chunk_size to something like 384000, i.e., ~384 MB.
 
Workflow
 
The bot loops over all files in path and tests if their file name matches *postfix, e.g. *.csv. If yes, the file will be read and inserted into the queue.
 
If delete_file is set, the file will be deleted after processing. If deletion is not possible, the bot will stop.
 
To prevent data loss, the bot also stops when no postfix is set and delete_file was set. This cannot be overridden.
 
The bot always sets the file name as feed.url.
 
Module: intelmq.bots.collectors.file.collector_file
 
Parameters (also expects feed parameters):
 
path
 
(required, string) Path to file.
 
postfix
 
(required, string) The postfix (file ending) of the files to look for. For example [.csv].
 
delete_file
 
(optional, boolean) Whether to delete the file after reading. Defaults to false.
"},{"location":"user/bots/#fireeye","title":"FireEye","text":"
This bot is capable of collecting hashes and URLs from a FireEye MAS appliance.
 
The Python library xmltodict is required to run this bot.
 
Workflow
 
The bot collects all alerts which occurred during specified duration. After this we make a second call and check if there is additional information like domains and hashes available. After collecting the openioc data we send this information to the Fireeye parser.
 
Module: intelmq.bots.collectors.fireeye.collector_fireeye
 
Parameters (also expects feed parameters):
 
host
 
(required, string) DNS name of the target appliance.
 
request_duration
 
(required, string) Allowed values: 24_hours or 48_hours. Length of the query in past eg. collect alerts from last 24hours/48hours.
 
http_username
 
(required, string) Password for authentication.
 
http_password
 
(required, string) Username for authentication.
"},{"location":"user/bots/#kafka","title":"Kafka","text":"
Requires the kafka python library.
 
Module: intelmq.bots.collectors.kafka.collector
 
Parameters (also expects feed parameters):
 
topic
 
(required, string) Kafka topic the collector should get messages from.
 
bootstrap_servers
 
(required, string) Kafka server(s) and port the collector should connect to. Defaults to localhost:9092
 
ssl_check_hostname
 
(optional, boolean) Whether to verify TLS certificates. Defaults to true.
 
ssl_client_certificate
 
(optional, string) Path to client certificate to use for TLS connections.
 
ssl_ca_certificate
 
(optional, string) Path to trusted CA certificate.
"},{"location":"user/bots/#misp-generic","title":"MISP Generic","text":"
Collects messages from MISP, a malware information sharing platform server.
 
Workflow
 
This collector will search for events on a MISP server that have a [to_process] tag attached to them (see the [misp_tag_to_process] parameter) and collect them for processing by IntelMQ. Once the MISP event has been processed the [to_process] tag is removed from the MISP event and a [processed] tag is then attached (see the [misp_tag_processed] parameter).
 
NB. The MISP tags must be configured to be 'exportable' otherwise they will not be retrieved by the collector.
 
Module: intelmq.bots.collectors.misp.collector
 
Parameters (also expects feed parameters):
 
misp_url
 
(required, string) URL of MISP server (with trailing '/').
 
misp_key
 
(required, string) MISP Authkey.
 
misp_tag_to_process
 
(required, string) MISP tag for events to be processed.
 
misp_tag_processed
 
(optional, string) MISP tag for processed events.
 
http_verify_cert
 
(optional, boolean) Verify the TLS certificate of the server. Defaults to true.
"},{"location":"user/bots/#request-tracker","title":"Request Tracker","text":"
Request Tracker Collector fetches attachments from an RTIR instance.
 
This rt bot will connect to RT and inspect the given search_queue for tickets matching all criteria in search_*, Any matches will be inspected. For each match, all (RT-) attachments of the matching RT tickets are iterated over and within this loop, the first matching filename in the attachment is processed. If none of the filename matches apply, the contents of the first (RT-) \"history\" item is matched against the regular expression for the URL (url_regex).
 
The parameter http_timeout_max_tries is of no use in this collector.
 
Search
 
The parameters prefixed with search_ allow configuring the ticket search.
 
Empty strings and null as value for search parameters are ignored.
 
File downloads
 
Attachments can be optionally unzipped, remote files are downloaded with the http_* settings applied.
 
If url_regex or attachment_regex are empty strings, false or null, they are ignored.
 
Ticket processing
 
Optionally, the RT bot can \"take\" RT tickets (i.e. the user is assigned this ticket now) and/or the status can be changed (leave set_status empty in case you don't want to change the status). Please note however that you MUST do one of the following: either \"take\" the ticket or set the status (set_status). Otherwise, the search will find the ticket every time and get stuck in an endless loop.
 
In case a resource needs to be fetched and this resource is permanently not available (status code is 4xx), the ticket status will be set according to the configuration to avoid processing the ticket over and over. For temporary failures the status is not modified, instead the ticket will be skipped in this run.
 
Time search
 
To find only tickets newer than a given absolute or relative time, you can use the search_not_older_than parameter. Absolute time specification can be anything parseable by dateutil, best use a ISO format.
 
Relative must be in this format: [NUMBER] [TIMESPAN]s, e.g. 3 days. Timespan can be hour, day, week, month or year. Trailing 's' is supported for all timespans. Relative times are subtracted from the current time directly before the search is performed.
 
The resulting reports contains the following special fields:
 
     
  • rtir_id: The ticket ID
    •  
  • extra.email_subject and extra.ticket_subject: The subject of the ticket
    •  
  • extra.email_from and extra.ticket_requestors: Comma separated list of the ticket's requestor's email addresses.
    •  
  • extra.ticket_owner: The ticket's owner name
    •  
  • extra.ticket_status: The ticket's status
    •  
  • extra.ticket_queue: The ticket's queue
    •  
  • extra.file_name: The name of the extracted file, the name of the downloaded file or the attachments' filename without .gz postfix.
    •  
  • time.observation: The creation time of the ticket or attachment.
    •  
 
Requirements
 
You need the rt-library >= 1.9 and < 3.0 from from nic.cz, available via pypi: pip3 install rt<3
 
Module: intelmq.bots.collectors.rt.collector_rt
 
Parameters (also expects feed parameters and HTTP parameters):
 
extract_attachment
 
(optional, boolean/array of strings) See documentation of the Generic URL Fetcher parameter extract_files for more details.
 
extract_download
 
(optional, boolean/array of strings) See documentation of the Generic URL Fetcher parameter extract_files for more details.
 
uri
 
(optional, string) URL of the REST interface of the RT. Defaults to http://localhost/rt/REST/1.0.
 
user
 
(optional, string) RT username. Defaults to intelmq.
 
password
 
(optional, string) RT password. Defaults to password.
 
search_not_older_than
 
(optional, string) Absolute time (use ISO format) or relative time, e.g. 3 days.
 
search_owner
 
(optional, string) Owner of the ticket to search for. Defaults to nobody.
 
search_queue
 
(optional, string) Queue of the ticket to search for. Defaults to Incident Reports.
 
search_requestor
 
(optional, string) E-mail address of the requestor.
 
search_status
 
(optional, string) Status of the ticket to search for. Defaults to new.
 
search_subject_like
 
(optional, string/array of strings) Part of the subject of the ticket to search for. Defaults to \"Report\".
 
search_subject_notlike
 
(optional, string/array of strings) Exclude subject containing given value, use list for multiple excluding values.
 
set_status
 
(optional, string) Status to set the ticket to after processing. Use false or null to keep current status. Defaults to open.
 
take_ticket
 
(optional, boolean) Whether to take the ticket. Defaults to true.
 
url_regex
 
(optional, string) Regular expression of an URL to search for in the ticket. Defaults to https://dl.shadowserver.org/[a-zA-Z0-9?_-]*.
 
attachment_regex
 
(optional, string) Eegular expression of an attachment in the ticket. Defaults to \\.csv\\.zip$.
"},{"location":"user/bots/#rsync","title":"Rsync","text":"
This bot downloads a file via rsync and then load data from downloaded file. Downloaded file is located in var/lib/bots/rsync_collector.
 
Requires the rsync executable.
 
Module: intelmq.bots.collectors.rsync.collector_rsync
 
Parameters (also expects feed parameters):
 
file
 
(required, string) The filename to process, combined with rsync_path.
 
rsync_path
 
(required, string) Path to the directory of the file. Allowed values are local directory (such as /home/username/) or remote directory (such as <username@remote_host>:/home/username/directory).
 
rsync_file_path_formatting
 
(optional, boolean) Whether the file and rsync_path should be formatted by the given format. E.g. if the path is /path/to_file/{time[%Y]}, then the resulting path is /path/to/file/2023 for the year 2023. (Python's Format Specification Mini-Language <https://docs.python.org/3/library/string.html#formatspec> is used for this.). You may use a JSON specifying time-delta <https://docs.python.org/3/library/datetime.html#datetime.timedelta> parameters to shift the current time accordingly. For example use {\"days\": -1} for the yesterday's date; the path /path/to/file/{time[%Y-%m-%d]} will get translated to \"/path/to/file/2018-12-31\" for the 1st Jan of 2023. Defaults to false.
 
extra_params
 
(optional, array of strings) A list of extra parameters to pass to rsync.
 
private_key
 
(optional, string) Private key to use for rsync authentication.
 
private_key_path
 
(optional, string) Path to private key to use for rsync authentication. Use private_key or private_key_path, not both.
 
strict_host_key_checking
 
(optional, boolean) Whether the host key should be checked. Defaults to false.
 
temp_directory
 
(optional, string) The temporary directory for rsync to use for collected files. Defaults to /opt/intelmq/var/run/{BOT-ID} or /var/run/intelmq/{BOT-ID}.
"},{"location":"user/bots/#shadowserver-reports-api","title":"Shadowserver Reports API","text":"
Connects to the Shadowserver API, requests a list of all the reports for a specific country and processes the ones that are new.
 
The Cache is required to memorize which files have already been processed (TTL needs to be high enough to cover the oldest files available!).
 
The resulting reports contain the following special field:
 
     
  • extra.file_name: The name of the downloaded file, with fixed filename extension.
    •  
 
Module: intelmq.bots.collectors.shadowserver.collector_reports_api
 
Parameters (also expects feed parameters and cache parameters):
 
apikey
 
(required, string) Your Shadowserver API key.
 
secret
 
(required, string) Your Shadowserver API secret.
 
reports
 
(required, string/array of strings) An array of strings (or a list of comma-separated values) of the mailing lists you want to process.
 
types
 
(optional, string/array of strings) An array of strings (or a list of comma-separated values) with the names of report types you want to process. If you leave this empty, all the available reports will be downloaded and processed (i.e. 'scan', 'drones', 'intel', 'sandbox_connection', 'sinkhole_combined'). The possible report types are equivalent to the file names defined the the schema.  Please see the Supported Reports of the Shadowserver parser for details.
 
Sample configuration
 
  shadowserver-collector:\n    description: Our bot responsible for getting reports from Shadowserver\n    enabled: true\n    group: Collector\n    module: intelmq.bots.collectors.shadowserver.collector_reports_api\n    name: Shadowserver_Collector\n    parameters:\n      destination_queues:\n        _default: [shadowserver-parser-queue]\n      file_format: csv\n      api_key: \"$API_KEY_received_from_the_shadowserver_foundation\"\n      secret: \"$SECRET_received_from_the_shadowserver_foundation\"\n    run_mode: continuous\n
"},{"location":"user/bots/#shodan-stream","title":"Shodan Stream","text":"
Queries the Shodan Streaming API.
 
Requires the shodan library to be installed:
 
     
  •  
    https://github.com/achillean/shodan-python/
     
    •  
  •  
    https://pypi.org/project/shodan/
     
    •  
 
Module: intelmq.bots.collectors.shodan.collector_stream
 
Parameters (also expects feed parameters and HTTP parameters):
 
Only the proxy is used (requires shodan-python > 1.8.1). Certificate is always verified.
 
countries
 
() A list of countries to query for. If it is a string, it will be spit by ,.
 
alert
 
() Alert ID from monitor.shodan.io.
 
If the stream is interrupted, the connection will be aborted using the timeout parameter. No error will be logged if the number of consecutive connection fails does not reach the parameter error_max_retries. Instead of errors, an INFO message is logged. This is a measurement against too frequent ERROR logging messages. The consecutive connection fails are reset if a data line has been successfully transferred. If the consecutive connection fails reaches the parameter error_max_retries, an exception will be thrown and rate_limit applies, if not null.
"},{"location":"user/bots/#tcp","title":"TCP","text":"
TCP is the bot responsible to receive events on a TCP port (ex: from TCP Output of another IntelMQ instance). Might not be working on Python 3.4.6.
 
Response
 
TCP collector just sends an \"OK\" message after every received message, this should not pose a problem for an arbitrary input. If you intend to link two IntelMQ instance via TCP, have a look at the TCP output bot documentation.
 
Module: intelmq.bots.collectors.tcp.collector
 
Parameters (also expects feed parameters):
 
ip
 
(required, string) IP of the destination server.
 
port
 
(required, integer) Port of destination server.
"},{"location":"user/bots/#blueliv-crimeserver","title":"Blueliv Crimeserver","text":"
Collects report messages from Blueliv API.
 
For more information visit https://github.com/Blueliv/api-python-sdk
 
Module: intelmq.bots.collectors.blueliv.collector_crimeserver
 
Requirements
 
Install the required library:
 
pip3 install -r intelmq/bots/collectors/blueliv/REQUIREMENTS.txt\n
 
Parameters (also expects feed parameters):
 
api_key
 
(required, string) location of information resource, see https://map.blueliv.com/?redirect=get-started#signup
 
api_url
 
(optional, string) The optional API endpoint. Defaults to https://freeapi.blueliv.com.
"},{"location":"user/bots/#calidog-certstream","title":"Calidog Certstream","text":"
A Bot to collect data from the Certificate Transparency Log (CTL). This bot works based on certstream library (https://github.com/CaliDog/certstream-python)
 
Module: intelmq.bots.collectors.calidog.collector_certstream
 
Parameters (also expects feed parameters):
"},{"location":"user/bots/#eset-eti","title":"ESET ETI","text":"
Collects data from ESET ETI TAXII server.
 
For more information visit https://www.eset.com/int/business/services/threat-intelligence/.
 
Module: intelmq.bots.collectors.eset.collector
 
Requirements
 
Install the required cabby library:
 
pip3 install -r intelmq/bots/collectors/eset/REQUIREMENTS.txt\n
 
Parameters (also expects feed parameters):
 
username
 
(required, string) Your username.
 
password
 
(required, string) Your password.
 
endpoint
 
(optional, string) Defaults to eti.eset.com.
 
time_delta
 
(optional, integer) The time (in seconds) span to look back. Default to 3600.
 
collection
 
(required, string) The collection to fetch.
"},{"location":"user/bots/#mcafee-opendxl","title":"McAfee openDXL","text":"
Collects messages via McAfee openDXL.
 
Module: intelmq.bots.collectors.opendxl.collector
 
Parameters (also expects feed parameters):
 
dxl_config_file
 
(required, string) Path to the the configuration file containing required information to connect.
 
dxl_topic
 
(optional, string) Name of the DXL topic to subscribe to. Defaults to /mcafee/event/atd/file/report.
"},{"location":"user/bots/#microsoft-azure","title":"Microsoft Azure","text":"
Collects blobs from Microsoft Azure using their library.
 
Iterates over all blobs in all containers in an Azure storage. The Cache is required to memorize which files have already been processed (TTL needs to be high enough to cover the oldest files available!).
 
This bot significantly changed in a backwards-incompatible way in IntelMQ Version 2.2.0 to support current versions of the Microsoft Azure Python libraries. azure-storage-blob>=12.0.0 is required.
 
Module: intelmq.bots.collectors.microsoft.collector_azure
 
Parameters (also expects feed parameters and cache parameters):
 
connection_string
 
(required, string) Connection string as given by Microsoft.
 
container_name
 
(required, string) Name of the container to connect to.
"},{"location":"user/bots/#microsoft-interflow","title":"Microsoft Interflow","text":"
This bot collects files from Microsoft Interflow API.
 
Iterates over all files available by this API. Make sure to limit the files to be downloaded with the parameters, otherwise you will get a lot of data! The cache is used to remember which files have already been downloaded. Make sure the TTL is high enough, higher than not_older_than.
 
Module: intelmq.bots.collectors.microsoft.collector_interflow
 
Parameters (also expects feed parameters):
 
api_key
 
(required, string) API generated in their portal.
 
file_match
 
(optional, string) Regular expression to match file names.
 
not_older_than
 
(optional, integer/datetime) an optional relative (minutes) or absolute time (UTC is assumed) expression to determine the oldest time of a file to be downloaded.
 
redis_cache_* and especially redis_cache_ttl
 
Settings for the cache where file names of downloaded files are saved. The cache's TTL must always be bigger than not_older_than.
 
Additional functionalities
 
Files are automatically ungzipped if the filename ends with .gz.
"},{"location":"user/bots/#stomp","title":"STOMP","text":"
Collects messages from a STOMP server.
 
Module: intelmq.bots.collectors.stomp.collector
 
Requirements
 
Install the stomp.py library from PyPI:
 
pip3 install -r intelmq/bots/collectors/stomp/REQUIREMENTS.txt\n
 
Alternatively, you may want to install it using your OS's native packaging tools, e.g.:
 
apt install python3-stomp\n
 
Apart from that, depending on what STOMP server you connect to, you may need to obtain, from the organization or company owning the server, one or more of the following security/authentication-related resources:
 
     
  • CA certificate file;
    •  
  • either: client certificate and client certificate's key files,   or: username (STOMP login) and password (STOMP passcode).
    •  
 
Also, you will need to know an appropriate STOMP destination (aka exchange point), e.g. /exchange/my.example.org/*.*.*.*.
 
Parameters (also expects feed parameters):
 
server
 
(required, string) STOMP server's hostname or IP, e.g. \"n6stream.cert.pl\" (which is default)
 
port
 
(optional, integer) STOMP server's port number (default: 61614)
 
exchange
 
(required, string) STOMP destination to subscribe to, e.g. \"/exchange/my.org/*.*.*.*\"
 
heartbeat
 
(optional, integer) default: 6000
 
ssl_ca_certificate
 
(optional, string) Path to CA file, or empty string to load system's default CA certificates
 
auth_by_ssl_client_certificate
 
(optional, boolean) Default: true (note: false is needed for new n6 auth)
 
ssl_client_certificate
 
(optional, string) Path to client certificate to use for TLS connections.
 
ssl_client_certificate_key
 
(optional, string) Path to client private key to use for TLS connections.
 
username
 
(optional, string) Username to use.
 
password
 
(optional, string) Password to use.
"},{"location":"user/bots/#twitter-remove","title":"Twitter (REMOVE?)","text":"
Collects tweets.
 
Collects tweets from target_timelines. Up to tweet_count tweets from each user and up to timelimit back in time. The tweet text is sent separately and if allowed, links to pastebin are followed and the text sent in a separate report
 
Module: intelmq.bots.collectors.twitter.collector_twitter
 
Parameters (also expects feed parameters):
 
target_timelines
 
() screen_names of twitter accounts to be followed
 
tweet_count
 
() number of tweets to be taken from each account
 
timelimit
 
() maximum age of the tweets collected in seconds
 
follow_urls
 
() list of screen_names for which URLs will be followed
 
exclude_replies
 
() exclude replies of the followed screen_names
 
include_rts
 
() whether to include retweets by given screen_name
 
consumer_key
 
() Twitter API login data
 
consumer_secret
 
() Twitter API login data
 
access_token_key
 
() Twitter API login data
 
access_token_secret
 
() Twitter API login data
"},{"location":"user/bots/#parser-bots","title":"Parser Bots","text":"
If not set differently during parsing, all parser bots copy the following fields from the report to an event:
 
     
  • feed.accuracy
    •  
  • feed.code
    •  
  • feed.documentation
    •  
  • feed.name
    •  
  • feed.provider
    •  
  • feed.url
    •  
  • rtir_id
    •  
  • time.observation
    •  
"},{"location":"user/bots/#common-parameters","title":"Common parameters","text":""},{"location":"user/bots/#default_fields","title":"default_fields","text":"
(optional, object) Map of statically added fields to each event (only applied if parsing the event doesn't set the value).
 
example usage:
 
defaults_fields:\n  classification.type: c2-server\n  protocol.transport: tcp\n
"},{"location":"user/bots/#copy_collector_provided_fields","title":"copy_collector_provided_fields","text":"
(optional, list) List of additional fields to be copy from the report (only applied if parsing the event doesn't set the value).
 
Example usage:
 
copy_collector_provided_fields:\n  - extra.file_name\n
"},{"location":"user/bots/#abusech-feodo-tracker","title":"Abuse.ch Feodo Tracker","text":"
Parses data from Abuse.ch Feodo Tracker (JSON format).
 
Module: intelmq.bots.parsers.abusech.parser_feodotracker
 
No additional parameters.
"},{"location":"user/bots/#alienvault-api","title":"AlienVault API","text":"
Parses data from AlienVault API.
 
Module: intelmq.bots.parsers.alienvault.parser
 
No additional parameters.
"},{"location":"user/bots/#alienvault-otx","title":"AlienVault OTX","text":"
Parses data from AlientVault Open Threat Exchange (OTX).
 
Module: intelmq.bots.parsers.alienvault.parser_otx
 
No additional parameters.
"},{"location":"user/bots/#anubisnetworks-cyberfeed-stream","title":"AnubisNetworks Cyberfeed Stream","text":"
Parses data from AnubisNetworks Cyberfeed Stream.
 
The feed format changes over time. The parser supports at least data from 2016 and 2020.
 
Events with the Malware \"TestSinkholingLoss\" are ignored, as they are for the feed provider's internal purpose only and should not be processed at all.
 
Module: intelmq.bots.parsers.anubisnetworks.parser
 
Parameters:
 
use_malware_family_as_classification_identifier
 
(optional, boolean) Use the malw.family field as classification.type. If false, check if the same as malw.variant. If it is the same, it is ignored. Otherwise saved as extra.malware.family. Defaults to true.
"},{"location":"user/bots/#bambenek","title":"Bambenek","text":"
Parses data from Bambenek DGA, Domain, and IP feeds.
 
Module: intelmq.bots.parsers.bambenek.parser
 
No additional parameters.
"},{"location":"user/bots/#blocklistde","title":"Blocklist.de","text":"
Parses data from Blocklist.de feeds.
 
Module: intelmq.bots.parsers.blocklistde.parser
 
No additional parameters.
"},{"location":"user/bots/#blueliv-crimeserver_1","title":"Blueliv Crimeserver","text":"
Parses data from Blueliv Crimeserver feed.
 
Module: intelmq.bots.parsers.blueliv.parser_crimeserver
 
No additional parameters.
"},{"location":"user/bots/#calidog-certstream_1","title":"Calidog Certstream","text":"
Parses data from Certificate Transparency Log.
 
For each domain in the leaf_cert.all_domains object one event with the domain in source.fqdn (and source.ip as fallback) is produced. The seen-date is saved in time.source and the classification type is other.
 
Module: intelmq.bots.parsers.calidog.parser_certstream
 
No additional parameters.
"},{"location":"user/bots/#cert-eu","title":"CERT-EU","text":"
Parses data from CERT-EU feed (CSV).
 
Module: intelmq.bots.parsers.certeu.parser_csv
 
No additional parameters.
"},{"location":"user/bots/#ci-army","title":"CI Army","text":"
Parses data from CI Army feed.
 
Module: intelmq.bots.parsers.ci_army.parser
 
No additional parameters.
"},{"location":"user/bots/#cleanmx","title":"CleanMX","text":"
Parses data from CleanMX feed.
 
Module: intelmq.bots.parsers.cleanmx.parser
 
No additional parameters.
"},{"location":"user/bots/#team-cymru-cap","title":"Team Cymru CAP","text":"
Parses data from Team Cymru's CSIRT Assistance Program (CAP) feed.
 
There are two different feeds available:
 
     
  • infected_$date.txt (\"old\")
    •  
  • $certname_$date.txt (\"new\")
    •  
 
The new will replace the old at some point in time, currently you need to fetch both. The parser handles both formats.
 
Old feed
 
As little information on the format is available, the mappings might not be correct in all cases. Some reports are not implemented at all as there is no data available to check if the parsing is correct at all. If you do get errors like Report ... not implement or similar please open an issue and report the (anonymized) example data. Thanks.
 
The information about the event could be better in many cases but as Cymru does not want to be associated with the report, we can't add comments to the events in the parser, because then the source would be easily identifiable for the recipient.
 
Module: intelmq.bots.parsers.cymru.parser_cap_program
 
No additional parameters.
"},{"location":"user/bots/#team-cymru-full-bogons","title":"Team Cymru Full Bogons","text":"
Parses data from full bogons feed.
 
http://www.team-cymru.com/bogon-reference.html
 
Module: intelmq.bots.parsers.cymru.parser_full_bogons
 
No additional parameters.
"},{"location":"user/bots/#cznic-haas","title":"CZ.NIC HaaS","text":"
Parses data from CZ.NIC Honeypot as a service (HaaS) feed.
 
Module: intelmq.bots.parsers.cznic.parser_haas
 
No additional parameters.
"},{"location":"user/bots/#cznic-proki","title":"CZ.NIC PROKI","text":"
Parses data from CZ.NIC PROKI API.
 
Module: intelmq.bots.parsers.cznic.parser_proki
 
No additional parameters.
"},{"location":"user/bots/#danger-rulez","title":"Danger Rulez","text":"
Parses data from Danger Rulez SSH blocklist.
 
Module: intelmq.bots.parsers.danger_rulez.parser
 
No additional parameters.
"},{"location":"user/bots/#dataplane","title":"Dataplane","text":"
Parses data from Dataplane feed.
 
Module: intelmq.bots.parsers.dataplane.parser
 
No additional parameters.
"},{"location":"user/bots/#dshield-asn","title":"DShield ASN","text":"
Parses data from DShield ASN feed.
 
Module: intelmq.bots.parsers.dshield.parser_asn
 
No additional parameters.
"},{"location":"user/bots/#dshield-block","title":"DShield Block","text":"
Parses data from DShield Block feed.
 
Module: intelmq.bots.parsers.dshield_parser_block
 
No additional parameters.
"},{"location":"user/bots/#eset","title":"ESET","text":"
Parses data from ESET ETI TAXII server.
 
Supported collections:
 
     
  • \"ei.urls (json)\"
    •  
  • \"ei.domains v2 (json)\"
    •  
 
Module: intelmq.bots.parsers.eset.parser
 
No additional parameters.
"},{"location":"user/bots/#dyn-todo","title":"Dyn (TODO)","text":""},{"location":"user/bots/#fireeye_1","title":"FireEye","text":"
Parses data from FireEye MAS appliance.
 
Module: intelmq.bots.parsers.fireeye.parser
 
No additional parameters.
"},{"location":"user/bots/#fraunhofer-dga","title":"Fraunhofer DGA","text":"
Parses data from Fraunhofer DGA feed.
 
Module: intelmq.bots.parsers.fraunhofer.parser_dga
 
No additional parameters.
"},{"location":"user/bots/#generic-csv","title":"Generic CSV","text":"
Parses CSV data.
 
Lines starting with # are skipped. Headers won't be interpreted.
 
Module: intelmq.bots.parsers.generic.parser_csv
 
Parameters
 
columns
 
(required, string/array of strings) A list of strings or a string of comma-separated values with field names. The names must match the IntelMQ Data Format field names. Empty column specifications and columns named __IGNORE__ are ignored. E.g.
 
columns:\n  - \"source.ip\"\n  - \"source.fqdn\"\n  - \"extra.http_host_header\"\n  - \"__IGNORE__\"\n
 
is equivalent to:
 
columns: \"source.ip,source.fqdn,extra.http_host_header,__IGNORE__\"\n
 
The fourth column is not used in this example.
 
It is possible to specify multiple columns using the | character. E.g.
 
columns:\n  - \"source.url|source.fqdn|source.ip\"\n  - \"source.fqdn\"\n  - \"extra.http_host_header\"\n  - \"__IGNORE__\"\n
 
First, the bot will try to parse the value as URL, if it fails, it will try to parse it as FQDN, if that fails, it will try to parse it as IP, if that fails, an error will be raised. Some use cases:
 
     
  • Mixed data set, e.g. URL/FQDN/IP/NETMASK:
    •  
 
columns:\n  - \"source.url|source.fqdn|source.ip|source.network\"\n
 
     
  • Parse a value and ignore if it fails:
    •  
 
columns:\n  - \"source.url|__IGNORE__\"\n
 
column_regex_search
 
(optional, object) A dictionary mapping field names (as given per the columns parameter) to regular expression. The field is evaluated using re.search. Eg. to get the ASN out of AS1234 use: {\"source.asn\": \"[0-9]*\"}. Make sure to properly escape any backslashes in your regular expression (see also this issue).
 
compose_fields
 
(optional, object) Compose fields from multiple columns, e.g. with data like this:
 
# Host,Path\nexample.com,/foo/\nexample.net,/bar/\n
 
Using this parameter:
 
compose_fields:\n  source.url: \"http://{0}{1}\"\n
 
You get:
 
http://example.com/foo/\nhttp://example.net/bar/\n
 
in the respective source.url fields. The value in the dictionary mapping is formatted whereas the columns are available with their index.
 
default_url_protocol
 
(optional, string) For URLs you can give a default protocol which will be prepended to the data. Defaults to null.
 
delimiter
 
(optional, string) Character used for columns separation. Defaults to , (comma).
 
skip_header
 
(optional, boolean/integer) Whether to skip the first N lines of the input (true equals to 1, false requalis to 0). Lines starting with # will be skipped additionally, make sure you do not skip more lines than needed! Defaults to false/0.
 
time_format
 
(optional, string) Allowed values: timestamp, windows_nt or epoch_millis. When null then fuzzy time parsing is used. Defaults to null.
 
type
 
(optional, string) Set the classification.type statically. Deprecated in favour of default_fields . Will be removed in IntelMQ 4.0.0.
 
data_type
 
(optional, object) Sets the data of specific type, currently only json is a supported value.
 
Example:
 
columns:\n  - source.ip\n  - source.url\n  - extra.tags\ndata_type:\n  extra.tags: json\n
 
It will ensure that extra.tags is treated as JSON.
 
filter_text
 
(optional, string) Only process the lines containing or not containing specified text. It is expected to be used in conjunction with filter_type.
 
filter_type
 
(optional, string) Allowed values: whitelist or blacklist. When whitelist is used, only lines containing the text specified in filter_text option will be processed. When blacklist is used, only lines NOT containing the text will be processed.
 
Example (processing ipset format files):
 
filter_text: 'ipset add '\nfilter_type: whitelist\ncolumns:\n  - __IGNORE__\n  - __IGNORE__\n  - __IGNORE__\n  - source.ip\n
 
type_translation
 
(optional, object) If the source does have a field with information for classification.type, but it does not correspond to IntelMQ's types, you can map them to the correct ones. The type_translation field can hold a dictionary, or a string with a JSON dictionary which maps the feed's values to IntelMQ's.
 
Example:
 
type_translation:\n  malware_download: \"malware-distribution\"\n
 
columns_required
 
(optional, array of booleans) An array of true/false for each column. By default, it is true for every column.
"},{"location":"user/bots/#github-feed","title":"Github Feed","text":"
Parses data publicly available on GitHub (should receive from github_api collector).
 
Module: intelmq.bots.parsers.github_feed.parser
 
No additional parameters.
"},{"location":"user/bots/#have-i-been-pwned-callback","title":"Have I Been Pwned Callback","text":"
Parsers data from the callback of Have I Been Pwned Enterprise Subscription.
 
Parses breaches and pastes and creates one event per e-mail address. The e-mail address is stored in source.account . classification.type is leak and classification.identifier is breach or paste.
 
Module: intelmq.bots.parsers.hibp.parser_callback
 
No additional parameters.
"},{"location":"user/bots/#html-table","title":"HTML Table","text":"
Parses tables in HTML documents.
 
Module: intelmq.bots.parsers.html_table.parser
 
Parameters:
 
(required, string/array of strings) A list of strings or a string of comma-separated values with field names. The names must match the IntelMQ Data Format field names. Empty column specifications and columns named __IGNORE__ are ignored. E.g.
 
columns:\n  - \"source.ip\"\n  - \"source.fqdn\"\n  - \"extra.http_host_header\"\n  - \"__IGNORE__\"\n
 
is equivalent to:
 
columns: \"source.ip,source.fqdn,extra.http_host_header,__IGNORE__\"\n
 
The fourth column is not used in this example.
 
It is possible to specify multiple columns using the | character. E.g.
 
columns:\n  - \"source.url|source.fqdn|source.ip\"\n  - \"source.fqdn\"\n  - \"extra.http_host_header\"\n  - \"__IGNORE__\"\n
 
First, the bot will try to parse the value as URL, if it fails, it will try to parse it as FQDN, if that fails, it will try to parse it as IP, if that fails, an error will be raised. Some use cases:
 
     
  • Mixed data set, e.g. URL/FQDN/IP/NETMASK:
    •  
 
columns:\n  - \"source.url|source.fqdn|source.ip|source.network\"\n
 
     
  • Parse a value and ignore if it fails:
    •  
 
columns:\n  - \"source.url|__IGNORE__\"\n
 
ignore_values
 
(optional, string/array of strings) A list of strings or a string of comma-separated values which are ignored when encountered.
 
Example:
 
ignore_values:\n  - \"\"\n  - \"unknown\"\n  - \"Not listed\"\n
 
The following configuration will lead to assigning all values to malware.name and extra.SBL except unknown and Not listed respectively.
 
columns:\n  - source.url\n  - malware.name\n  - extra.SBL\nignore_values:\n  - ''\n  - unknown\n  - Not listed\n
 
Parameters columns and ignore_values must have same length!
 
attribute_name
 
(optional, string) Filtering table with table attributes. To be used in conjunction with attribute_value. E.g. class, id, style.
 
attribute_value
 
(optional, string) To filter all tables with attribute class='details' use
 
attribute_name: \"class\"\nattribute_value: \"details\"\n
 
table_index
 
(optional, integer) Index of the table if multiple tables present. If attribute_name and attribute_value given, index according to tables remaining after filtering with table attribute. Defaults to 0.
 
split_column
 
(optional, ) Padded column to be split to get values, to be used in conjunction with split_separator and split_index, optional.
 
split_separator
 
(optional, string) Delimiter string for padded column.
 
split_index
 
(optional, integer) Index of unpadded string in returned list from splitting split_column with split_separator as delimiter string. Defaults to 0.
 
Example:
 
split_column: \"source.fqdn\"\nsplit_separator: \" \"\nsplit_index: 1\n
 
With above configuration, column corresponding to source.fqdn with value D lingvaworld.ru will be assigned as source.fqdn: lingvaworld.ru.
 
skip_table_head
 
(optional, boolean) Skip the first row of the table. Defaults to true.
 
default_url_protocol
 
(optional, string) For URLs you can give a default protocol which will be pretended to the data. Defaults to http://.
 
time_format
 
(optional, string) Allowed values: timestamp, windows_nt or epoch_millis. When null then fuzzy time parsing is used. Defaults to null.
 
html_parser
 
(optional, string) The HTML parser to use. Allowed values: html.parser or lxml (see also https://www.crummy.com/software/BeautifulSoup/bs4/doc/). Defaults to html.parser.
"},{"location":"user/bots/#json-todo","title":"JSON (TODO)","text":"
TODO
 
Module: intelmq.bots.parsers.json.parser
"},{"location":"user/bots/#keyvalue-parser","title":"Key=Value Parser","text":"
Parses text lines in key=value format, for example FortiGate firewall logs.
 
Parsing limitations
 
The input must not have (quoted) occurrences of the separator in the values. For example, this is not parsable (with space as separator):
 
key=\"long value\" key2=\"other value\"\n
 
In firewall logs like FortiGate, this does not occur. These logs usually look like:
 
srcip=192.0.2.1 srcmac=\"00:00:5e:00:17:17\"\n
 
Module: intelmq.bots.parsers.key_value.parser
 
Parameters:
 
pair_separator
 
(optional, string) String separating key=value pairs. Defaults to space.
 
kv_separator
 
(optional, string) String separating the key and the value. Defaults to =.
 
keys
 
(optional, object) Mapping of original key names to IntelMQ Data Format.
 
Example:
 
keys:\n  srcip: source.ip\n  dstip: destination.ip\n
 
The value mapped to time.source is parsed. If the value is numeric, it is interpreted. Otherwise, or if it fails, it is parsed fuzzy with dateutil. If the value cannot be parsed, a warning is logged per line.
 
strip_quotes
 
(optional, boolean) Whether to remove opening and closing quotes from values. Defaults to true.
"},{"location":"user/bots/#malwarepatrol","title":"MalwarePatrol","text":"
Parses data from MalwarePatrol feed.
 
Module: intelmq.bots.parsers.malwarepatrol.parser_dansguardian
 
No additional parameters.
"},{"location":"user/bots/#malwareurl","title":"MalwareURL","text":"
Parses data from MalwareURL feed.
 
Module: intelmq.bots.parsers.malwareurl.parser
 
No additional parameters.
"},{"location":"user/bots/#mcafee-advanced-threat-defense-file","title":"McAfee Advanced Threat Defense File","text":"
Parse IoCs from McAfee Advanced Threat Defense reports (hash, IP, URL).
 
Module: intelmq.bots.parsers.mcafee.parser_atd
 
Parameters:
 
verdict_severity
 
(optional, integer) Minimum report severity to parse. Defaults to 4.
"},{"location":"user/bots/#microsoft-ctip","title":"Microsoft CTIP","text":"
Parses data from the Microsoft CTIP feed.
 
Can parse the JSON format provided by the Interflow interface (lists of dictionaries) as well as the format provided by the Azure interface (one dictionary per line). The provided data differs between the two formats/providers.
 
The parser is capable of parsing both feeds:
 
     
  • ctip-c2
    •  
  • ctip-infected-summary The feeds only differ by a few fields, not in the format.
    •  
 
The feeds contain a field called Payload which is nearly always a base64 encoded JSON structure. If decoding works, the contained fields are saved as extra.payload.*, otherwise the field is saved as extra.payload.text.
 
Module: intelmq.bots.parsers.microsoft.parser_ctip
 
Parameters:
 
overwrite
 
(optional, boolean) Overwrite an existing field feed.name with DataFeed of the source. Defaults to false.
"},{"location":"user/bots/#misp","title":"MISP","text":"
Parses MISP events.
 
MISP events collected by the MISPCollectorBot are passed to this parser for processing. Supported MISP event categories and attribute types are defined in the SUPPORTED_MISP_CATEGORIES and MISP_TYPE_MAPPING class constants.
 
Module: intelmq.bots.parsers.misp.parser
 
No additional parameters.
"},{"location":"user/bots/#n6","title":"N6","text":"
Parses n6 data into IntelMQ format.
 
Test messages are ignored, this is logged with debug logging level. Also contains a mapping for the classification ( results in taxonomy, type and identifier). The name field is normally used as malware.name, if that fails due to disallowed characters, these characters are removed and the original value is saved as event_description.text. This can happen for names like further iocs: text with invalid ' char.
 
If a n6 message contains multiple IP addresses, multiple events are generated, resulting in events only differing in the address information.
 
Module: intelmq.bots.parsers.n6.parser_n6stomp
 
No additional parameters.
"},{"location":"user/bots/#openphish-free","title":"OpenPhish Free","text":"
Parses data from OpenPhish Free feed.
 
Module: intelmq.bots.parsers.openphish.parser
 
No additional parameters.
"},{"location":"user/bots/#openphish-premium","title":"OpenPhish Premium","text":"
Parses data from OpenPhish Premium feed (JSON).
 
Module: intelmq.bots.parsers.openphish.parser_commercial
 
No additional parameters.
"},{"location":"user/bots/#phishtank","title":"Phishtank","text":"
Parses data from Phishtank feed.
 
Module: intelmq.bots.parsers.phishtank.parser
 
No additional parameters.
"},{"location":"user/bots/#shadowserver","title":"Shadowserver","text":"
The Shadowserver parser operates on CSV formatted data.
 
How this bot works?
 
There are two possibilities for the bot to determine which report type the data belongs to in order to determine the correct mapping of the columns:
 
     
  1.  
    Automatic report type detection
     
    Since IntelMQ version 2.1 the parser can detect the feed based on metadata provided by the collector.
     
    When processing a report, this bot takes extra.file_name from the report and looks in config.py how the report should be parsed. If this lookup is not possible, and the feedname is not given as parameter, the feed cannot be parsed.
     
    The field extra.file_name has the following structure: %Y-%m-%d-${report_name}[-suffix].csv where the optional suffix can be something like country-geo. For example, some possible filenames are 2019-01-01-scan_http-country-geo.csv or 2019-01-01-scan_tftp.csv. The important part is the report_name, between the date and the suffix. Since version 2.1.2 the date in the filename is optional, so filenames like scan_tftp.csv are also detected.
     
    2.  
  2.  
    Fixed report type
     
    If the method above is not possible and for upgraded instances, the report type can be set with the feedname parameter. Report type is derived from the subject of Shadowserver e-mails. A list of possible values of the feedname parameter can be found in the table below in the column \"Report Type\".
     
    3.  
 
Module:
 
intelmq.bots.parsers.shadowserver.parser
 
Parameters:
 
feedname
 
(optional, string) Name of the Shadowserver report. The value for each report type can be found in the schema feed_name field.
 
For example using curl -s https://interchange.shadowserver.org/intelmq/v1/schema | jq .[].feed_name.
 
overwrite
 
(optional, boolean) If an existing feed.name should be overwritten.
 
auto_update
 
(optional, boolean) Enable automatic schema download.
 
Supported reports:
 
The report configuration is stored in a shadowserver-schema.json file downloaded from https://interchange.shadowserver.org/intelmq/v1/schema.
 
The parser will attempt to download a schema update on startup when the auto_update option is enabled.
 
Schema downloads can also be scheduled as a cron job for the intelmq user:
 
  02  01 *   *   *     intelmq.bots.parsers.shadowserver.parser --update-schema\n
 
For air-gapped systems automation will be required to download and copy the file to VAR_STATE_PATH/shadowserver-schema.json.
 
The parser will automatically reload the configuration when the file changes.
 
Schema contract
 
Once set in the schema, the classification.identifier, classification.taxonomy, and classification.type fields will remain static for a specific report.
 
The schema revision history is maintained at https://github.com/The-Shadowserver-Foundation/report_schema/.
 
Sample configuration
 
  shadowserver-parser:\n    bot_id: shadowserver-parser\n    name: Shadowserver Parser\n    enabled: true\n    group: Parser\n    groupname: parsers\n    module: intelmq.bots.parsers.shadowserver.parser\n    parameters:\n      destination_queues:\n        _default: [file-output-queue]\n      auto_update: true\n    run_mode: continuous\n
"},{"location":"user/bots/#shodan","title":"Shodan","text":"
Parses data from Shodan (search, stream etc).
 
The parser is by far not complete as there are a lot of fields in a big nested structure. There is a minimal mode available which only parses the important/most useful fields and also saves everything in extra.shodan keeping the original structure. When not using the minimal mode if may be useful to ignore errors as many parsing errors can happen with the incomplete mapping.
 
Module: intelmq.bots.parsers.shodan.parser
 
Parameters:
 
ignore_errors
 
(optional, boolean) Defaults to true.
 
minimal_mode
 
(optional, boolean) Defaults to false.
"},{"location":"user/bots/#spamhaus-drop","title":"Spamhaus DROP","text":"
Parses data from Spamhaus DROP feed.
 
Module: intelmq.bots.parsers.spamhaus.parser_drop
 
No additional parameters.
"},{"location":"user/bots/#spamhaus-cert","title":"Spamhaus CERT","text":"
Parses data from Spamhaus CERT feed.
 
Module: intelmq.bots.parsers.spamhaus.parser_cert
 
No additional parameters.
"},{"location":"user/bots/#surbl","title":"Surbl","text":"
Parses data from surbl feed.
 
Module: intelmq.bots.parsers.surbl.parser
 
No additional parameters.
"},{"location":"user/bots/#threatminer","title":"Threatminer","text":"
Parses data from Threatminer feed.
 
Module: intelmq.bots.parsers.threatminer.parser
 
No additional parameters.
"},{"location":"user/bots/#turris","title":"Turris","text":"
Parses data from Turris Greylist feed.
 
Module: intelmq.bots.parsers.turris.parser
 
No additional parameters.
"},{"location":"user/bots/#twitter","title":"Twitter","text":"
Extracts URLs from text, fuzzy, aimed at parsing tweets.
 
Module: intelmq.bots.parsers.twitter.parser
 
Parameters:
 
domain_whitelist
 
(optional, array of strings) domains to be filtered out
 
substitutions
 
(optional, string) Semicolon delimited list of even length of pairs of substitutions (for example: .;.;,;. substitutes . for . and , for .).
 
classification_type
 
(optional, string) Statically set classification.type.
 
default_scheme
 
(optional, string) Default scheme for URLs if not given. See also the next section.
 
Default scheme
 
The dependency url-normalize changed it's behavior in version 1.4.0 from using http:// as default scheme to https://. Version 1.4.1 added the possibility to specify it. Thus you can only use the default_scheme parameter with a current version of this library >= 1.4.1, with 1.4.0 you will always get https:// as default scheme and for older versions < 1.4.0 http:// is used.
 
This does not affect URLs which already include the scheme.
"},{"location":"user/bots/#vxvault","title":"VxVault","text":"
Parses data from VxVault feed.
 
Module: intelmq.bots.parsers.vxvault.parser
 
No additional parameters.
"},{"location":"user/bots/#zoneh","title":"ZoneH","text":"
Parses data from ZoneH.
 
This bot is designed to consume defacement reports from zone-h.org. It expects fields normally present in CSV files distributed by email.
 
Module: intelmq.bots.parsers.zoneh.parser
 
No additional parameters.
"},{"location":"user/bots/#expert-bots","title":"Expert Bots","text":"
Expert bots are used for enriching, filtering and/or other data manipulation.
"},{"location":"user/bots/#abusix","title":"Abusix","text":"
This bot adds source.abuse_contact and destination.abuse_contact e-mail addresses. They are obtained via DNS TXT queries to Abusix servers.
 
Requirements
 
This bot can optionally use the python module querycontacts by Abusix itself: https://pypi.org/project/querycontacts/
 
pip3 install querycontacts\n
 
If the package is not installed, our own routines are used.
 
Module: intelmq.bots.experts.abusix.expert
 
Parameters (also expects cache parameters):
 
No additional parameters.
"},{"location":"user/bots/#aggregate","title":"Aggregate","text":"
Aggregates events based upon given fields & timespan.
 
Define specific fields to filter incoming events and aggregate them. Also set the timespan you want the events to get aggregated.
 
The \"cleanup\" procedure, sends out the aggregated events or drops them based upon the given threshold value. It is called on every incoming message and on the bot's initialization. If you're potentially running on low traffic ( no incoming events within the given timestamp ) it is recommended to reload or restart the bot via cronjob each 30 minutes (adapt to your configured timespan). Otherwise you might loose information.
 
I. e.:
 
crontab -e\n\n0,30 * * * * intelmqctl reload my-aggregate-bot\n
 
For reloading/restarting please check the intelmqctl documentation.
 
Module: intelmq.bots.experts.aggregate.expert
 
Parameters (also expects cache parameters):
 
Warning
 
redis_cache_ttl is not used at it would result in data loss.
 
fields
 
(required, string) Given fields which are used to aggregate like classification.type, classification.identifier.
 
threshold
 
(required, integer) If the aggregated event is lower than the given threshold after the timespan, the event will get dropped.
 
timespan
 
(required, string) Timespan to aggregate events during the given time. I. e. 1 hour
"},{"location":"user/bots/#asn-lookup","title":"ASN Lookup","text":"
This bot uses an offline database to add source.asn and destination.asn based on the respective IP address.
 
Requirements
 
Install pyasn module.
 
pip3 install pyasn\n
 
Module: intelmq.bots.experts.asn_lookup.expert
 
Parameters:
 
database
 
(required, string) Path to the downloaded database.
 
Database
 
Use this command to create/update the database and reload the bot:
 
intelmq.bots.experts.asn_lookup.expert --update-database\n
 
The database is fetched from routeviews.org and licensed under the Creative Commons Attribution 4.0 International license (see the routeviews FAQ).
"},{"location":"user/bots/#csv-converter","title":"CSV Converter","text":"
Converts an event to CSV format, saved in the output field.
 
To use the CSV-converted data in an output bot - for example in a file output, use the configuration parameter single_key of the output bot and set it to output.
 
Module: intelmq.bots.experts.csv_converter.expert
 
Parameters:
 
delimiter
 
(optional, string) Defaults to ,.
 
fieldnames
 
(required, string) Comma-separated list of field names, e.g. \"time.source,classification.type,source.ip\".
"},{"location":"user/bots/#team-cymru-whois","title":"Team Cymru Whois","text":"
This bot adds geolocation, ASN and BGP prefix based on IP address.
 
Public documentation: https://www.team-cymru.com/IP-ASN-mapping.html#dns
 
Module: intelmq.bots.experts.cymru_whois.expert
 
Parameters (also expects cache parameters):
 
overwrite
 
(optional, boolean) Whether to overwrite existing fields. Defaults to true.
"},{"location":"user/bots/#remove-affix","title":"Remove Affix","text":"
Remove part of string from string fields, example: www. from source.fqdn.
 
Module: intelmq.bots.experts.remove_affix.expert
 
Parameters:
 
remove_prefix
 
(optional, boolean) True - cut from start, False - cut from end. Defaults to true.
 
affix
 
(required, string) example 'www.'
 
field
 
(required, string) Which field to modify. 'source.fqdn'
"},{"location":"user/bots/#domain-suffix","title":"Domain Suffix","text":"
This bots uses an offline database to add the public suffix to the event, derived by a domain. See or information on the public suffix list: https://publicsuffix.org/list/. Only rules for ICANN domains are processed. The list can (and should) contain Unicode data, punycode conversion is done during reading.
 
Note that the public suffix is not the same as the top level domain (TLD). E.g. co.uk is a public suffix, but the TLD is uk. Privately registered suffixes (such as blogspot.co.at) which are part of the public suffix list too, are ignored.
 
Rule processing
 
A short summary how the rules are processed:
 
The simple ones:
 
com\nat\ngv.at\n
 
example.com leads to com, example.gv.at leads to gv.at.
 
Wildcards:
 
*.example.com\n
 
www.example.com leads to www.example.com.
 
And additionally the exceptions, together with the above wildcard rule:
 
!www.example.com\n
 
www.example.com does now not lead to www.example.com, but to example.com.
 
Module: intelmq.bots.experts.domain_suffix.expert
 
Parameters:
 
field
 
(required, string) Allowed values: fqdn or reverse_dns.
 
suffix_file
 
(required, string) path to the suffix file
 
Database
 
Use this command to create/update the database and reload the bot:
 
intelmq.bots.experts.domain_suffix.expert --update-database\n
"},{"location":"user/bots/#domain-valid","title":"Domain Valid","text":"
Checks if a domain is valid by performing multiple validity checks (see below).
 
If the field given in domain_field does not exist in the event, the event is dropped. If the domain contains underscores (_), the event is dropped. If the domain is not valid according to the validators library, the event is dropped. If the domain's last part (the TLD) is not in the TLD-list configured by parameter tlds_domains_list, the field is dropped. Latest TLD list: https://data.iana.org/TLD/
 
Module: intelmq.bots.experts.domain_valid.expert
 
Parameters:
 
domain_field
 
(required, string) The name of the field to be validated.
 
tlds_domains_list
 
(required, string) Path to a local file with all valid TLDs. Defaults to /opt/intelmq/var/lib/bots/domain_valid/tlds-alpha-by-domain.txt
"},{"location":"user/bots/#deduplicator","title":"Deduplicator","text":"
Bot responsible for dropping duplicate events. Deduplication can be performed based on an arbitrary set of fields.
 
Module: intelmq.bots.experts.deduplicator.expert
 
Parameters (also expects cache parameters):
 
bypass
 
(optional, boolean) Whether to bypass the deduplicator or not. When set to true, messages will not be deduplicated. Defaults to false.
 
filter_type
 
(optional, string) Allowed values: blacklist or whitelist. The filter type will be used to define how Deduplicator bot will interpret the parameter filter_keys in order to decide whether an event has already been seen or not, i.e., duplicated event or a completely new event.
 
     
  • whitelist configuration: only the keys listed in filter_keys will be considered to verify if an event is   duplicated or not.
    •  
  • blacklist configuration: all keys except those in filter_keys will be considered to verify if an event is   duplicated or not.
    •  
 
filter_keys
 
(optional, string) string with multiple keys separated by comma. Please note that time.observation key will not be considered even if defined, because the system always ignore that key.
 
When using a whitelist field pattern and a small number of fields (keys), it becomes more important, that these fields exist in the events themselves. If a field does not exist, but is part of the hashing/deduplication, this field will be ignored. If such events should not get deduplicated, you need to filter them out before the deduplication process, e.g. using a sieve expert. See also this discussion thread on the mailing-list.
 
Configuration Example
 
Example 1
 
The bot with this configuration will detect duplication only based on source.ip and destination.ip keys.
 
parameters:\n  redis_cache_db: 6\n  redis_cache_host: \"127.0.0.1\"\n  redis_cache_password: null\n  redis_cache_port: 6379\n  redis_cache_ttl: 86400\n  filter_type: \"whitelist\"\n  filter_keys: \"source.ip,destination.ip\"\n
 
Example 2
 
The bot with this configuration will detect duplication based on all keys, except source.ip and destination.ip keys.
 
parameters:\n  redis_cache_db: 6\n  redis_cache_host: \"127.0.0.1\"\n  redis_cache_password: null\n  redis_cache_port: 6379\n  redis_cache_ttl: 86400\n  filter_type: \"blacklist\"\n  filter_keys: \"source.ip,destination.ip\"\n
 
Flushing the cache
 
To flush the deduplicator's cache, you can use the redis-cli tool. Enter the database used by the bot and submit the flushdb command:
 
redis-cli -n 6\nflushdb\n
"},{"location":"user/bots/#do-portal","title":"DO Portal","text":"
The DO portal retrieves the contact information from a DO portal instance: http://github.com/certat/do-portal/
 
Module: intelmq.bots.experts.do_portal.expert
 
Parameters:
 
mode
 
(required, string) Allowed values: replace or append. How to handle new abuse contacts in case there are existing ones.
 
portal_url
 
(required, string) The URL to the portal, without the API-path. The used URL is $portal_url + '/api/1.0/ripe/contact?cidr=%s'.
 
portal_api_key
 
(required, string) The API key of the user to be used. Must have sufficient privileges.
"},{"location":"user/bots/#field-reducer","title":"Field Reducer","text":"
The field reducer bot is capable of removing fields from events.
 
Module: intelmq.bots.experts.field_reducer.expert
 
Parameters:
 
type
 
(required, string) Allowed values: whitelist or blacklist. When whitelist is set, tnly the fields in keys will passed along. When blacklist is set then the fields in keys will be removed from events.
 
keys
 
(required, array of strings) Can be an array of field names or a string with a comma-separated list of field names.
"},{"location":"user/bots/#filter","title":"Filter","text":"
The filter bot is capable of filtering specific events.
 
A simple filter for messages (drop or pass) based on a exact string comparison or regular expression.
 
Module: intelmq.bots.experts.filter.expert
 
Parameters:
 
Parameters for filtering with key/value attributes
 
filter_key
 
(required, string) - key from data format
 
filter_value
 
(required, string) - value for the key
 
filter_action
 
(required, string) - action when a message match to the criteria (possible actions: keep/drop)
 
filter_regex
 
(optional, boolean) - attribute determines if the filter_value shall be treated as regular expression or not.
 
If this attribute is not empty (can be true, yes or whatever), the bot uses python's re.search function to evaluate the filter with regular expressions. If this attribute is empty or evaluates to false, an exact string comparison is performed. A check on string inequality can be achieved with the usage of Paths described below.
 
Parameters for time based filtering
 
not_before
 
(optional, string) Events before this time will be dropped. Example: 1 week.
 
not_after
 
(optional, string) - Events after this time will be dropped.
 
Both parameters accept string values describing absolute or relative time:
 
     
  • absolute
    •  
  • basically anything parseable by datetime parser, eg.
    •  
 
2015-09-12T06:22:11+00:00\n
 
time.source
 
(optional, string) Taken from the event will be compared to this value to decide the filter behavior.
 
     
  • relative
    •  
  • accepted string formatted like this \" \", where epoch could be any of following strings (could   optionally end with trailing 's'): hour, day, week, month, year 
  • time.source taken from the event will be compared to the value (now - relative) to decide the filter behavior
    •  
    Examples of time filter definition
     
       
    • not_before: \"2015-09-12T06:22:11+00:00\" - events older than the specified time will be dropped
      •  
    • not_after: \"6 months\" - just events older than 6 months will be passed through the pipeline
      •  
     
    Possible paths
     
       
    • _default: default path, according to the configuration
      •  
    • action_other: Negation of the default path
      •  
    • filter_match: For all events the filter matched on
      •  
    • filter_no_match: For all events the filter does not match
      •  
     action match _default action_other filter_match filter_no_match keep \u2713 \u2713 \u2717 \u2713 \u2717 keep \u2717 \u2717 \u2713 \u2717 \u2713 drop \u2713 \u2717 \u2713 \u2713 \u2717 drop \u2717 \u2713 \u2717 \u2717 \u2713 
    In DEBUG logging level, one can see that the message is sent to both matching paths, also if one of the paths is not configured. Of course the message is only delivered to the configured paths.
    "},{"location":"user/bots/#format-field","title":"Format Field","text":"
    String method operations on column values.
     
    Module: intelmq.bots.experts.format_field.expert
     
    Parameters:
     
    Parameters for stripping chars
     
    strip_columns (optional, string/array of strings) A list of strings or a string of comma-separated values with field names. The names must match the IntelMQ Data Format field names.
     
    For example:
     
    columns:\n  - malware.name\n  - extra.tags\n
     
    is equivalent to:
     
    columns: \"malware.name,extra.tags\"\n
     
    strip_chars
     
    (optional, string) Set of characters to remove as leading/trailing characters. Defaults to space.
     
    Parameters for replacing chars
     
    replace_column
     
    () key from data format
     
    old_value
     
    () the string to search for
     
    new_value
     
    () the string to replace the old value with
     
    replace_count () number specifying how many occurrences of the old value you want to replace(default: [1])
     
    Parameters for splitting string to list of string
     
    split_column
     
    () key from data format
     
    split_separator
     
    () specifies the separator to use when splitting the string(default: ,)
     
    Order of operation: strip -> replace -> split. These three methods can be combined such as first strip and then split.
    "},{"location":"user/bots/#generic-db-lookup","title":"Generic DB Lookup","text":"
    This bot is capable for enriching intelmq events by lookups to a database. Currently only PostgreSQL and SQLite are supported.
     
    If more than one result is returned, a ValueError is raised.
     
    Module: intelmq.bots.experts.generic_db_lookup.expert
     
    Parameters:
     
    Connection
     
    engine
     
    (required, string) Allowed values: postgresql or sqlite.
     
    database
     
    (optional, string) Database name or the SQLite filename. Defaults to intelmq.
     
    table
     
    (optional, string) Name of the table. Defaults to contacts.
     
    PostgreSQL specific parameters
     
    host
     
    (optional, string) Hostname of the PostgreSQL server. Defaults to localhost.
     
    port
     
    (optional, integer) Port of the PostgreSQL server. Defaults to 5432.
     
    user
     
    (optional, string) Username for accessing PostgreSQL. Defaults to intelmq.
     
    password
     
    (optional, string) Password for accessing PostgreSQL. Defaults to ?.
     
    sslmode
     
    (optional, string) Type of TLS mode to use. Defaults to require.
     
    Lookup
     
    match_fields
     
    (optional, object) The value is a key-value mapping an arbitrary number IntelMQ field names to table column names. The values are compared with = only. Defaults to source.asn: \"asn\".
     
    Replace fields
     
    overwrite
     
    (optional, boolean) Whether to overwrite existing fields. Defaults to false.
     
    replace_fields
     
    (optional, object) Key-value mapping an arbitrary number of table column names to IntelMQ field names. Defaults to {\"contact\": \"source.abuse_contact\"}.
    "},{"location":"user/bots/#gethostbyname","title":"Gethostbyname","text":"
    This bot resolves to IP address (source.ip and destination.ip). Can possibly use also the source.url and destination.url for extracting FQDN.
     
    This bot resolves the DNS name (source.fqdn and destination.fqdn) using the gethostbyname syscall to an IP address (source.ip and destination.ip). The following gaierror resolution errors are ignored and treated as if the hostname cannot be resolved:
     
       
    • -2/EAI_NONAME: NAME or SERVICE is unknown
      •  
    • -4/EAI_FAIL: Non-recoverable failure in name res.
      •  
    • -5/EAI_NODATA: No address associated with NAME.
      •  
    • -8/EAI_SERVICE: SERVICE not supported for `ai_socktype'.
      •  
    • -11/EAI_SYSTEM: System error returned in `errno'.
      •  
     
    Other errors result in an exception if not ignored by the parameter gaierrors_to_ignore. All gaierrors can be found here: http://www.castaglia.org/proftpd/doc/devel-guide/src/lib/glibc-gai_strerror.c.html
     
    Module: intelmq.bots.experts.gethostbyname.expert
     
    Parameters:
     
    fallback_to_url
     
    (optional, boolean) When true and no source.fqdn present, use source.url instead for producing source.ip.
     
    gaierrors_to_ignore
     
    (optional, array of integers) Gaierror codes to ignore, e.g. -3 for EAI_AGAIN (Temporary failure in name resolution). Only accepts the integer values, not the names.
     
    overwrite
     
    (optional, boolean) Whether to overwrite existing source.ip and/or source.destination fields. Defaults to false.
    "},{"location":"user/bots/#http-status","title":"HTTP Status","text":"
    The bot fetches the HTTP status for a given URL and saves it in the event.
     
    Module: intelmq.bots.experts.http.expert_status
     
    Parameters:
     
    field
     
    (required, string) The name of the field containing the URL to be checked.
     
    success_status_codes
     
    (optional, array of integers) An array of success status codes. If this parameter is omitted or the list is empty, successful status codes are the ones between 200 and 400.
     
    overwrite
     
    (optional, boolean) Whether to overwrite existing status field. Defaults to false.
    "},{"location":"user/bots/#http-content","title":"HTTP Content","text":"
    Fetches an HTTP resource and checks if it contains a specific string.
     
    The bot fetches an HTTP resource and checks if it contains a specific string.
     
    Module: intelmq.bots.experts.http.expert_content
     
    Parameters:
     
    field
     
    (optional, string) The name of the field containing the URL to be checked. Defaults to source.url.
     
    needle
     
    (optional, string) The string that the content available on URL is checked for.
     
    overwrite
     
    (optional, boolean) Whether to overwrite existing status field. Defaults to false.
    "},{"location":"user/bots/#idea-converter","title":"IDEA Converter","text":"
    Converts the event to IDEA format and saves it as JSON in the field output. All other fields are not modified.
     
    Documentation about IDEA: https://idea.cesnet.cz/en/index
     
    Module: intelmq.bots.experts.idea.expert
     
    Parameters:
     
    test_mode
     
    (optional, boolean) Adds Test category to mark all outgoing IDEA events as informal (meant to simplify setting up and debugging new IDEA producers). Defaults to true.
    "},{"location":"user/bots/#jinja2-template","title":"Jinja2 Template","text":"
    This bot lets you modify the content of your IntelMQ message fields using Jinja2 templates.
     
    Documentation about Jinja2 templating language: https://jinja.palletsprojects.com/
     
    Module: intelmq.bots.experts.jinja.expert
     
    Parameters:
     
    fields
     
    (required, object) a dict containing as key the name of the field where the result of the Jinja2 template should be written to and as value either a Jinja2 template or a filepath to a Jinja2 template file (starting with file:///). Because the experts decides if it is a filepath based on the value starting with file:/// it is not possible to simply write values starting with file:/// to fields. The object containing the existing message will be passed to the Jinja2 template with the name msg.
     
    fields:\n  output: The provider is {{ msg['feed.provider'] }}!\n  feed.url: \"{{ msg['feed.url'] | upper }}\"\n  extra.somejinjaoutput: file:///etc/intelmq/somejinjatemplate.j2\n
    "},{"location":"user/bots/#lookyloo","title":"Lookyloo","text":"
    Lookyloo is a website screenshotting and analysis tool. For more information and installation instructions visit https://www.lookyloo.eu/
     
    The bot sends a request for source.url to the configured Lookyloo instance and saves the retrieved website screenshot link in the field screenshot_url. Lookyloo only queues the website for screenshotting, therefore the screenshot may not be directly ready after the bot requested it. The pylookyloo library is required for this bot. The http_user_agent parameter is passed on, but not other HTTP-related parameter like proxies.
     
    Events without source.url are ignored.
     
    Module: intelmq.bots.experts.lookyloo.expert
     
    Parameters:
     
    instance_url
     
    (required, string) LookyLoo instance to connect to.
    "},{"location":"user/bots/#maxmind-geoip","title":"MaxMind GeoIP","text":"
    This bot uses an offline database for adding geolocation information based on the IP address (source.ip and destination.ip).
     
    Requirements
     
    The bot requires the MaxMind's geoip2 Python library, version 2.2.0 has been tested.
     
    To download the database a free license key is required. More information can be found at https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/.
     
    Module: intelmq.bots.experts.maxmind_geoip.expert
     
    Parameters:
     
    database
     
    (required, string) Path to the local database file.
     
    overwrite
     
    (optional, boolean) Whether to overwrite existing fields. Defaults to true.
     
    use_registered
     
    (optional, boolean) MaxMind has two country ISO codes: One for the physical location of the address and one for the registered location. See also https://github.com/certtools/intelmq/pull/1344 for a short explanation. Defaults to false (backwards-compatibility).
     
    license_key
     
    (required, string) MaxMind license key is necessary for downloading the GeoLite2 database.
     
    Database
     
    Use this command to create/update the database and reload the bot:
     
    intelmq.bots.experts.maxmind_geoip.expert --update-database\n
    "},{"location":"user/bots/#misp_1","title":"MISP","text":"
    Queries a MISP instance for the source.ip and adds the MISP Attribute UUID and MISP Event ID of the newest attribute found.
     
    Module: intelmq.bots.experts.misp.expert
     
    Parameters:
     
    misp_key
     
    (required, string) MISP Authkey.
     
    misp_url
     
    (required, string) URL of MISP server (with trailing '/')
     
    http_verify_cert
     
    (optional, boolean) Verify the TLS certificate of the server. Default to true.
    "},{"location":"user/bots/#mcafee-active-response-lookup","title":"McAfee Active Response Lookup","text":"
    Queries DXL bus for hashes, IP addresses or FQDNs.
     
    Module: intelmq.bots.experts.mcafee.expert_mar
     
    Parameters:
     
    dxl_config_file
     
    (required, string) Location of the file containing required information to connect to DXL bus.
     
    lookup_type
     
    (required, string) Allowed values:
     
       
    • Hash - Looks up malware.hash.md5, malware.hash.sha1 and malware.hash.sha256.
      •  
    • DestSocket - Looks up destination.ip and destination.port.
      •  
    • DestIP - Looks up destination.ip.
      •  
    • DestFQDN - Looks up in destination.fqdn.
      •  
    "},{"location":"user/bots/#modify","title":"Modify","text":"
    This bots allows you to change arbitrary field values of events using a configuration file.
     
    Module: intelmq.bots.experts.modify.expert
     
    Parameters:
     
    configuration_path
     
    (required, string) Location of the configuration file.
     
    case_sensitive
     
    (optional, boolean) Defaults to true.
     
    maximum_matches
     
    (optional, boolean) Maximum number of matches. Processing stops after the limit is reached. Defaults to null (no limit).
     
    overwrite
     
    (optional, boolean) Overwrite any existing fields by matching rules. Defaults to false.
     
    Configuration File
     
    The modify expert bot allows you to change arbitrary field values of events just using a configuration file. Thus it is possible to adapt certain values or adding new ones only by changing JSON-files without touching the code of many other bots.
     
    The configuration is called modify.conf and looks like this:
     
    [\n  {\n    \"rulename\": \"Standard Protocols http\",\n    \"if\": {\n      \"source.port\": \"^(80|443)$\"\n    },\n    \"then\": {\n      \"protocol.application\": \"http\"\n    }\n  },\n  {\n    \"rulename\": \"Spamhaus Cert conficker\",\n    \"if\": {\n      \"malware.name\": \"^conficker(ab)?$\"\n    },\n    \"then\": {\n      \"classification.identifier\": \"conficker\"\n    }\n  },\n  {\n    \"rulename\": \"bitdefender\",\n    \"if\": {\n      \"malware.name\": \"bitdefender-(.*)$\"\n    },\n    \"then\": {\n      \"malware.name\": \"{matches[malware.name][1]}\"\n    }\n  },\n  {\n    \"rulename\": \"urlzone\",\n    \"if\": {\n      \"malware.name\": \"^urlzone2?$\"\n    },\n    \"then\": {\n      \"classification.identifier\": \"urlzone\"\n    }\n  },\n  {\n    \"rulename\": \"default\",\n    \"if\": {\n      \"feed.name\": \"^Spamhaus Cert$\"\n    },\n    \"then\": {\n      \"classification.identifier\": \"{msg[malware.name]}\"\n    }\n  }\n]\n
     
    In our example above we have five groups labeled Standard Protocols http, Spamhaus Cert conficker, bitdefender, urlzone and default. All sections will be considered, in the given order (from top to bottom).
     
    Each rule consists of conditions and actions. Conditions and actions are dictionaries holding the field names of events and regular expressions to match values (selection) or set values (action). All matching rules will be applied in the given order. The actions are only performed if all selections apply.
     
    If the value for a condition is an empty string, the bot checks if the field does not exist. This is useful to apply default values for empty fields.
     
    Actions
     
    You can set the value of the field to a string literal or number.
     
    In addition you can use the standard Python string format syntax to access the values from the processed event as msg and the match groups of the conditions as matches, see the bitdefender example above. Group 0 ([0]) contains the full matching string. See also the documentation on re.Match.group.
     
    Note that matches will also contain the match groups from the default conditions if there were any.
     
    Examples
     
    We have an event with feed.name = Spamhaus Cert and malware.name = confickerab. The expert loops over all sections in the file and eventually enters section Spamhaus Cert. First, the default condition is checked, it matches! OK, going on. Otherwise the expert would have selected a different section that has not yet been considered. Now, go through the rules, until we hit the rule conficker. We combine the conditions of this rule with the default conditions, and both rules match! So we can apply the action: classification.identifier is set to conficker, the trivial name.
     
    Assume we have an event with feed.name = Spamhaus Cert and malware.name = feodo. The default condition matches, but no others. So the default action is applied. The value for classification.identifier will be set to feodo by {msg[malware.name]}.
     
    Types
     
    If the rule is a string, a regular expression search is performed, also for numeric values (str() is called on them). If the rule is numeric for numeric values, a simple comparison is done. If other types are mixed, a warning will be thrown.
     
    For boolean values, the comparison value needs to be true or false as in JSON they are written all-lowercase.
    "},{"location":"user/bots/#national-cert-contact-lookup-by-certat","title":"National CERT Contact Lookup by CERT.AT","text":"
    https://contacts.cert.at offers an IP address to national CERT contact (and cc) mapping.
     
    Module: intelmq.bots.experts.national_cert_contact_certat.expert
     
    Parameters:
     
    filter
     
    (optional, boolean) Whether to act as a filter for AT.
     
    overwrite_cc
     
    (optional, boolean) Set to true if you want to overwrite any potentially existing cc fields in the event. Defaults to false.
    "},{"location":"user/bots/#rdap","title":"RDAP","text":"
    This bot queries RDAP servers for additional information about a domain.
     
    Module: intelmq.bots.experts.rdap.expert
     
    Parameters:
     
    rdap_order
     
    (optional, array of strings) Search order of contacts with these roles. Defaults to [\"abuse\", \"technical\"].
     
    rdap_bootstrapped_servers
     
    (optional, object) Customized RDAP servers. Do not forget the trailing slash. For example:
     
    {\n  \"at\": {\n    \"url\": \"rdap.server.at/v1/\",\n    \"auth\": {\n      \"type\": \"jwt\",\n      \"token\": \"ey...\"\n    }\n  },\n  \"de\": \"rdap.service:1337/v1/\"\n}\n
    "},{"location":"user/bots/#recordedfuture-ip-risk","title":"RecordedFuture IP Risk","text":"
    This bot tags events with the score found in RecordedFuture large IP risklist.
     
    Record risk score associated to source and destination IP if they are present. Assigns 0 to IP addresses not in the RF list.
     
    For both source.ip and destination.ip the corresponding risk score is fetched from a local database created from RecordedFuture's API. The score is recorded in extra.rf_iprisk.source and extra.rf_iprisk.destination. If a lookup for an IP fails a score of 0 is recorded.
     
    See https://www.recordedfuture.com/products/api/ and speak with your RecordedFuture representative for more information.
     
    The list is obtained from recorded future API and needs a valid API TOKEN The large list contains all IP's with a risk score of 25 or more. If IP's are not present in the database a risk score of 0 is given.
     
    Module: intelmq.bots.experts.recordedfuture_iprisk.expert
     
    Parameters:
     
    database
     
    (required, string) Path to the local database file.
     
    api_token
     
    (required, string) This needs to contain valid API token to download the latest database data.
     
    overwrite
     
    (optional, boolean) Whether to overwrite existing fields. Defaults to false.
     
    Database
     
    Use this command to create/update the database and reload the bot:
     
    intelmq.bots.experts.recordedfuture_iprisk.expert --update-database\n
    "},{"location":"user/bots/#reverse-dns","title":"Reverse DNS","text":"
    For both source.ip and destination.ip the PTR record is fetched and the first valid result is used for source.reverse_dns or destination.reverse_dns.
     
    Module: intelmq.bots.experts.reverse_dns.expert
     
    Parameters (also expects cache parameters):
     
    cache_ttl_invalid_response
     
    (required, integer) The TTL for cached invalid responses.
     
    overwrite
     
    (optional, boolean) Whether to overwrite existing fields. Defaults to false.
    "},{"location":"user/bots/#rfc1918","title":"RFC1918","text":"
    Several RFCs define ASNs, IP Addresses and Hostnames (and TLDs) reserved for documentation. Events or fields of events can be dropped if they match the criteria of either being reserved for documentation (e.g. AS 64496, Domain example.com) or belonging to a local area network (e.g. 192.168.0.0/24). These checks can applied to URLs, IP Addresses, FQDNs and ASNs.
     
    It is configurable if the whole event should be dropped (\"policies\") or just the field removed, as well as which fields should be checked.
     
    Sources:
     
       
    • 1918
      •  
    • 2606
      •  
    • 3849
      •  
    • 4291
      •  
    • 5737
      •  
    • https://en.wikipedia.org/wiki/IPv4
      •  
    • https://en.wikipedia.org/wiki/Autonomous_system_(Internet)
      •  
     
    Module: intelmq.bots.experts.rfc1918.expert
     
    Parameters:
     
    fields
     
    (required, string) Comma-separated list of fields. Allowed values:
     
       
    • destination.asn & source.asn
      •  
    • destination.fqdn & source.fqdn
      •  
    • destination.ip & source.ip
      •  
    • destination.url & source.url
      •  
     
    policy
     
    (required, string) Comma-separated list of policies. Allowed values:
     
       
    • drop - the entire events is dropped
      •  
    • del - the affected field is removed
      •  
     
    With the example parameter values given above, this means that:
     
       
    • If a destination.ip value is part of a reserved network block, the field will be removed (policy del).
      •  
    • If a source.asn value is in the range of reserved AS numbers, the event will be removed altogether (policy drop).
      •  
    • If a source.url value contains a host with either an IP address part of a reserved network block, or a reserved   domain name (or with a reserved TLD), the event will be dropped (policy drop).
      •  
    "},{"location":"user/bots/#ripe","title":"RIPE","text":"
    Online RIPE Abuse Contact and Geolocation Finder for IP addresses and Autonomous Systems.
     
    Module: intelmq.bots.experts.ripe.expert
     
    Parameters (also expects cache parameters):
     
    mode
     
    (optional, string) Allowed values: append or replace. Defaults to append.
     
    query_ripe_db_asn
     
    (optional, boolean) Query for IPs at http://rest.db.ripe.net/abuse-contact/%s.json. Defaults to true.
     
    query_ripe_db_ip
     
    (optional, boolean) Query for ASNs at http://rest.db.ripe.net/abuse-contact/as%s.json. Defaults to true.
     
    query_ripe_stat_asn
     
    (optional, boolean) Query for ASNs at https://stat.ripe.net/data/abuse-contact-finder/data.json?resource=%s. Defaults to true.
     
    query_ripe_stat_ip
     
    (optional, boolean) Query for IPs at https://stat.ripe.net/data/abuse-contact-finder/data.json?resource=%s. Defaults to true.
     
    query_ripe_stat_geolocation
     
    (optional, boolean) Query for IPs at https://stat.ripe.net/data/maxmind-geo-lite/data.json?resource=%s. Defaults to true.
    "},{"location":"user/bots/#securitytxt","title":"SecurityTXT","text":"
    SecurityTXT is an initiative to standardize how websites publish their abuse contact information. It is standardized in RFC 9116 \"A File Format to Aid in Security Vulnerability Disclosure\". Refer to the linked document RFC for more information on security.txt. This bot looks for security.txt files on a URL or IP, retrieves the primary contact information out of it and adds this to the event.
     
    Requirements
     
    To use this bot, you need to install the required dependencies:
     
    pip3 install -r intelmq/bots/experts/securitytxt/REQUIREMENTS.txt\n
     
    Module: intelmq.bots.experts.securitytxt.expert
     
    Parameters
     
    url_field
     
    The field in the event that contains the URL/IP on which to look for the the security.txt file. Default: source.reverse_dns
     
    contact_field
     
    The field in the event in which to put the found contact details. Default: source.abuse_contact
     
    only_email_address (bool)
     
    Contact details can be web URLs or email addresses. When this value is set to True, it only selects email addresses as contact information. Default: true
     
    overwrite (bool)
     
    Boolean indicating whether to override existing data in contact_field. Default: true
     
    check_expired (bool)
     
    Boolean indicating whether to check if the security.txt has expired according to its own expiry date. Default: false
     
    check_canonical (bool)
     
    Boolean indicating whether to check if the url is contained in the list of canonical urls. Default: false
    "},{"location":"user/bots/#sieve","title":"Sieve","text":"
    This bot is used to filter and/or modify events based on a set of rules. The rules are specified in an external configuration file and with a syntax similar to the Sieve language used for mail filtering.
     
    Each rule defines a set of matching conditions on received events. Events can be matched based on keys and values in the event. Conditions can be combined using parenthesis and the boolean operators && and ||. If the processed event matches a rule's conditions, the corresponding actions are performed. Actions can specify whether the event should be kept or dropped in the pipeline (filtering actions) or if keys and values should be changed (modification actions).
     
    Requirements
     
    To use this bot, you need to install the required dependencies:
     
    pip3 install -r intelmq/bots/experts/sieve/REQUIREMENTS.txt\n
     
    Module: intelmq.bots.experts.sieve.expert
     
    Parameters:
     
    file
     
    (required, string) Path to sieve file. Syntax can be validated with intelmq_sieve_expert_validator.
     
    Examples
     
    The following excerpts illustrate some of the basic features of the sieve file format:
     
    if :exists source.fqdn {\n keep // aborts processing of subsequent rules and forwards the event.\n}\n\n\nif :notexists source.abuse_contact || source.abuse_contact =~ '.*@example.com' {\n drop // aborts processing of subsequent rules and drops the event.\n}\n\nif source.ip << '192.0.0.0/24' {\n add! comment = 'bogon' // sets the field comment to this value and overwrites existing values\n path 'other-path' // the message is sent to the given path\n}\n\nif classification.type :in ['phishing', 'malware-distribution'] && source.fqdn =~ '.*.(ch|li)$' {\n add! comment = 'domainabuse'\n keep\n} elif classification.type == 'scanner' {\n add! comment = 'ignore'\n drop\n} else {\n remove comment\n}\n
     
    Reference
     
    Sieve File Structure
     
    The sieve file contains an arbitrary number of rules of the form:
     
    if EXPRESSION {\n ACTIONS\n} elif EXPRESSION {\n ACTIONS\n} else {\n ACTIONS\n}\n
     
    Nested if-statements and mixed if statements and rules in the same scope are possible.
     
    Expressions
     
    Each rule specifies on or more expressions to match an event based on its keys and values. Event keys are specified as strings without quotes. String values must be enclosed in single quotes. Numeric values can be specified as integers or floats and are unquoted. IP addresses and network ranges (IPv4 and IPv6) are specified with quotes. List values for use with list/set operators are specified as string, float, int, bool and string literals separated by commas and enclosed in square brackets. Expression statements can be combined and chained using parentheses and the boolean operators && and ||. The following operators may be used to match events:
     
       
    • :exists and :notexists match if a given key exists, for example:
      •  
     
    if :exists source.fqdn { ... }\n
     
       
    • == and != match for equality of strings, numbers, and booleans, for example:
      •  
     
    if feed.name != 'acme-security' || feed.accuracy == 100 || extra.false_positive == false { ... }\n
     
       
    •  
      :contains matches on substrings (str.find).
       
      •  
    •  
      =~ matches strings based on the given regular expression. !~ is the inverse regular expression match.
       
      •  
    •  
      For :contains, =~ and !~, the value is converted to string before matching. If the value is a dict, convert the value to JSON.
       
      •  
    •  
      Numerical comparisons are evaluated with <, <=, >, >=.
       
      •  
    •  
      << matches if an IP address is contained in the specified network range:
       
      •  
     
    if source.ip << '10.0.0.0/8' { ... }\n
     
       
    • String values to match against can also be specified as lists of strings, which have separate operators. For example:
      •  
     
    if source.ip :in ['8.8.8.8', '8.8.4.4'] { ... }\n
     
    In this case, the event will match if it contains a key source.ip with either value 8.8.8.8 or 8.8.4.4.
     
    There are also :containsany to match at least one of a list of substrings, and :regexin to match at least one of a list of regular expressions, similar to the :contains and =~ operators.
     
       
    • Lists of numeric values support :in to check for inclusion in a list of numbers:
      •  
     
    if source.port :in [80, 443] { ... }\n
     
       
    • :equals tests for equality between lists, including order. Example for checking a hostname-port pair:
      •  
     
    if extra.host_tuple :equals ['dns.google', 53] { ... }\n
     
       
    • :setequals tests for set-based equality (ignoring duplicates and value order) between a list of given values.   Example for checking for the first nameserver of two domains, regardless of the order they are given in the list:
      •  
     
    if extra.hostnames :setequals ['ns1.example.com', 'ns1.example.mx'] { ... }\n
     
       
    • :overlaps tests if there is at least one element in common between the list specified by a key and a list of values.   Example for checking if at least one of the ICS, database or vulnerable tags is given:
      •  
     
    if extra.tags :overlaps ['ics', 'database', 'vulnerable'] { ... }\n
     
       
    • :subsetof tests if the list of values from the given key only contains values from a set of values specified as the   argument. Example for checking for a host that has only ns1.example.com and/or ns2.* as its apparent hostname:
      •  
     
    if extra.hostnames :subsetof ['ns1.example.com', 'ns2.example.com'] { ... }\n
     
       
    • :supersetof tests if the list of values from the given key is a superset of the values specified as the argument.   Example for matching hosts with at least the IoT and vulnerable tags:
      •  
     
    if extra.tags :supersetof ['iot', 'vulnerable'] { ... }\n
     
       
    • :before tests if the date value occurred before given time ago. The time might be absolute (basically anything parseable by pendulum parser, eg. \u201c2015-09-12T06:22:11+00:00\u201d) or relative (accepted string formatted like this \u201c \u201d, where epoch could be any of following strings (could optionally end with trailing \u2018s\u2019): hour, day, week, month, year) 
      if time.observation :before '1 week' { ... }\n
       
         
      • :after  tests if the date value occurred after given time ago; see :before
        •  
       
      if time.observation :after '2015-09-12' { ... }  # happened after midnight the 12th Sep\n
       
         
      • Boolean values can be matched with == or != followed by true or false. Example:
        •  
       
      if extra.has_known_vulns == true { ... }\n
       
         
      • The combination of multiple expressions can be done using parenthesis and boolean operators:
        •  
       
      if (source.ip == '127.0.0.1') && (comment == 'add field' || classification.taxonomy == 'vulnerable') { ... }\n
       
         
      • Any single expression or a parenthesised group of expressions can be negated using !:
        •  
       
      if ! source.ip :contains '127.0.0.' || ! ( source.ip == '172.16.0.5' && source.port == 25 ) { ... }\n
       
      !!! note Since 3.0.0, list-based operators are used on list values, such as foo :in [1, 2, 3] instead of foo == [1, 2, 3] and foo :regexin ['.mx', '.zz'] rather than foo =~ ['.mx', '.zz'], and similarly for :containsany vs :contains. Besides that, :notcontains has been removed, with e.g foo :notcontains ['.mx', '.zz'] now being represented using negation as ! foo :contains ['.mx', '.zz'].
       
      Actions
       
      If part of a rule matches the given conditions, the actions enclosed in { and } are applied. By default, all events that are matched or not matched by rules in the sieve file will be forwarded to the next bot in the pipeline, unless the drop action is applied.
       
         
      • add adds a key value pair to the event. It can be a string, number, or boolean. This action only applies if the key   is not yet defined in the event. If the key is already defined, the action is ignored. Example:
        •  
       
      add comment = 'hello, world'\n
       
      Some basic mathematical expressions are possible, but currently support only relative time specifications objects are supported. For example:
       
      add time.observation += '1 hour'\nadd time.observation -= '10 hours'\n
       
         
      •  
        add! same as above, but will force overwrite the key in the event.
         
        •  
      •  
        update modifies an existing value for a key. Only applies if the key is already defined. If the key is not defined   in the event, this action is ignored. This supports mathematical expressions like above. Example:
         
        •  
       
      update feed.accuracy = 50\n
       
      Some basic mathematical expressions are possible, but currently support only relative time specifications objects are supported. For example:
       
      update time.observation += '1 hour'\nupdate time.observation -= '10 hours'\n
       
         
      • remove removes a key/value from the event. Action is ignored if the key is not defined in the event. Example:
        •  
       
      remove extra.comments\n
       
         
      •  
        keep sends the message to the next bot in the pipeline (same as the default behaviour), and stops sieve rules   processing.
         
        •  
      •  
        path sets the path (named queue) the message should be sent to (implicitly or with the command keep. The named   queue needs to configured in the pipeline, see the User Guide for more information.
         
        •  
       
      path 'named-queue'\n
       
      You can as well set multiple destination paths with the same syntax as for value lists:
       
      path ['one', 'two']\n
       
      This will result in two identical message, one sent to the path one and the other sent to the path two.
       
      If the path is not configured, the error looks like:
       
      File \"/path/to/intelmq/intelmq/lib/pipeline.py\", line 353, in send for destination_queue in self.destination_queues path]: KeyError: 'one'\n
       
         
      • drop marks the event to be dropped. The event will not be forwarded to the next bot in the pipeline. The sieve file   processing is interrupted upon reaching this action. No other actions may be specified besides the drop action   within { and }.
        •  
       
      Comments
       
      Comments may be used in the sieve file: all characters after // and until the end of the line will be ignored.
      "},{"location":"user/bots/#splunk-saved-search-lookup","title":"Splunk Saved Search Lookup","text":"
      Runs a saved search in Splunk using fields in an event, adding fields from the search result into the event.
       
      Splunk documentation on saved searches: https://docs.splunk.com/Documentation/Splunk/latest/Report/Createandeditreports
       
      The saved search should take parameters according to the search_parameters configuration and deliver results according to result_fields. The examples above match a saved search of this format:
       
      index=\"dhcp\" ipv4address=\"$ip$\" | ... | fields _time username ether\n
       
      The time window used is the one saved with the search.
       
      Waits for Splunk to return an answer for each message, so slow searches will delay the entire botnet. If you anticipate a load of more than one search every few seconds, consider running multiple load-balanced copies of this bot.
       
      Module: intelmq.bots.experts.splunk_saved_search.expert
       
      Parameters (also expects HTTP parameters):
       
      auth_token
       
      (required, string) Splunk API authentication token.
       
      url
       
      (required, string) base URL of the Splunk REST API.
       
      retry_interval
       
      (optional, integer) Number of seconds to wait between polling for search results to be available. Defaults to 5.
       
      saved_search
       
      (required, string) Name of Splunk saved search to run.
       
      search_parameters
       
      (optional, object) Mapping of IntelMQ event fields containing the data to search for to parameters of the Splunk saved search. Defaults to {}. Example:
       
      search_parameters:\n  source.ip: ip\n
       
      result_fields
       
      (optional, object) Mapping of Splunk saved search result fields to IntelMQ event fields to store the results in. Defaults to {}. Example:
       
      result_fields:\n  username: source.account\n
       
      not_found
       
      (optional, array of strings) How to handle empty search results. Allowed values:
       
         
      • warn - log a warning message
        •  
      • send - send the event on unmodified
        •  
      • drop - drop the message
        •  
      • send - and drop are mutually exclusive
        •  
       
      All specified actions are performed. Defaults to [ \"warn\", \"send\" ].
       
      multiple_result_handling
       
      (optional, array of strings) How to handle more than one search result. Allowed values:
       
         
      • limit - limit the search so that duplicates are impossible
        •  
      • warn - log a warning message
        •  
      • use_first - use the first search result
        •  
      • ignore - do not modify the event
        •  
      • send - send the event on
        •  
      • drop - drop the message
        •  
      • limit cannot be combined with any other value
        •  
      • send and drop are mutually exclusive
        •  
      • ignore and use_first are mutually exclusive
        •  
       
      All specified actions are performed. Defaults to [\"warn\", \"use_first\", \"send\" ].
       
      overwrite
       
      (optional, boolean/null) Whether search results overwrite values already in the message or not. If null, attempting to add a field that already exists throws an exception. Defaults to null.
      "},{"location":"user/bots/#taxonomy","title":"Taxonomy","text":"
      This bot adds the classification.taxonomy field according to the RSIT taxonomy.
       
      Please note that there is a slight mismatch of IntelMQ's taxonomy to the upstream taxonomy. See also this issue.
       
      Information on the \"Reference Security Incident Taxonomy\" can be found here: https://github.com/enisaeu/Reference-Security-Incident-Taxonomy-Task-Force
       
      For brevity, \"type\" means classification.type and \"taxonomy\" means classification.taxonomy.
       
         
      • If taxonomy is missing, and type is given, the according taxonomy is set.
        •  
      • If neither taxonomy, not type is given, taxonomy is set to \"other\" and type to \"unknown\".
        •  
      • If taxonomy is given, but type is not, type is set to \"unknown\".
        •  
       
      Module: intelmq.bots.experts.taxonomy.expert
       
      No additional parameters.
      "},{"location":"user/bots/#threshold","title":"Threshold","text":"
      Check if the number of similar messages during a specified time interval exceeds a set value.
       
      Limitations
       
      This bot has certain limitations and is not a true threshold filter (yet). It works like this:
       
         
      1. Every incoming message is hashed according to the filter_* parameters.
        2.  
      2. The hash is looked up in the cache and the count is incremented by 1, and the TTL of the key is (re-)set to the    timeout.
        3.  
      3. If the new count matches the threshold exactly, the message is forwarded. Otherwise it is dropped.
        4.  
       
      Note
       
      Even if a message is sent, any further identical messages are dropped, if the time difference to the last message is less than the timeout! The counter is not reset if the threshold is reached.
       
      Module: intelmq.bots.experts.threshold.expert
       
      Parameters (also expects cache parameters):
       
      filter_keys
       
      (required, string/array of strings) Array or comma-separated list of field names to consider or ignore when determining which messages are similar.
       
      filter_type
       
      (required, string) Allowed values: whitelist or blacklist. When whitelist is used, only lines containing the text specified in filter_text option will be processed. When blacklist is used, only lines NOT containing the text will be processed.
       
      threshold
       
      (required, integer) Number of messages required before propagating one. In forwarded messages, the threshold is saved in the message as extra.count.
       
      add_keys
       
      (optional, object) List of keys and their respective values to add to the propagated messages. Example:
       
      add_keys:\n  classification.type: \"spam\"\n  comment: \"Started more than 10 SMTP connections\"\n
      "},{"location":"user/bots/#tor-exit-node","title":"Tor Exit Node","text":"
      This bot uses an offline database to determine whether the host is a Tor exit node.
       
      Module: intelmq.bots.experts.tor_nodes.expert
       
      Parameters:
       
      database
       
      (required, string) Path to the database file.
       
      Database
       
      Use this command to create/update the database and reload the bot:
       
      intelmq.bots.experts.tor_nodes.expert --update-database\n
      "},{"location":"user/bots/#trusted-introducer-lookup","title":"Trusted Introducer Lookup","text":"
      Lookups data from Trusted Introducer public teams list.
       
      Module: intelmq.bots.experts.trusted_introducer_lookup.expert
       
      Parameters:
       
      order
       
      (required, string) Allowed values: domain and asn. You can set multiple values, so first match wins.
       
         
      • When domain is set, it will lookup the source.fqdn field. It will go from high-order to low-order, i.e.   1337.super.example.com -> super.example.com -> example.com -> .com
        •  
      • If asn is set, it will lookup source.asn.
        •  
       
      After a match, the abuse contact will be fetched from the trusted introducer teams list and will be stored in the event as source.abuse_contact. If there is no match, the event will not be enriched and will be sent to the next configured step.
      "},{"location":"user/bots/#tuency","title":"Tuency","text":"
      Queries the IntelMQ API of a Tuency Contact Database instance.
       
      Tuency is a contact management database addressing the needs of CERTs. Users of tuency can configure contact addresses and delivery settings for IP objects (addresses, netblocks), Autonomous Systems, and (sub-)domains. This expert queries the information for source.ip and source.fqdn using the following other fields:
       
         
      • classification.taxonomy
        •  
      • classification.type
        •  
      • feed.provider
        •  
      • feed.name
        •  
       
      These fields therefore need to exist, otherwise the message is skipped.
       
      The API parameter \"feed_status\" is currently set to \"production\" constantly, until IntelMQ supports this field.
       
      The API answer is processed as following. For the notification interval:
       
         
      • If suppress is true, then extra.notify is set to false.
        •  
      • Otherwise:
        •  
      • If the interval is immediate, then extra.ttl is set to 0.
        •  
      • Otherwise the interval is converted into seconds and saved in   extra.ttl.
        •  
       
      For the contact lookup: For both fields ip and domain, the destinations objects are iterated and its email fields concatenated to a comma-separated list in source.abuse_contact.
       
      The IntelMQ fields used by this bot may change in the next IntelMQ release, as soon as better suited fields are available.
       
      Module: intelmq.bots.experts.tuency.expert
       
      Parameters:
       
      url
       
      (required, string) Tuency instance URL. Without the API path.
       
      authentication_token
       
      (required, string) The Bearer authentication token. Without the Bearer prefix.
       
      overwrite
       
      (optional, boolean) Whether the existing data in source.abuse_contact should be overwritten. Defaults to true.
      "},{"location":"user/bots/#truncate-by-delimiter","title":"Truncate By Delimiter","text":"
      Cut string if length is bigger than maximum length.
       
      Module: intelmq.bots.experts.truncate_by_delimiter.expert
       
      Parameters:
       
      delimiter
       
      (required, string) The delimiter to be used for truncating. Defaults to ..
       
      max_length
       
      (required, integer) The maximum string length.
       
      field
       
      (required, string) The field to be truncated, e.g. source.fqdn. The given field is truncated step-by-step using the delimiter from the beginning, until the field is shorter than max_length.
       
      Example: Cut through a long domain with a dot. The string is truncated until the domain does not exceed the configured maximum length.
       
         
      • Input domain (e.g. source.fqdn): www.subdomain.web.secondsubomain.test.domain.com
        •  
      • delimiter: .
        •  
      • max_length: 20
        •  
      • Resulting value test.domain.com (length: 15 characters)
        •  
      "},{"location":"user/bots/#url","title":"URL","text":"
      This bot extracts additional information from source.url and destination.url fields. It can fill the following fields:
       
         
      • source.fqdn
        •  
      • source.ip
        •  
      • source.port
        •  
      • source.urlpath
        •  
      • source.account
        •  
      • destination.fqdn
        •  
      • destination.ip
        •  
      • destination.port
        •  
      • destination.urlpath
        •  
      • destination.account
        •  
      • protocol.application
        •  
      • protocol.transport
        •  
       
      Module: intelmq.bots.experts.url.expert
       
      Parameters:
       
      overwrite
       
      (optional, boolean) Whether to overwrite existing fields. Defaults to false.
       
      skip_fields
       
      (optional, array of string) An array of field names that shouldn't be extracted from the URL.
      "},{"location":"user/bots/#url2fqdn","title":"Url2FQDN","text":"
      This bot is deprecated and will be removed in version 4.0. Use URL Expert bot instead.
       
      This bot extracts the Host from the source.url and destination.url fields and writes it to source.fqdn or destination.fqdn if it is a hostname, or source.ip or destination.ip if it is an IP address.
       
      Module: intelmq.bots.experts.url2fqdn.expert
       
      Parameters:
       
      overwrite
       
      (optional, boolean) Whether to overwrite existing fields. Defaults to false.
      "},{"location":"user/bots/#uwhoisd","title":"uWhoisd","text":"
      uWhoisd is a universal Whois server that supports caching and stores whois entries for historical purposes.
       
      The bot sends a request for source.url, source.fqdn, source.ip or source.asn to the configured uWhoisd instance and saves the retrieved whois entry:
       
         
      • If both source.url and source.fqdn are present, it will only do a request for source.fqdn, as the hostname of source.url should be the same as source.fqdn. The whois entry will be saved in extra.whois.fqdn.
        •  
      • If source.ip is present, the whois entry will be saved in extra.whois.ip.
        •  
      • If source.asn is present, he whois entry will be saved in extra.whois.asn.
        •  
       
      Events without source.url, source.fqdn, source.ip, or source.asn, are ignored.
       
      Note
       
      Requesting a whois entry for a fully qualified domain name (FQDN) only works if the request only contains the domain. uWhoisd will automatically strip the subdomain part if it is present in the request.
       
      Example: https://www.theguardian.co.uk
       
         
      • TLD: co.uk (uWhoisd uses the Mozilla public suffix list as a reference)
        •  
      • Domain: theguardian.co.uk
        •  
      • Subdomain: www
        •  
       
      The whois request will be for theguardian.co.uk
       
      Module: intelmq.bots.experts.uwhoisd.expert
       
      Parameters:
       
      server
       
      (optional, string) Hostname of the uWhoisd server. Defaults to localhost.
       
      port
       
      (optional, integer) Port of the uWhoisd server. Defaults to 4243.
      "},{"location":"user/bots/#wait","title":"Wait","text":"
      Waits for a some time or until a queue size is lower than a given number.
       
      Only one of the two modes is possible. If a queue name is given, the queue mode is active. If the sleep_time is a number, sleep mode is active. Otherwise the dummy mode is active, the events are just passed without an additional delay.
       
      Note that SIGHUPs and reloads interrupt the sleeping.
       
      Module: intelmq.bots.experts.wait.expert
       
      Parameters:
       
      queue_db
       
      (optional, integer) Database number of the database. Defaults to 2.
       
      queue_host
       
      (optional, string) Hostname of the database. Defaults to localhost.
       
      queue_name
       
      (optional, string) Name of the queue to be watched. This is not the name of a bot but the queue's name. Defaults to null.
       
      queue_password
       
      (optional, string) Password for the database. Defaults to null.
       
      queue_polling_interval
       
      (required, float) Interval to poll the list length in seconds. Defaults to ?.
       
      queue_port
       
      (optional, integer) Port of the database. Defaults to 6379.
       
      queue_size
       
      (optional, integer) Maximum size of the queue. Defaults to 0.
       
      sleep_time
       
      (optional, integer) Time to sleep before sending the event. Defaults to null.
      "},{"location":"user/bots/#output-bots","title":"Output Bots","text":""},{"location":"user/bots/#amqp-topic","title":"AMQP Topic","text":"
      Sends the event to a specified topic of an AMQP server
       
      Sends data to an AMQP Server See https://www.rabbitmq.com/tutorials/amqp-concepts.html for more details on amqp topic exchange.
       
      Requires the pika python library.
       
      Module: intelmq.bots.outputs.amqptopic.output
       
      Parameters:
       
      connection_attempts
       
      (optional, integer) The number of connection attempts to defined server. Defaults to 3.
       
      connection_heartbeat
       
      (optional, integer) Heartbeat to server (in seconds). Defaults to 3600.
       
      connection_host
       
      (optional, string) Hostname of the AMQP server. Defaults to 127.0.0.1.
       
      connection_port
       
      (optional, integer) Port of the AMQP server. Defaults to 5672.
       
      connection_vhost
       
      (optional, string) Virtual host to connect, on an http(s) connection would be http://IP/<your virtual host>.
       
      content_type
       
      (optional, string) Content type to deliver to AMQP server. Currently only supports application/json.
       
      delivery_mode
       
      (optional, integer) Allowed values:
       
         
      • 1 - Non-persistent delivery.
        •  
      • 2 - Persistent delivery. Messages are delivered to 'durable' queues and will be saved to disk.
        •  
       
      exchange_durable
       
      (optional, boolean) When set to true, the exchange will survive broker restart, otherwise will be a transient exchange.
       
      exchange_name
       
      (optional, string) The name of the exchange to use.
       
      exchange_type
       
      (optional, string) Type of the exchange, e.g. topic, fanout etc.
       
      keep_raw_field
       
      (optional, boolean) Whether to keep the raw field or not. Defaults to false.
       
      password
       
      (optional, boolean) Password for authentication on your AMQP server. Leave empty if authentication is not required.
       
      require_confirmation
       
      (optional, boolean) If set to True, an exception will be raised if a confirmation error is received.
       
      routing_key
       
      (required, string) The routing key for your amqptopic.
       
      single_key
       
      (optional, boolean) Only send the field instead of the full event (expecting a field name as string). Defaults to false.
       
      username
       
      (required, string) Username for authentication on your AMQP server.
       
      use_ssl
       
      (optional, boolean) Use ssl for the connection, make sure to also set the correct port, usually 5671. Defaults to false.
       
      message_hierarchical_output
       
      (optional, boolean) Convert the message to hierarchical JSON. Defaults to false.
       
      message_with_type
       
      (optional, boolean) Whether to include the type in the sent message. Defaults to false.
       
      message_jsondict_as_string
       
      (optional, boolean) Whether to convert JSON fields (extra) to string. Defaults to false.
       
      Examples of usage
       
         
      • Useful to send events to a RabbitMQ exchange topic to be further processed in other platforms.
        •  
       
      Confirmation
       
      If routing key or exchange name are invalid or non existent, the message is accepted by the server but we receive no confirmation. If parameter require_confirmation is True and no confirmation is received, an error is raised.
       
      Common errors
       
      Unroutable messages / Undefined destination queue
       
      The destination exchange and queue need to exist beforehand, with your preferred settings (e.g. durable, lazy queue. If the error message says that the message is \"unroutable\", the queue doesn't exist.
      "},{"location":"user/bots/#blackhole","title":"Blackhole","text":"
      This bot discards all incoming messages.
       
      Module: intelmq.bots.outputs.blackhole.output
       
      No additional parameters.
      "},{"location":"user/bots/#bro-file","title":"Bro File","text":"
      This bot outputs to BRO (zeek) file.
       
      File example:
       
      #fields indicator indicator_type meta.desc meta.cif_confidence meta.source xxx.xxx.xxx.xxx Intel::ADDR phishing 100 MISP XXX www.testdomain.com Intel::DOMAIN apt 85 CERT\n
       
      Module: intelmq.bots.outputs.bro_file.output
       
      No additional parameters.
      "},{"location":"user/bots/#cifv3-api","title":"CIFv3 API","text":"
      This bot outputs to a CIFv3 API instance and adds new indicator if not there already.
       
      By default, CIFv3 does an upsert check and will only insert entirely new indicators. Otherwise, upsert matches will have their count increased by 1. By default, the CIF3 output bot will batch indicators up to 500 at a time prior to doing a single bulk send. If the output bot doesn't receive a full 500 indicators within 5 seconds of the first received indicator, it will send what it has so far.
       
      CIFv3 should be able to process indicators as fast as IntelMQ can send them.
       
      Module: intelmq.bots.outputs.cif3.output
       
      Parameters:
       
      add_feed_provider_as_tag
       
      (required, boolean) Use false when in doubt.
       
      cif3_additional_tags
       
      (required, array of strings) An array of tags to set on submitted indicator(s).
       
      cif3_feed_confidence
       
      (required, float) Used when mapping a feed's confidence fails or if static confidence parameter is true.
       
      cif3_static_confidence
       
      (required, boolean) Whether to always use cif3_feed_confidence value as confidence rather than dynamically interpret feed value (use false when in doubt).
       
      cif3_token
       
      (required, string) Token key for accessing CIFv3 API.
       
      cif3_url
       
      (required, string) URL of the CIFv3 instance.
       
      fireball
       
      (required, integer) Used to batch events before submitting to a CIFv3 instance, use 0 to disable batch and send each event as received. Defaults to 500.
       
      http_verify_cert
       
      (optional, boolean) Verify the TLS certificate of the server. Defaults to true.
      "},{"location":"user/bots/#elasticsearch","title":"Elasticsearch","text":"
      This bot outputs to Elasticsearch.
       
      Module: intelmq.bots.outputs.elasticsearch.output
       
         
      • lookup: yes
        •  
      • public: yes
        •  
      • cache: no
        •  
      • description: Output Bot that sends events to Elasticsearch
        •  
       
      Only ElasticSearch version 7 supported.
       
      It is also possible to feed data into ElasticSearch using ELK-Stack via Redis and Logstash, see ELK-Stack {.interpreted-text role=\"doc\"} for more information. This methods supports various different versions of ElasticSearch.
       
      Parameters:
       
      elastic_host
       
      (optional, string) Name/IP for the Elasticsearch server. Defaults to 127.0.0.1.
       
      elastic_port
       
      (optional, int) Port for the Elasticsearch server. Defaults to 9200.
       
      elastic_index
       
      (optional, string) Index for the Elasticsearch output. Defaults to intelmq.
       
      rotate_index
       
      (optional, string) Allowed values: never, daily, weekly, monthly or yearly. If set, will index events using the date information associated with the event. Defaults to never.
       
      Using 'intelmq' as the elastic_index, the following are examples of the generated index names:
       
      'never' --> intelmq\n'daily' --> intelmq-2018-02-02\n'weekly' --> intelmq-2018-42\n'monthly' --> intelmq-2018-02\n'yearly' --> intelmq-2018\n
       
      http_username
       
      (optional, string) HTTP basic authentication username.
       
      http_password
       
      (optional, string) HTTP basic authentication password.
       
      use_ssl
       
      (optional, boolean) Whether to use SSL/TLS when connecting to Elasticsearch. Defaults to false.
       
      http_verify_cert
       
      (optional, boolean) Whether to require verification of the server's certificate. Defaults to false.
       
      ssl_ca_certificate
       
      (optional, string) Path to trusted CA certificate.
       
      ssl_show_warnings
       
      (optional, boolean) Whether to show warnings if the server's certificate cannot be verified. Defaults to true.
       
      replacement_char
       
      (optional, string) If set, dots ('.') in field names will be replaced with this character prior to indexing. This is for backward compatibility with ES 2.X. Defaults to null. Recommended for Elasticsearch 2.X: _
       
      flatten_fields
       
      (optional, array of strings) In ES, some query and aggregations work better if the fields are flat and not JSON. Here you can provide a list of fields to convert. Defaults to ['extra'].
       
      Can be a list of strings (fieldnames) or a string with field names separated by a comma (,). eg extra,field2 or ['extra', 'field2'].
       
      See contrib/elasticsearch/elasticmapper for a utility for creating Elasticsearch mappings and templates.
       
      If using rotate_index, the resulting index name will be of the form elastic_index-event date. To query all intelmq indices at once, use an alias (https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-aliases.html), or a multi-index query.
       
      The data in ES can be retrieved with the HTTP-Interface:
       
       curl -XGET 'http://localhost:9200/intelmq/events/_search?pretty=True'\n
      "},{"location":"user/bots/#file_1","title":"File","text":"
      This bot outputs messages (reports or events) to a file.
       
      Multihreading is disabled for this bot, as this would lead to corrupted files.
       
      Module: intelmq.bots.outputs.file.output
       
      Parameters:
       
      encoding_errors_mode
       
      (optional, string) See for more details and options: https://docs.python.org/3/library/functions.html#open For example with backslashreplace all characters which cannot be properly encoded will be written escaped with backslashes. Defaults to strict.
       
      file
       
      (optional, string) Path to the output file. Missing directories will be created if possible with the mode 755. Defaults to /opt/intelmq/var/lib/bots/file-output/events.txt.
       
      format_filename
       
      (optional, boolean) Whether the file name should be formatted. Defaults to false.
       
      Uses Python formatted strings. See: https://docs.python.org/3/library/string.html#formatstrings
       
      Example:
       
         
      • The filename .../{event[source.abuse_contact]}.txt will be (for example) .../abuse@example.com.txt.
        •  
      • .../{event[time.source]:%Y-%m-%d} results in the date of the event used as filename.
        •  
       
      If the field used in the format string is not defined, None will be used as fallback.
       
      hierarchical_output
       
      (optional, boolean) Whether the resulting dictionary should be hierarchical (field names split by a dot). Defaults to false.
       
      single_key
       
      (optional, string) Output only a single specified key. In case of raw key the data is base64 decoded. Defaults to null (output the whole message).
      "},{"location":"user/bots/#files","title":"Files","text":"
      This bot outputs each message to a separate file.
       
      Module: intelmq.bots.outputs.files.output
       
      Parameters:
       
      dir
       
      (optional, string) Path to the output directory. Defaults to /opt/intelmq/var/lib/bots/files-output/incoming.
       
      tmp
       
      (optional, string) Temporary directory to use (must reside on the same filesystem as dir). Defaults to /opt/intelmq/var/lib/bots/files-output/tmp.
       
      suffix
       
      (optional, strings) Extension of created files. Defaults to .json.
       
      hierarchical_output
       
      (optional, boolean) Whether the resulting dictionary should be hierarchical (field names split by a dot). Defaults to false.
       
      single_key
       
      (optional, string) Output only a single specified key. In case of raw key the data is base64 decoded. Defaults to null (output the whole message).
      "},{"location":"user/bots/#mcafee-enterprise-security-manager","title":"McAfee Enterprise Security Manager","text":"
      This bot outputs messages to McAfee Enterprise Security Manager watchlist.
       
      Module: intelmq.bots.outputs.mcafee.output_esm_ip
       
      Parameters:
       
         
      • Feed parameters (see above)
        •  
       
      esm_ip
       
      (optional, string) Hostname of the ESM server. Defaults to 1.2.3.4.
       
      esm_user
       
      (optional, string) Username of user entitled to write to watchlist. Defaults to NGCP.
       
      esm_pw
       
      (required, string) Password of user entitled to write to watchlist.
       
      esm_watchlist
       
      (required, string) Name of the watchlist to write to.
       
      field
       
      (optional, string) Name of the IntelMQ field to be written to ESM. Defaults to source.ip.
      "},{"location":"user/bots/#misp-feed","title":"MISP Feed","text":"
      Create a directory layout in the MISP Feed format.
       
      The PyMISP library >= 2.4.119.1 is required, see REQUIREMENTS.txt.
       
      Module: intelmq.bots.outputs.misp.output_feed
       
      Parameters:
       
         
      • Feed parameters (see above)
        •  
       
      misp_org_name
       
      () Org name which creates the event, string
       
      misp_org_uuid
       
      () Org UUID which creates the event, string
       
      output_dir
       
      () Output directory path, e.g. [/opt/intelmq/var/lib/bots/mispfeed-output]. Will be created if it does not exist and possible.
       
      interval_event
       
      () The output bot creates one event per each interval, all data in this time frame is part of this event. Default \"1 hour\", string.
       
      Usage in MISP
       
      Configure the destination directory of this feed as feed in MISP, either as local location, or served via a web server. See the MISP documentation on Feeds for more information
      "},{"location":"user/bots/#misp-api","title":"MISP API","text":"
      Module: intelmq.bots.outputs.misp.output_api
       
      Connect to a MISP instance and add event as MISPObject if not there already.
       
      The PyMISP library >= 2.4.120 is required, see REQUIREMENTS.txt.
       
      Parameters:
       
         
      • Feed parameters (see above)
        •  
       
      add_feed_provider_as_tag
       
      () boolean (use [true] when in doubt)
       
      add_feed_name_as_tag
       
      () boolean (use [true] when in doubt)
       
      misp_additional_correlation_fields
       
      () list of fields for which the correlation flags will be enabled (in addition to those which are in significant_fields)
       
      misp_additional_tags
       
      () list of tags to set not be searched for when looking for duplicates
       
      misp_key
       
      () string, API key for accessing MISP
       
      misp_publish
       
      () boolean, if a new MISP event should be set to \"publish\".
       
      Expert setting as MISP may really make it \"public\"! (Use [false] when in doubt.)
       
      misp_tag_for_bot
       
      () string, used to mark MISP events
       
      misp_to_ids_fields
       
      () list of fields for which the [to_ids] flags will be set
       
      misp_url
       
      () string, URL of the MISP server
       
      significant_fields
       
      () list of intelmq field names
       
      The significant_fields values will be searched for in all MISP attribute values and if all values are found in the same MISP event, no new MISP event will be created. Instead if the existing MISP events have the same feed.provider and match closely, their timestamp will be updated.
       
      If a new MISP event is inserted the significant_fields and the misp_additional_correlation_fields will be the attributes where correlation is enabled.
       
      Make sure to build the IntelMQ Botnet in a way the rate of incoming events is what MISP can handle, as IntelMQ can process many more events faster than MISP (which is by design as MISP is for manual handling). Also remove the fields of the IntelMQ events with an expert bot that you do not want to be inserted into MISP.
       
      (More details can be found in the docstring of output_api.py.
      "},{"location":"user/bots/#mongodb","title":"MongoDB","text":"
      MongoDB is the bot responsible to send events to a MongoDB database
       
      Saves events in a MongoDB either as hierarchical structure or flat with full key names. time.observation and time.source are saved as datetime objects, not as ISO formatted string.
       
      Module: intelmq.bots.outputs.mongodb.output
       
      Requirements
       
      pip3 install pymongo>=2.7.1\n
       
      The bot has been tested with pymongo versions 2.7.1, 3.4 and 3.10.1 (server versions 2.6.10 and 3.6.8).
       
      Parameters:
       
      host
       
      (optional, string) Hostname of the MongoDB server. Defaults to localhost.
       
      port
       
      (optional, integer) Port of the MongoDB server. Defaults to 27017.
       
      database
       
      (required, string) Name of the MongoDB database to use.
       
      db_user
       
      (optional, string) User that should be used if authentication is required.
       
      db_pass
       
      (optional, string) Password.
       
      collection
       
      (required, string) Name of the MongoDB collection to use.
       
      hierarchical_output
       
      (optional, boolean) MongoDB does not allow saving keys with dots, we split the dictionary in sub-dictionaries. Defaults to true.
       
      replacement_char
       
      (optional, string) Replacement character for replacing the dots in key names if hierarchical output is not used. Defaults to _.
      "},{"location":"user/bots/#redis","title":"Redis","text":"
      This bot outputs events to a remote Redis server/queue.
       
      Examples of usage
       
         
      • Can be used to send events to be processed in another system. E.g.: send events to Logstash.
        •  
      • In a multi tenant installation can be used to send events to external/remote IntelMQ instance. Any expert bot queue   can receive the events.
        •  
      • In a complex configuration can be used to create logical sets in IntelMQ-Manager.
        •  
       
      Module: intelmq.bots.outputs.redis.output
       
      Parameters:
       
      redis_server_ip
       
      (optional, string) Hostname of the Redis server. Defaults to 127.0.0.1.
       
      redis_server_port
       
      (optional, integer) Port of the Redis server. Defaults to 6379.
       
      redis_db
       
      (optional, integer) Redis database number. Defaults to 2.
       
      redis_password
       
      (optional, string) Redis server password. Defaults to null.
       
      redis_queue
       
      (required, string) Redis queue name (such as remote-server-queue).
       
      redis_timeout
       
      (optional, integer) Connection timeout, in milliseconds. Defaults to 5000.
       
      hierarchical_output
       
      (optional, boolean) Whether the resulting dictionary should be hierarchical (field names split by a dot). Defaults to false.
       
      with_type
       
      (optional, boolean) Whether to include __type field. Defaults to true.
      "},{"location":"user/bots/#request-tracker_1","title":"Request Tracker","text":"
      Output Bot that creates Request Tracker tickets from events.
       
      Module: intelmq.bots.outputs.rt.output
       
      Description
       
      The bot creates tickets in Request Tracker and uses event fields for the ticket body text. The bot follows the workflow of the RTIR:
       
         
      • create ticket in Incidents queue (or any other queue)
        •  
      • all event fields are included in the ticket body,
        •  
      • event attributes are assigned to tickets' CFs according to the attribute mapping,
        •  
      • ticket taxonomy can be assigned according to the CF mapping. If you use taxonomy different   from ENISA RSIT, consider using some   extra attribute field and do value mapping with modify or sieve bot,
        •  
      • create linked ticket in Investigations queue, if these conditions are met
        •  
      • if first ticket destination was Incidents queue,
        •  
      • if there is source.abuse_contact is specified,
        •  
      • if description text is specified in the field appointed by configuration,
        •  
      • RT/RTIR supposed to do relevant notifications by script working on condition \"On Create\",
        •  
      • configuration option investigation_fields specifies which event fields has to be included in the investigation,
        •  
      • Resolve Incident ticket, according to configuration (Investigation ticket status should depend on RT script   configuration),
        •  
       
      Take extra caution not to flood your ticketing system with enormous amount of tickets. Add extra filtering for that to pass only critical events to the RT, and/or deduplicating events.
       
      Parameters:
       
      rt_uri
       
      ()
       
      rt_user
       
      ()
       
      rt_password
       
      ()
       
      verify_cert
       
      () RT API endpoint connection details, string.
       
      queue
       
      () ticket destination queue. If set to 'Incidents', 'Investigations' ticket will be created if create_investigation is set to true, string.
       
      CF_mapping
       
      (optional, object) Mapping event fields to ticket CFs. Defaults to:
       
      classification.taxonomy: Classification\nclassification.type: Incident Type\nevent_description.text: Description\nextra.incident.importance: Importance\nextra.incident.severity: Incident Severity\nextra.organization.name: Customer\nsource.ip: IP\n
       
      final_status
       
      (optional, string) The final status for the created ticket. Defaults to resolved. The linked Investigation ticket will be resolved automatically by RTIR scripts.
       
      create_investigation
       
      (optional, boolean) Whether an Investigation ticket should be created (in case of RTIR workflow). Defaults to false.
       
      investigation_fields
       
      (optional, string) Comma-separated string of attributes to include in an Investigation ticket. Defaults to time.source,source.ip,source.port,source.fqdn,source.url,classification.taxonomy,classification.type,classification.identifier,event_description.url,event_description.text,malware.name,protocol.application,protocol.transport.
       
      description_attr
       
      (optional, string) Event field to be used as a text message being sent to the recipient. If it is not specified or not found in the event, the Investigation ticket is not going to be created. Defaults to event_decription.text.
      "},{"location":"user/bots/#rest-api","title":"REST API","text":"
      REST API is the bot responsible to send events to a REST API listener through POST.
       
      Module: intelmq.bots.outputs.restapi.output
       
      Parameters:
       
      host
       
      (required, host) Destination URL of the POST request.
       
      auth_type
       
      (required, string) Allowed values: http_basic_auth or http_header. Type of authentication to use.
       
      auth_token
       
      (required, string) Username or HTTP header key.
       
      auth_token_name
       
      (required, string) Password or HTTP header value.
       
      hierarchical_output
       
      (optional, boolean) Whether the resulting dictionary should be hierarchical (field names split by a dot). Defaults to false.
       
      use_json
       
      (optional, boolean) Whether to use JSON. Defaults to true.
      "},{"location":"user/bots/#rpz-file","title":"RPZ File","text":"
      This bot outputs events into DNS RPZ blocklist file used for \"DNS firewall\".
       
      The prime motivation for creating this feature was to protect users from badness on the Internet related to known-malicious global identifiers such as host names, domain names, IP addresses, or nameservers. More information: https://dnsrpz.info
       
      Example: 
      $TTL 3600 @ SOA rpz.yourdomain.eu. hostmaster.rpz.yourdomain.eu. 2105260601 60 60 432000 60 NS localhost. ; ;\nyourdomain.eu. CERT.XX Response Policy Zones (RPZ) ; Last updated: 2021-05-26 06:01:41 (UTC) ; ; Terms Of\nUse: https://rpz.yourdomain.eu ; For questions please contact rpz [at] yourdomain.eu ; *.maliciousdomain.com CNAME\nrpz.yourdomain.eu. *.secondmaliciousdomain.com CNAME rpz.yourdomain.eu.\n
       
      Module: intelmq.bots.outputs.rpz_file.output
       
      Parameters:
       
      cname
       
      (optional, string) example rpz.yourdomain.eu
       
      organization_name
       
      (optional, string) Your organisation name
       
      rpz_domain
       
      (optional, string) Information website about RPZ
       
      hostmaster_rpz_domain
       
      () Technical website
       
      rpz_email
       
      () Contact email
       
      ttl
       
      () Time to live
       
      ncachttl
       
      () DNS negative cache
       
      serial
       
      () Time stamp or another numbering
       
      refresh
       
      () Refresh time
       
      retry
       
      () Retry time
       
      expire
       
      () Expiration time
       
      test_domain
       
      () For test domain, it's added in first rpz file (after header)
      "},{"location":"user/bots/#smtp-batch","title":"SMTP Batch","text":"
      Aggregate events by e-mail addresses in the source.abuse_contact field and batch send them at once as a zipped CSV file attachment in a GPG signed message.
       
      When the bot is run normally by IntelMQ, it just aggregates the events for later use into a custom Redis database. If run through CLI (by a cron or manually), it shows e-mail messages that are ready to be sent and let you send them to the tester's e-mail OR to abuse contact e-mails. E-mails are sent in a zipped CSV file, delimited by a comma, while keeping strings in double quotes. Note: The field \"raw\" gets base64 decoded if possible. Bytes \\n and \\r are replaced with \"\\n\" and \"\\r\" strings in order to guarantee best CSV files readability both in Microsoft Office and LibreOffice. (A multiline string may be stored in \"raw\" which completely confused Microsoft Excel.)
       
      Launch it like this: 
      </usr/local/bin executable> <bot-id> --cli [--tester tester's email]\n
       Example: 
      intelmq.bots.outputs.smtp_batch.output smtp-batch-output --cli --tester your-email@example.com\n
       
      CLI flags: 
      -h, --help            show this help message and exit\n--cli                 initiate CLI interface\n--tester TESTING_TO   tester's e-mail\n--ignore-older-than-days IGNORE_OLDER_THAN_DAYS\n                      1..n skip all events with time.observation older than 1..n day; 0 disabled (allow all)\n--gpg-key GPG_KEY     fingerprint of gpg key to be used\n--limit-results LIMIT_RESULTS\n                      Just send first N mails.\n--send                Sends now, without dialog.\n
       
      You can schedule the batch sending easily with a cron script, I.E. put this into crontab -e of the intelmq user:
       
      # Send the e-mails every day at 6 AM\n0 6 * * *  /usr/local/bin/intelmq.bots.outputs.smtp_batch.output smtp-batch-output-cz cli --ignore-older-than-days 4 --send &> /tmp/intelmq-send.log\n
       
      Module: intelmq.bots.outputs.smtp_batch.output
       
      Parameters:
       
      alternative_mails
       
      (optional, string) Path to CSV in the form original@email.com,alternative@email.com. Needed when some of the recipients ask you to forward their e-mails to another address. Delimit multiple recipients by the semicolon. The field is internally parsed by Envelope so pretty anything is allowed:
       
      original@email.com,alternative@email.com\noriginal2@email.com,person1@email.com;person2@email.com\noriginal3@email.com, Mary <person1@example.com>; John <person2@example.com>\n
       
      attachment_name
       
      (optional, string)  Attachment file name for the outgoing messages. May contain date formatting like this %Y-%m-%d. Example: \"events_%Y-%m-%d\" will appear as \"events_2022-12-01.zip\". Defaults to \"intelmq_%Y-%m-%d\".
       
      bcc
       
      (optional, array of strings) An array of e-mails to be put in the Bcc field for every mail.
       
      email_from
       
      (required, string) Sender's e-mail of the outgoing messages.
       
      gpg_key
       
      (optional, string) The Key or the fingerprint of a GPG key stored in ~/.gnupg keyring folder.
       
      gpg_pass
       
      (optional, string) Password for the GPG key if needed.
       
      mail_template
       
      (required, string) Path to the file containing the body of the mail for the outgoing messages.
       
      ignore_older_than_days
       
      (optional, integer) Skips events with time.observation older than now-N. (If your queue gets stuck for a reason, you do not want to send old and probably already solved events.) Defaults to 0 (allow all).
       
      limit_results
       
      (optional, integer) Intended as a debugging option, allows loading just first N e-mails from the queue.
       
      redis_cache_db
       
      (required, integer)  Redis database used for event aggregation. As the databases < 10 are reserved for the IntelMQ core, recommended is a bigger number.
       
      redis_cache_host
       
      (required, string) Hostname of the Redis database.
       
      redis_cache_port
       
      (required, string)  Port of the Redis database.
       
      redis_cache_ttl
       
      (required, integer) TTL in seconds used for caching. Recommended 1728000 for 20 days.
       
      smtp_server
       
      (required, string/array/object) SMTP server information and credentials. See SMTP parameter of the envelope module.
       
      Examples: 
      smtp_server: \"mailer\"\nsmtp_server: {\"host\": \"mailer\", \"port\": 587, \"user\": \"john\", \"password\": \"123\"}\nsmtp_server: [\"mailer\", 587, \"john\", \"password\"]\n
       
      subject
       
      (required, string) Subject for the outgoing messages. May contain date formatting like this %Y-%m-%d. Example: \"IntelMQ weekly warning (%d.%m.%Y)\".
       
      testing_to
       
      (optional, string) Tester's e-mail.
      "},{"location":"user/bots/#smtp","title":"SMTP","text":"
      Sends a MIME Multipart message containing the text and the event as CSV for every single event.
       
      Module: intelmq.bots.outputs.smtp.output
       
      Parameters:
       
      fieldnames
       
      (optional, string/array of strings) Array of field names (or comma-separated list) to be included in the email. If empty, no attachment is sent - this can be useful if the actual data is already in the body (parameter text) or the subject.
       
      mail_from
       
      (optional, string) Sender's e-email address. Defaults to cert@localhost.
       
      mail_to
       
      (required, string) Comma-separated string of recipient email addresses. Supports formatting.
       
      smtp_host
       
      (optional, string) Hostname of the SMTP server. Defaults to localhost.
       
      smtp_password
       
      (optional, string) Password for authentication to your SMTP server. Defaults to null.
       
      smtp_port
       
      (optional, integer) Port of the SMTP server. Defaults to 25.
       
      smtp_username
       
      (optional, string) Username for authentication to your SMTP server. Defaults to null.
       
      fail_on_errors
       
      (optional, boolean) Whether any error should cause the bot to fail (raise an exception) or otherwise rollback. If false, the bot eventually waits and re-try (e.g. re-connect) etc. to solve the issue. If true, the bot raises an exception and - depending on the IntelMQ error handling configuration - stops. Defaults to false.
       
      ssl
       
      (optional, boolean) Defaults to false.
       
      starttls
       
      (optional, boolean) Defaults to true.
       
      subject
       
      (optional, string) Subject of the e-mail message. Supports formatting. Defaults to Incident in your AS {ev[source.asn]}.
       
      text
       
      (optional, string) Body of the e-mail message. Supports formatting. Defaults to 
      Dear network owner,\n\nWe have been informed that the following device might have security problems.\n\nYour localhost CERT\n
       
      For several strings you can use values from the string using the standard Python string format syntax. Access the event's values with {ev[source.ip]} and similar. Any not existing fields will result in None. For example, to set the recipient(s) to the value given in the event's source.abuse_contact field, use this as mail_to parameter: {ev[source.abuse_contact]}
       
      Authentication is optional. If both username and password are given, these mechanism are tried: CRAM-MD5, PLAIN, and LOGIN.
       
      Client certificates are not supported. If http_verify_cert is true, TLS certificates are checked.
      "},{"location":"user/bots/#sql","title":"SQL","text":"
      SQL is the bot responsible to send events to a PostgreSQL, SQLite, or MSSQL Database.
       
      Note
       
      When activating autocommit, transactions are not used. See: http://initd.org/psycopg/docs/connection.html#connection.autocommit
       
      Module: intelmq.bots.outputs.sql.output
       
      Parameters:
       
      The parameters marked with 'PostgreSQL' will be sent to libpq via psycopg2. Check the libpq parameter documentation for the versions you are using.
       
      autocommit
       
      (optional, boolean) Psycopg's autocommit mode. Defaults to true.
       
      engine
       
      (required, string) Allowed values are postgresql, sqlite, or mssql.
       
      database
       
      (optional, string) Database name or SQLite database file. Defaults to intelmq-events.
       
      host
       
      (optional, string) Hostname of the database server. Defaults to localhost.
       
      jsondict_as_string
       
      (optional, boolean) Whether to save JSON fields as JSON string. Defaults to true.
       
      message_jsondict_as_string
       
      (optional, boolean) Whether to save JSON fields as JSON string. Defaults to true.
       
      port
       
      (optional, integer) Port of the database server. Defaults to 5432.
       
      user
       
      (optional, string) Username for connecting to the database system. Defaults to intelmq.
       
      password
       
      (optional, string) Password for connecting to the database system. Defaults to null.
       
      sslmode
       
      (optional, string) Database sslmode, Allowed values: disable, allow, prefer, require, verify-ca or verify-full. See: https://www.postgresql.org/docs/current/static/images/libpq-connect.html#libpq-connect-sslmode. Defaults to require.
       
      table
       
      (optional, string) Name of the database table to use. Defaults to events.
       
      fields
       
      (optional, array) Array of event fields to output to the database. Defaults to null (use all fields).
       
      reconnect_delay
       
      (optional, integer) Number of seconds to wait before reconnecting in case of an error. Defaults to 0.
       
      fail_on_errors
       
      (optional, boolean) Whether an error should cause the bot to fail (raise an exception) or otherwise rollback. If false, the bot eventually waits and re-try (e.g. re-connect) etc. to solve the issue. If true, the bot raises an exception and - depending on the IntelMQ error handling configuration - stops. Defaults to false.
      "},{"location":"user/bots/#stomp_1","title":"STOMP","text":"
      This bot pushes data to any STOMP stream. STOMP stands for Streaming Text Oriented Messaging Protocol. See: https://en.wikipedia.org/wiki/Streaming_Text_Oriented_Messaging_Protocol
       
      Module: intelmq.bots.outputs.stomp.output
       
      Requirements
       
      Install the stomp.py library from PyPI:
       
      pip3 install -r intelmq/bots/outputs/stomp/REQUIREMENTS.txt\n
       
      Alternatively, you may want to install it using your OS's native packaging tools, e.g.:
       
      apt install python3-stomp\n
       
      Apart from that, depending on what STOMP server you connect to, you may need to obtain, from the organization or company owning the server, one or more of the following security/authentication-related resources:
       
         
      • CA certificate file;
        •  
      • either: client certificate and client certificate's key files,   or: username (STOMP login) and password (STOMP passcode).
        •  
       
      Also, you will need to know an appropriate STOMP destination (aka exchange point), e.g. /exchange/_push.
       
      Parameters:
       
      server
       
      (optional, string) STOMP server's hostname or IP, e.g. \"n6stream.cert.pl\" or \"127.0.0.1\" (which is default)
       
      port
       
      (optional, integer) STOMP server's port number (default: 61614)
       
      exchange
       
      (optional, string) STOMP destination to push at, e.g. \"/exchange/_push\" (which is default)
       
      heartbeat
       
      (optional, integer) Defaults to 60000.
       
      ssl_ca_certificate
       
      (optional, string) path to CA file, or empty string to load system's default CA certificates
       
      auth_by_ssl_client_certificate
       
      (optional, boolean) default: true (note: false is needed for new n6 auth)
       
      ssl_client_certificate
       
      (optional, string) Path to client certificate to use for TLS connections.
       
      ssl_client_certificate_key
       
      (optional, string) Path to client private key to use for TLS connections.
       
      username
       
      (optional, string) STOMP login (e.g., n6 user login), used only if auth_by_ssl_client_certificate is false
       
      password
       
      (optional, string) STOMP passcode (e.g., n6 user API key), used only if auth_by_ssl_client_certificate is false
       
      message_hierarchical_output
       
      (optional, boolean) Defaults to false.
       
      message_jsondict_as_string
       
      (optional, boolean) Defaults to false.
       
      message_with_type
       
      (optional, boolean) Defaults to false.
       
      single_key
       
      (optional, string) Output only a single specified key. In case of raw key the data is base64 decoded. Defaults to null (output the whole message).
      "},{"location":"user/bots/#tcp_1","title":"TCP","text":"
      TCP is the bot responsible to send events to a TCP port (Splunk, another IntelMQ, etc..).
       
      Multihreading is disabled for this bot.
       
      Sending to an IntelMQ TCP collector
       
      If you intend to link two IntelMQ instance via TCP, set the parameter counterpart_is_intelmq to true. The bot then awaits an \"Ok\" message to be received after each message is sent. The TCP collector just sends \"Ok\" after every message it gets.
       
      Module: intelmq.bots.outputs.tcp.output
       
      Parameters:
       
      counterpart_is_intelmq
       
      (optional, boolean) Whether the receiver is an IntelMQ TCP collector bot. Defaults to true.
       
      ip
       
      (required, string) Hostname of the destination server.
       
      hierarchical_output
       
      (optional, boolean) True for a nested JSON, false for a flat JSON (when sending to a TCP collector).
       
      port
       
      (required, integer) Port of destination server.
       
      separator
       
      (optional, string) Separator of messages, e.g. \"n\", optional. When sending to a TCP collector, parameter shouldn't be present. In that case, the output waits every message is acknowledged by \"Ok\" message the TCP collector bot implements.
      "},{"location":"user/bots/#templated-smtp","title":"Templated SMTP","text":"
      Sends a MIME Multipart message built from an event and static text using Jinja2 templates.
       
      See the Jinja2 documentation at https://jinja.palletsprojects.com/.
       
      Authentication is attempted only if both username and password are specified.
       
      Templates are in Jinja2 format with the event provided in the variable event. E.g.:
       
      mail_to: \"{{ event['source.abuse_contact'] }}\"\n
       
      As an extension to the Jinja2 environment, the function from_json is available for parsing JSON strings into Python structures. This is useful if you want to handle complicated structures in the output field of an event. In that case, you would start your template with a line like:
       
      {%- set output = from_json(event['output']) %}\n
       
      and can then use output as a regular Python object in the rest of the template.
       
      Attachments are templated strings, especially useful for sending structured data. E.g. to send a JSON document including malware.name and all other fields starting with source.:
       
      attachments:\n  - content-type: application/json\n    text: |\n      {\n        \"malware\": \"{{ event['malware.name'] }}\",\n        {%- set comma = joiner(\", \") %}\n        {%- for key in event %}\n           {%- if key.startswith('source.') %}\n        {{ comma() }}\"{{ key }}\": \"{{ event[key] }}\"\n           {%- endif %}\n        {%- endfor %}\n      }\n    name: report.json\n
       
      You are responsible for making sure that the text produced by the template is valid according to the content-type.
       
      If you are migrating from the SMTP output bot that produced CSV format attachments, use the following configuration to produce a matching format:
       
      attachments:\n  - content-type: text/csv\n    text: |\n      {%- set fields = [\"classification.taxonomy\", \"classification.type\", \"classification.identifier\", \"source.ip\",\"source.asn\", \"source.port\"] %}\n      {%- set sep = joiner(\";\") %}\n      {%- for field in fields %}{{ sep() }}{{ field }}{%- endfor %}\n      {% set sep = joiner(\";\") %}\n      {%- for field in fields %}{{ sep() }}{{ event[field] }}{%- endfor %}\n    name: event.csv\n
       
      Module: intelmq.bots.outputs.templated_smtp.output
       
      Requirements
       
      Install the required jinja2 library:
       
      pip3 install -r intelmq/bots/collectors/templated_smtp/REQUIREMENTS.txt\n
       
      Parameters:
       
      attachments
       
      (required, array of objects) Each object must have content-type, text (attachment text) and name (filename of the attachment) fields.
       
      - content-type: simple string/jinja template\n  text: simple string/jinja template\n  name: simple string/jinja template\n
       
      body
       
      (optional, string) Simple string or Jinja template. The default body template prints every field in the event except raw, in undefined order, one field per line, as \"field: value\".
       
      mail_from
       
      (optional, string) Simple string or Jinja template. Sender's address.
       
      mail_to
       
      (required, string) Simple string or Jinja template. Comma-separated array of recipient addresses.
       
      smtp_host
       
      (optional, string) Hostname of the SMTP server. Defaults to localhost.
       
      smtp_password
       
      (optional, string) Password (if any) for authenticated SMTP. Defaults to null.
       
      smtp_port
       
      (optional, integer) TCP port to connect to. Defaults to 25.
       
      smtp_username
       
      (optional, string)  Username (if any) for authenticated SMTP. Defaults to null.
       
      tls
       
      (optional, boolean) Whether to use use SMTPS. When true, also set smtp_port to the SMTPS port. Defaults to false.
       
      starttls
       
      (optional, boolean) Whether to use opportunistic STARTTLS over SMTP. Defaults to true.
       
      subject
       
      (optional, string) Simple string or Jinja template. E-mail subject line. Defaults to \"IntelMQ event\".
       
      verify_cert
       
      (optional, boolean) Whether to verify the server certificate in STARTTLS or SMTPS. Defaults to true.
      "},{"location":"user/bots/#touch","title":"Touch","text":"
      Touches a file for every event received. Does not output the event!
       
      Module: intelmq.bots.outputs.touch.output
       
      Parameters:
       
      path
       
      (optional, string) Path to the file to touch.
      "},{"location":"user/bots/#udp","title":"UDP","text":"
      Output Bot that sends events to a remote UDP server.
       
      Multihreading is disabled for this bot.
       
      Module: intelmq.bots.outputs.udp.output
       
      Parameters:
       
      format
       
      (optional, string) Allowed values: json or delimited. The JSON format outputs the event 'as-is'. Delimited will deconstruct the event and print each field:value separated by the field delimit. See examples below.
       
      field_delimiter
       
      (optional, string) If the format is delimited then this parameter is used as a delimiter between fields. Defaults to |.
       
      header
       
      (required, string) Header text to be sent in the UDP datagram.
       
      keep_raw_field
       
      (optional, boolean) Whether to keep raw field. Defaults to false.
       
      udp_host
       
      (optional, string) Hostname of the destination server.
       
      udp_port
       
      (required, integer) Port of the destination server.
       
      Examples of usage
       
      Consider the following event:
       
      {\n  \"raw\": \"MjAxNi8wNC8yNV8xMTozOSxzY2hpenppbm8ub21hcmF0aG9uLmNvbS9na0NDSnVUSE0vRFBlQ1pFay9XdFZOSERLbC1tWFllRk5Iai8sODUuMjUuMTYwLjExNCxzdGF0aWMtaXAtODUtMjUtMTYwLTExNC5pbmFkZHIuaXAtcG9vbC5jb20uLEFuZ2xlciBFSywtLDg5NzI=\",\n  \"source.asn\": 8972,\n  \"source.ip\": \"85.25.160.114\",\n  \"source.url\": \"http://schizzino.omarathon.com/gkCCJuTHM/DPeCZEk/WtVNHDKl-mXYeFNHj/\",\n  \"source.reverse_dns\": \"static-ip-85-25-160-114.inaddr.ip-pool.com\",\n  \"classification.type\": \"malware-distribution\",\n  \"event_description.text\": \"Angler EK\",\n  \"feed.url\": \"http://www.malwaredomainlist.com/updatescsv.php\",\n  \"feed.name\": \"Malware Domain List\",\n  \"feed.accuracy\": 100,\n  \"time.observation\": \"2016-04-29T10:59:34+00:00\",\n  \"time.source\": \"2016-04-25T11:39:00+00:00\"\n}\n
       
      With the following parameters:
       
      format: json\nheader: header example\nkeep_raw_field: true\nip: 127.0.0.1\nport: 514\n
       
      Resulting line in syslog:
       
      Apr 29 11:01:29 header example {\"raw\": \"MjAxNi8wNC8yNV8xMTozOSxzY2hpenppbm8ub21hcmF0aG9uLmNvbS9na0NDSnVUSE0vRFBlQ1pFay9XdFZOSERLbC1tWFllRk5Iai8sODUuMjUuMTYwLjExNCxzdGF0aWMtaXAtODUtMjUtMTYwLTExNC5pbmFkZHIuaXAtcG9vbC5jb20uLEFuZ2xlciBFSywtLDg5NzI=\", \"source\": {\"asn\": 8972, \"ip\": \"85.25.160.114\", \"url\": \"http://schizzino.omarathon.com/gkCCJuTHM/DPeCZEk/WtVNHDKl-mXYeFNHj/\", \"reverse_dns\": \"static-ip-85-25-160-114.inaddr.ip-pool.com\"}, \"classification\": {\"type\": \"malware-distribution\"}, \"event_description\": {\"text\": \"Angler EK\"}, \"feed\": {\"url\": \"http://www.malwaredomainlist.com/updatescsv.php\", \"name\": \"Malware Domain List\", \"accuracy\": 100.0}, \"time\": {\"observation\": \"2016-04-29T10:59:34+00:00\", \"source\": \"2016-04-25T11:39:00+00:00\"}}\n
       
      With the following Parameters:
       
      field_delimiter: |\nformat: delimited\nheader: IntelMQ-event\nkeep_raw_field: false\nip: 127.0.0.1\nport: 514\n
       
      Resulting line in syslog:
       
      Apr 29 11:17:47 localhost IntelMQ-event|source.ip: 85.25.160.114|time.source:2016-04-25T11:39:00+00:00|feed.url:http://www.malwaredomainlist.com/updatescsv.php|time.observation:2016-04-29T11:17:44+00:00|source.reverse_dns:static-ip-85-25-160-114.inaddr.ip-pool.com|feed.name:Malware Domain List|event_description.text:Angler EK|source.url:http://schizzino.omarathon.com/gkCCJuTHM/DPeCZEk/WtVNHDKl-mXYeFNHj/|source.asn:8972|classification.type:malware-distribution|feed.accuracy:100.0\n
      "},{"location":"user/event/","title":"Event","text":""},{"location":"user/event/#event","title":"Event","text":"
      An event represents individual piece of data processed by IntelMQ. It uses JSON format.
       
      Example Event:
       
      {\n    \"source.geolocation.cc\": \"JO\",\n    \"malware.name\": \"qakbot\",\n    \"source.ip\": \"82.212.115.188\",\n    \"source.asn\": 47887,\n    \"classification.type\": \"c2-server\",\n    \"extra.status\": \"offline\",\n    \"source.port\": 443,\n    \"classification.taxonomy\": \"malicious-code\",\n    \"source.geolocation.latitude\": 31.9522,\n    \"feed.accuracy\": 100,\n    \"extra.last_online\": \"2023-02-16\",\n    \"time.observation\": \"2023-02-16T09:55:12+00:00\",\n    \"source.geolocation.city\": \"amman\",\n    \"source.network\": \"82.212.115.0/24\",\n    \"time.source\": \"2023-02-15T14:19:09+00:00\",\n    \"source.as_name\": \"NEU-AS\",\n    \"source.geolocation.longitude\": 35.939,\n    \"feed.name\": \"abusech-feodo-c2-tracker\"\n  }\n
      "},{"location":"user/event/#minimum-requirements","title":"Minimum Requirements","text":"
      Below, we have enumerated the minimum recommended requirements for an actionable abuse event. These keys should be present for the abuse report to make sense for the end recipient. Please note that if you choose to anonymize your sources, you can substitute feed.name with feed.code. At least one of the fields ip, fqdn, url or account should be present. All the rest of the keys are optional. This list of required fields is not enforced by IntelMQ.
       Field Terminology feed.name Should classification.type Should classification.taxonomy Should time.source Should time.observation Should source.ip Should* source.fqdn Should* source.url Should* source.account Should* 
      * at least one of them
      "},{"location":"user/event/#classification","title":"Classification","text":"
      IntelMQ classifies events using three labels: classification.taxonomy, classification.type and classification.identifier. This tuple of three values can be used for deduplication of events and describes what happened.
       
      The taxonomy can be automatically added by the taxonomy expert bot based on the given type. The following classification scheme loosely follows the Reference Security Incident Taxonomy (RSIT):
       Classification Taxonomy Classification Type Description abusive-content harmful-speech Discreditation or discrimination of somebody, cyber stalking, racism or threats against one or more individuals. abusive-content spam Or 'Unsolicited Bulk Email', this means that the recipient has not granted verifiable permission for the message to be sent and that the message is sent as part of a larger collection of messages, all having a functionally comparable content. abusive-content violence Child pornography, glorification of violence, etc. availability ddos Distributed Denial of Service attack, e.g. SYN-Flood or UDP-based reflection/amplification attacks. availability dos Denial of Service attack, e.g. sending specially crafted requests to a web application which causes the application to crash or slow down. availability misconfiguration Software misconfiguration resulting in service availability issues, e.g. DNS server with outdated DNSSEC Root Zone KSK. availability outage Outage caused e.g. by air condition failure or natural disaster. availability sabotage Physical sabotage, e.g cutting wires or malicious arson. fraud copyright Offering or Installing copies of unlicensed commercial software or other copyright protected materials (Warez). fraud masquerade Type of attack in which one entity illegitimately impersonates the identity of another in order to benefit from it. fraud phishing Masquerading as another entity in order to persuade the user to reveal private credentials. fraud unauthorized-use-of-resources Using resources for unauthorized purposes including profit-making ventures, e.g. the use of e-mail to participate in illegal profit chain letters or pyramid schemes. information-content-security data-leak Leaked confidential information like credentials or personal data. information-content-security data-loss Loss of data, e.g. caused by harddisk failure or physical theft. information-content-security unauthorised-information-access Unauthorized access to information, e.g. by abusing stolen login credentials for a system or application, intercepting traffic or gaining access to physical documents. information-content-security unauthorised-information-modification Unauthorised modification of information, e.g. by an attacker abusing stolen login credentials for a system or application or a ransomware encrypting data. information-gathering scanner Attacks that send requests to a system to discover weaknesses. This also includes testing processes to gather information on hosts, services and accounts. Examples: fingerd, DNS querying, ICMP, SMTP (EXPN, RCPT, ...), port scanning. information-gathering sniffing Observing and recording of network traffic (wiretapping). information-gathering social-engineering Gathering information from a human being in a non-technical way (e.g. lies, tricks, bribes, or threats). This IOC refers to a resource, which has been observed to perform brute-force attacks over a given application protocol. intrusion-attempts brute-force Multiple login attempts (Guessing/cracking of passwords, brute force). intrusion-attempts exploit An attack using an unknown exploit. intrusion-attempts ids-alert IOCs based on a sensor network. This is a generic IOC denomination, should it be difficult to reliably denote the exact type of activity involved for example due to an anecdotal nature of the rule that triggered the alert. intrusions application-compromise Compromise of an application by exploiting (un)known software vulnerabilities, e.g. SQL injection. intrusions burglary Physical intrusion, e.g. into corporate building or data center. intrusions privileged-account-compromise Compromise of a system where the attacker gained administrative privileges. intrusions system-compromise Compromise of a system, e.g. unauthorised logins or commands. This includes compromising attempts on honeypot systems. intrusions unprivileged-account-compromise Compromise of a system using an unprivileged (user/service) account. malicious-code c2-server This is a command and control server in charge of a given number of botnet drones. malicious-code infected-system This is a compromised machine, which has been observed to make a connection to a command and control server. malicious-code malware-configuration This is a resource which updates botnet drones with a new configuration. malicious-code malware-distribution URI used for malware distribution, e.g. a download URL included in fake invoice malware spam. other blacklist Some sources provide blacklists, which clearly refer to abusive behavior, such as spamming, but fail to denote the exact reason why a given identity has been blacklisted. The reason may be that the justification is anecdotal or missing entirely. This type should only be used if the typing fits the definition of a blacklist, but an event specific denomination is not possible for one reason or another. Not in RSIT. other dga-domain DGA Domains are seen various families of malware that are used to periodically generate a large number of domain names that can be used as rendezvous points with their command and control servers. Not in RSIT. other other All incidents which don't fit in one of the given categories should be put into this class. other malware An IoC referring to a malware (sample) itself. Not in RSIT. other proxy This refers to the use of proxies from inside your network. Not in RSIT. test test Meant for testing. Not in RSIT. other tor This IOC refers to incidents related to TOR network infrastructure. Not in RSIT. other undetermined The categorisation of the incident is unknown/undetermined. vulnerable ddos-amplifier Publicly accessible services that can be abused for conducting DDoS reflection/amplification attacks, e.g. DNS open-resolvers or NTP servers with monlist enabled. vulnerable information-disclosure Publicly accessible services potentially disclosing sensitive information, e.g. SNMP or Redis. vulnerable potentially-unwanted-accessible Potentially unwanted publicly accessible services, e.g. Telnet, RDP or VNC. vulnerable vulnerable-system A system which is vulnerable to certain attacks. Example: misconfigured client proxy settings (example: WPAD), outdated operating system  version, etc. vulnerable weak-crypto Publicly accessible services offering weak crypto, e.g. web servers susceptible to POODLE/FREAK attacks."},{"location":"user/event/#meaning-of-source-and-destination-identities","title":"Meaning of source and destination identities","text":"
      Meaning of source and destination identities for each classification.type can be different. Usually the main information is in the source.* fields.
       
      The classification.identifier is often a normalized malware name, grouping many variants or the affected network protocol.
       
      Examples of the meaning of the source and destination fields for various classification.type and possible identifiers are shown here.
       Classification Type Source Destination Possible Identifiers blacklist blacklisted device brute-force attacker target c2-server (sinkholed) c&c server zeus, palevo, feodo ddos attacker target dga-domain infected device dropzone server hosting stolen data exploit hosting server ids-alert triggering device infected-system infected device contacted c&c server malware infected device zeus, palevo, feodo malware-configuration infected device malware-distribution server hosting malware phishing phishing website proxy server allowing policy/security bypass scanner scanning device scanned device http, modbus, wordpress spam infected device targeted server system-compromise server vulnerable-system vulnerable device heartbleed, openresolver, snmp, wpad 
      Examples:
       
         
      •  
        If an event describes IP address that connects to a zeus command and control server, it's about the infected device. Therefore the classification.taxonomy is malicious-code, classification.type is infected-system and the classification.identifier is zeus.
         
        •  
      •  
        If an event describes IP address where a command and control server is running, the event's classification.type is c2server. The malware.name can have the full name, eg. zeus_p2p.
         
        •  
      "},{"location":"user/event/#additional-information","title":"Additional Information","text":"
      Information that do not fit into any of the event fields should be placed in the extra namespace.Therefore the keys must be prefixed extra. string. There are no other rules on key names and values for additional information.
      "},{"location":"user/event/#fields-reference","title":"Fields Reference","text":"
      Here you can find detailed information about all the possible fields used in an event.
      "},{"location":"user/event/#classificationidentifier","title":"classification.identifier","text":"
      Type: String
       
      The lowercase identifier defines the actual software or service (e.g. heartbleed or ntp_version) or standardized malware name (e.g. zeus). Note that you MAY overwrite this field during processing for your individual setup. This field is not standardized across IntelMQ setups/users.
      "},{"location":"user/event/#classificationtaxonomy","title":"classification.taxonomy","text":"
      Type: ClassificationTaxonomy
       
      We recognize the need for the CSIRT teams to apply a static (incident) taxonomy to abuse data. With this goal in mind the type IOC will serve as a basis for this activity. Each value of the dynamic type mapping translates to a an element in the static taxonomy. The European CSIRT teams for example have decided to apply the eCSIRT.net incident classification. The value of the taxonomy key is thus a derivative of the dynamic type above. For more information about check ENISA taxonomies <http://www.enisa.europa.eu/activities/cert/support/incident-management/browsable/incident-handling-process/incident-taxonomy/existing-taxonomies>_.
      "},{"location":"user/event/#classificationtype","title":"classification.type","text":"
      Type: ClassificationType
       
      The abuse type IOC is one of the most crucial pieces of information for any given abuse event. The main idea of dynamic typing is to keep our ontology flexible, since we need to evolve with the evolving threatscape of abuse data. In contrast with the static taxonomy below, the dynamic typing is used to perform business decisions in the abuse handling pipeline. Furthermore, the value data set should be kept as minimal as possible to avoid type explosion, which in turn dilutes the business value of the dynamic typing. In general, we normally have two types of abuse type IOC: ones referring to a compromised resource or ones referring to pieces of the criminal infrastructure, such as a command and control servers for example.
      "},{"location":"user/event/#comment","title":"comment","text":"
      Type: String
       
      Free text commentary about the abuse event inserted by an analyst.
      "},{"location":"user/event/#destinationabuse_contact","title":"destination.abuse_contact","text":"
      Type: LowercaseString
       
      Abuse contact for destination address. A comma separated list.
      "},{"location":"user/event/#destinationaccount","title":"destination.account","text":"
      Type: String
       
      An account name or email address, which has been identified to relate to the destination of an abuse event.
      "},{"location":"user/event/#destinationallocated","title":"destination.allocated","text":"
      Type: DateTime
       
      Allocation date corresponding to BGP prefix.
      "},{"location":"user/event/#destinationas_name","title":"destination.as_name","text":"
      Type: String
       
      The autonomous system name to which the connection headed.
      "},{"location":"user/event/#destinationasn","title":"destination.asn","text":"
      Type: ASN
       
      The autonomous system number to which the connection headed.
      "},{"location":"user/event/#destinationdomain_suffix","title":"destination.domain_suffix","text":"
      Type: FQDN
       
      The suffix of the domain from the public suffix list.
      "},{"location":"user/event/#destinationfqdn","title":"destination.fqdn","text":"
      Type: FQDN
       
      A DNS name related to the host from which the connection originated. DNS allows even binary data in DNS, so we have to allow everything. A final point is stripped, string is converted to lower case characters.
      "},{"location":"user/event/#destinationgeolocationcc","title":"destination.geolocation.cc","text":"
      Type: UppercaseString
       
      Country-Code according to ISO3166-1 alpha-2 for the destination IP.
      "},{"location":"user/event/#destinationgeolocationcity","title":"destination.geolocation.city","text":"
      Type: String
       
      Some geolocation services refer to city-level geolocation.
      "},{"location":"user/event/#destinationgeolocationcountry","title":"destination.geolocation.country","text":"
      Type: String
       
      The country name derived from the ISO3166 country code (assigned to cc field).
      "},{"location":"user/event/#destinationgeolocationlatitude","title":"destination.geolocation.latitude","text":"
      Type: Float
       
      Latitude coordinates derived from a geolocation service, such as MaxMind geoip db.
      "},{"location":"user/event/#destinationgeolocationlongitude","title":"destination.geolocation.longitude","text":"
      Type: Float
       
      Longitude coordinates derived from a geolocation service, such as MaxMind geoip db.
      "},{"location":"user/event/#destinationgeolocationregion","title":"destination.geolocation.region","text":"
      Type: String
       
      Some geolocation services refer to region-level geolocation.
      "},{"location":"user/event/#destinationgeolocationstate","title":"destination.geolocation.state","text":"
      Type: String
       
      Some geolocation services refer to state-level geolocation.
      "},{"location":"user/event/#destinationip","title":"destination.ip","text":"
      Type: IPAddress
       
      The IP which is the target of the observed connections.
      "},{"location":"user/event/#destinationlocal_hostname","title":"destination.local_hostname","text":"
      Type: String
       
      Some sources report an internal hostname within a NAT related to the name configured for a compromised system
      "},{"location":"user/event/#destinationlocal_ip","title":"destination.local_ip","text":"
      Type: IPAddress
       
      Some sources report an internal (NATed) IP address related a compromised system. N.B. RFC1918 IPs are OK here.
      "},{"location":"user/event/#destinationnetwork","title":"destination.network","text":"
      Type: IPNetwork
       
      CIDR for an autonomous system. Also known as BGP prefix. If multiple values are possible, select the most specific.
      "},{"location":"user/event/#destinationport","title":"destination.port","text":"
      Type: Integer
       
      The port to which the connection headed.
      "},{"location":"user/event/#destinationregistry","title":"destination.registry","text":"
      Type: Registry
       
      The IP registry a given ip address is allocated by.
      "},{"location":"user/event/#destinationreverse_dns","title":"destination.reverse_dns","text":"
      Type: FQDN
       
      Reverse DNS name acquired through a reverse DNS query on an IP address. N.B. Record types other than PTR records may also appear in the reverse DNS tree. Furthermore, unfortunately, there is no rule prohibiting people from writing anything in a PTR record. Even JavaScript will work. A final point is stripped, string is converted to lower case characters.
      "},{"location":"user/event/#destinationtor_node","title":"destination.tor_node","text":"
      Type: Boolean
       
      If the destination IP was a known tor node.
      "},{"location":"user/event/#destinationurl","title":"destination.url","text":"
      Type: URL
       
      A URL denotes on IOC, which refers to a malicious resource, whose interpretation is defined by the abuse type. A URL with the abuse type phishing refers to a phishing resource.
      "},{"location":"user/event/#destinationurlpath","title":"destination.urlpath","text":"
      Type: String
       
      The path portion of an HTTP or related network request.
      "},{"location":"user/event/#event_descriptiontarget","title":"event_description.target","text":"
      Type: String
       
      Some sources denominate the target (organization) of a an attack.
      "},{"location":"user/event/#event_descriptiontext","title":"event_description.text","text":"
      Type: String
       
      A free-form textual description of an abuse event.
      "},{"location":"user/event/#event_descriptionurl","title":"event_description.url","text":"
      Type: URL
       
      A description URL is a link to a further description of the the abuse event in question.
      "},{"location":"user/event/#event_hash","title":"event_hash","text":"
      Type: UppercaseString
       
      Computed event hash with specific keys and values that identify a unique event. At present, the hash should default to using the SHA1 function. Please note that for an event hash to be able to match more than one event (deduplication) the receiver of an event should calculate it based on a minimal set of keys and values present in the event. Using for example the observation time in the calculation will most likely render the checksum useless for deduplication purposes.
      "},{"location":"user/event/#extra","title":"extra","text":"
      Type: JSONDict
       
      All anecdotal information, which cannot be parsed into the data harmonization elements. E.g. os.name, os.version, etc.  Note: this is only intended for mapping any fields which can not map naturally into the data harmonization. It is not intended for extending the data harmonization with your own fields.
      "},{"location":"user/event/#feedaccuracy","title":"feed.accuracy","text":"
      Type: Accuracy
       
      A float between 0 and 100 that represents how accurate the data in the feed is
      "},{"location":"user/event/#feedcode","title":"feed.code","text":"
      Type: String
       
      Code name for the feed, e.g. DFGS, HSDAG etc.
      "},{"location":"user/event/#feeddocumentation","title":"feed.documentation","text":"
      Type: String
       
      A URL or hint where to find the documentation of this feed.
      "},{"location":"user/event/#feedname","title":"feed.name","text":"
      Type: String
       
      Name for the feed, usually found in collector bot configuration.
      "},{"location":"user/event/#feedprovider","title":"feed.provider","text":"
      Type: String
       
      Name for the provider of the feed, usually found in collector bot configuration.
      "},{"location":"user/event/#feedurl","title":"feed.url","text":"
      Type: URL
       
      The URL of a given abuse feed, where applicable
      "},{"location":"user/event/#malwarehashmd5","title":"malware.hash.md5","text":"
      Type: String
       
      A string depicting an MD5 checksum for a file, be it a malware sample for example.
      "},{"location":"user/event/#malwarehashsha1","title":"malware.hash.sha1","text":"
      Type: String
       
      A string depicting a SHA1 checksum for a file, be it a malware sample for example.
      "},{"location":"user/event/#malwarehashsha256","title":"malware.hash.sha256","text":"
      Type: String
       
      A string depicting a SHA256 checksum for a file, be it a malware sample for example.
      "},{"location":"user/event/#malwarename","title":"malware.name","text":"
      Type: LowercaseString
       
      The malware name in lower case.
      "},{"location":"user/event/#malwareversion","title":"malware.version","text":"
      Type: String
       
      A version string for an identified artifact generation, e.g. a crime-ware kit.
      "},{"location":"user/event/#mispattribute_uuid","title":"misp.attribute_uuid","text":"
      Type: LowercaseString
       
      MISP - Malware Information Sharing Platform & Threat Sharing UUID of an attribute.
      "},{"location":"user/event/#mispevent_uuid","title":"misp.event_uuid","text":"
      Type: LowercaseString
       
      MISP - Malware Information Sharing Platform & Threat Sharing UUID.
      "},{"location":"user/event/#output","title":"output","text":"
      Type: JSON
       
      Event data converted into foreign format, intended to be exported by output plugin.
      "},{"location":"user/event/#protocolapplication","title":"protocol.application","text":"
      Type: LowercaseString
       
      e.g. vnc, ssh, sip, irc, http or smtp.
      "},{"location":"user/event/#protocoltransport","title":"protocol.transport","text":"
      Type: LowercaseString
       
      e.g. tcp, udp, icmp.
      "},{"location":"user/event/#raw","title":"raw","text":"
      Type: Base64
       
      The original line of the event from encoded in base64.
      "},{"location":"user/event/#rtir_id","title":"rtir_id","text":"
      Type: Integer
       
      Request Tracker Incident Response ticket id.
      "},{"location":"user/event/#screenshot_url","title":"screenshot_url","text":"
      Type: URL
       
      Some source may report URLs related to a an image generated of a resource without any metadata. Or an URL pointing to resource, which has been rendered into a webshot, e.g. a PNG image and the relevant metadata related to its retrieval/generation.
      "},{"location":"user/event/#sourceabuse_contact","title":"source.abuse_contact","text":"
      Type: LowercaseString
       
      Abuse contact for source address. A comma separated list.
      "},{"location":"user/event/#sourceaccount","title":"source.account","text":"
      Type: String
       
      An account name or email address, which has been identified to relate to the source of an abuse event.
      "},{"location":"user/event/#sourceallocated","title":"source.allocated","text":"
      Type: DateTime
       
      Allocation date corresponding to BGP prefix.
      "},{"location":"user/event/#sourceas_name","title":"source.as_name","text":"
      Type: String
       
      The autonomous system name from which the connection originated.
      "},{"location":"user/event/#sourceasn","title":"source.asn","text":"
      Type: ASN
       
      The autonomous system number from which originated the connection.
      "},{"location":"user/event/#sourcedomain_suffix","title":"source.domain_suffix","text":"
      Type: FQDN
       
      The suffix of the domain from the public suffix list.
      "},{"location":"user/event/#sourcefqdn","title":"source.fqdn","text":"
      Type: FQDN
       
      A DNS name related to the host from which the connection originated. DNS allows even binary data in DNS, so we have to allow everything. A final point is stripped, string is converted to lower case characters.
      "},{"location":"user/event/#sourcegeolocationcc","title":"source.geolocation.cc","text":"
      Type: UppercaseString
       
      Country-Code according to ISO3166-1 alpha-2 for the source IP.
      "},{"location":"user/event/#sourcegeolocationcity","title":"source.geolocation.city","text":"
      Type: String
       
      Some geolocation services refer to city-level geolocation.
      "},{"location":"user/event/#sourcegeolocationcountry","title":"source.geolocation.country","text":"
      Type: String
       
      The country name derived from the ISO3166 country code (assigned to cc field).
      "},{"location":"user/event/#sourcegeolocationcymru_cc","title":"source.geolocation.cymru_cc","text":"
      Type: UppercaseString
       
      The country code denoted for the ip by the Team Cymru asn to ip mapping service.
      "},{"location":"user/event/#sourcegeolocationgeoip_cc","title":"source.geolocation.geoip_cc","text":"
      Type: UppercaseString
       
      MaxMind Country Code (ISO3166-1 alpha-2).
      "},{"location":"user/event/#sourcegeolocationlatitude","title":"source.geolocation.latitude","text":"
      Type: Float
       
      Latitude coordinates derived from a geolocation service, such as MaxMind geoip db.
      "},{"location":"user/event/#sourcegeolocationlongitude","title":"source.geolocation.longitude","text":"
      Type: Float
       
      Longitude coordinates derived from a geolocation service, such as MaxMind geoip db.
      "},{"location":"user/event/#sourcegeolocationregion","title":"source.geolocation.region","text":"
      Type: String
       
      Some geolocation services refer to region-level geolocation.
      "},{"location":"user/event/#sourcegeolocationstate","title":"source.geolocation.state","text":"
      Type: String
       
      Some geolocation services refer to state-level geolocation.
      "},{"location":"user/event/#sourceip","title":"source.ip","text":"
      Type: IPAddress
       
      The ip observed to initiate the connection
      "},{"location":"user/event/#sourcelocal_hostname","title":"source.local_hostname","text":"
      Type: String
       
      Some sources report a internal hostname within a NAT related to the name configured for a compromised system
      "},{"location":"user/event/#sourcelocal_ip","title":"source.local_ip","text":"
      Type: IPAddress
       
      Some sources report a internal (NATed) IP address related a compromised system. N.B. RFC1918 IPs are OK here.
      "},{"location":"user/event/#sourcenetwork","title":"source.network","text":"
      Type: IPNetwork
       
      CIDR for an autonomous system. Also known as BGP prefix. If multiple values are possible, select the most specific.
      "},{"location":"user/event/#sourceport","title":"source.port","text":"
      Type: Integer
       
      The port from which the connection originated.
      "},{"location":"user/event/#sourceregistry","title":"source.registry","text":"
      Type: Registry
       
      The IP registry a given ip address is allocated by.
      "},{"location":"user/event/#sourcereverse_dns","title":"source.reverse_dns","text":"
      Type: FQDN
       
      Reverse DNS name acquired through a reverse DNS query on an IP address. N.B. Record types other than PTR records may also appear in the reverse DNS tree. Furthermore, unfortunately, there is no rule prohibiting people from writing anything in a PTR record. Even JavaScript will work. A final point is stripped, string is converted to lower case characters.
      "},{"location":"user/event/#sourcetor_node","title":"source.tor_node","text":"
      Type: Boolean
       
      If the source IP was a known tor node.
      "},{"location":"user/event/#sourceurl","title":"source.url","text":"
      Type: URL
       
      A URL denotes an IOC, which refers to a malicious resource, whose interpretation is defined by the abuse type. A URL with the abuse type phishing refers to a phishing resource.
      "},{"location":"user/event/#sourceurlpath","title":"source.urlpath","text":"
      Type: String
       
      The path portion of an HTTP or related network request.
      "},{"location":"user/event/#status","title":"status","text":"
      Type: String
       
      Status of the malicious resource (phishing, dropzone, etc), e.g. online, offline.
      "},{"location":"user/event/#timeobservation","title":"time.observation","text":"
      Type: DateTime
       
      The time the collector of the local instance processed (observed) the event.
      "},{"location":"user/event/#timesource","title":"time.source","text":"
      Type: DateTime
       
      The time of occurrence of the event as reported the feed (source).
      "},{"location":"user/event/#tlp","title":"tlp","text":"
      Type: TLP
       
      Traffic Light Protocol level of the event.
      "},{"location":"user/feeds/","title":"Feeds","text":""},{"location":"user/feeds/#feeds","title":"Feeds","text":"
      The available feeds are grouped by the provider of the feeds. For each feed the collector and parser that can be used is documented as well as any feed-specific parameters. To add feeds to this file add them to intelmq/etc/feeds.yaml and then rebuild the documentation.
      "},{"location":"user/feeds/#abusech","title":"Abuse.ch","text":""},{"location":"user/feeds/#feodo-tracker","title":"Feodo Tracker","text":"
      List of botnet Command & Control servers (C&Cs) tracked by Feodo Tracker, associated with Dridex and Emotet (aka Heodo).
       
      Public: yes
       
      Revision: 2022-11-15
       
      Documentation: https://feodotracker.abuse.ch/
       
      Additional Information: The data in the column Last Online is used for time.source if available, with 00:00 as time. Otherwise first_seen is used as time.source.
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://feodotracker.abuse.ch/downloads/ipblocklist.json\n  name: Feodo Tracker\n  provider: Abuse.ch\n  rate_limit: 3600\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.abusech.parser_feodotracker\n
      "},{"location":"user/feeds/#urlhaus","title":"URLhaus","text":"
      URLhaus is a project from abuse.ch with the goal of sharing malicious URLs that are being used for malware distribution. URLhaus offers a country, ASN (AS number) and Top Level Domain (TLD) feed for network operators / Internet Service Providers (ISPs), Computer Emergency Response Teams (CERTs) and domain registries.
       
      Public: yes
       
      Revision: 2020-07-07
       
      Documentation: https://urlhaus.abuse.ch/feeds/
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://urlhaus.abuse.ch/feeds/tld/<TLD>/, https://urlhaus.abuse.ch/feeds/country/<CC>/, or https://urlhaus.abuse.ch/feeds/asn/<ASN>/\n  name: URLhaus\n  provider: Abuse.ch\n  rate_limit: 86400\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.generic.parser_csv\nparameters:\n  columns: [\"time.source\", \"source.url\", \"status\", \"classification.type|__IGNORE__\", \"source.fqdn|__IGNORE__\", \"source.ip\", \"source.asn\", \"source.geolocation.cc\"]\n  default_url_protocol: http://\n  delimiter: ,\n  skip_header: False\n  type_translation: [{\"malware_download\": \"malware-distribution\"}]\n
      "},{"location":"user/feeds/#alienvault","title":"AlienVault","text":""},{"location":"user/feeds/#otx","title":"OTX","text":"
      AlienVault OTX Collector is the bot responsible to get the report through the API. Report could vary according to subscriptions.
       
      Public: no
       
      Revision: 2018-01-20
       
      Documentation: https://otx.alienvault.com/
       
      Collector configuration
       
      module: intelmq.bots.collectors.alienvault_otx.collector\nparameters:\n  api_key: {{ your API key }}\n  name: OTX\n  provider: AlienVault\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.alienvault.parser_otx\n
      "},{"location":"user/feeds/#reputation-list","title":"Reputation List","text":"
      List of malicious IPs.
       
      Public: yes
       
      Revision: 2018-01-20
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://reputation.alienvault.com/reputation.data\n  name: Reputation List\n  provider: AlienVault\n  rate_limit: 3600\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.alienvault.parser\n
      "},{"location":"user/feeds/#anubisnetworks","title":"AnubisNetworks","text":""},{"location":"user/feeds/#cyberfeed-stream","title":"Cyberfeed Stream","text":"
      Fetches and parsers the Cyberfeed data stream.
       
      Public: no
       
      Revision: 2020-06-15
       
      Documentation: https://www.anubisnetworks.com/ https://www.bitsight.com/
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http_stream\nparameters:\n  http_url: https://prod.cyberfeed.net/stream?key={{ your API key }}\n  name: Cyberfeed Stream\n  provider: AnubisNetworks\n  strip_lines: true\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.anubisnetworks.parser\nparameters:\n  use_malware_familiy_as_classification_identifier: True\n
      "},{"location":"user/feeds/#bambenek","title":"Bambenek","text":""},{"location":"user/feeds/#c2-domains","title":"C2 Domains","text":"
      Master Feed of known, active and non-sinkholed C&Cs domain names. Requires access credentials.
       
      Public: no
       
      Revision: 2018-01-20
       
      Documentation: https://osint.bambenekconsulting.com/feeds/
       
      Additional Information: License: https://osint.bambenekconsulting.com/license.txt
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_password: __PASSWORD__\n  http_url: https://faf.bambenekconsulting.com/feeds/c2-dommasterlist.txt\n  http_username: __USERNAME__\n  name: C2 Domains\n  provider: Bambenek\n  rate_limit: 3600\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.bambenek.parser\n
      "},{"location":"user/feeds/#c2-ips","title":"C2 IPs","text":"
      Master Feed of known, active and non-sinkholed C&Cs IP addresses. Requires access credentials.
       
      Public: no
       
      Revision: 2018-01-20
       
      Documentation: https://osint.bambenekconsulting.com/feeds/
       
      Additional Information: License: https://osint.bambenekconsulting.com/license.txt
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_password: __PASSWORD__\n  http_url: https://faf.bambenekconsulting.com/feeds/c2-ipmasterlist.txt\n  http_username: __USERNAME__\n  name: C2 IPs\n  provider: Bambenek\n  rate_limit: 3600\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.bambenek.parser\n
      "},{"location":"user/feeds/#dga-domains","title":"DGA Domains","text":"
      Domain feed of known DGA domains from -2 to +3 days
       
      Public: yes
       
      Revision: 2018-01-20
       
      Documentation: https://osint.bambenekconsulting.com/feeds/
       
      Additional Information: License: https://osint.bambenekconsulting.com/license.txt
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://faf.bambenekconsulting.com/feeds/dga-feed.txt\n  name: DGA Domains\n  provider: Bambenek\n  rate_limit: 3600\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.bambenek.parser\n
      "},{"location":"user/feeds/#benkow","title":"Benkow","text":""},{"location":"user/feeds/#malware-panels-tracker","title":"Malware Panels Tracker","text":"
      Benkow Panels tracker is a list of fresh panel from various malware. The feed is available on the webpage: http://benkow.cc/passwords.php
       
      Public: yes
       
      Revision: 2022-11-16
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: http://benkow.cc/export.php\n  name: Malware Panels Tracker\n  provider: Benkow\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.generic.parser_csv\nparameters:\n  columns: [\"__IGNORE__\", \"malware.name\", \"source.url\", \"source.fqdn|source.ip\", \"time.source\"]\n  columns_required: [false, true, true, false, true]\n  defaults_fields: {'classification.type': 'c2-server'}\n  delimiter: ;\n  skip_header: True\n
      "},{"location":"user/feeds/#blocklistde","title":"Blocklist.de","text":""},{"location":"user/feeds/#apache","title":"Apache","text":"
      Blocklist.DE Apache Collector is the bot responsible to get the report from source of information. All IP addresses which have been reported within the last 48 hours as having run attacks on the service Apache, Apache-DDOS, RFI-Attacks.
       
      Public: yes
       
      Revision: 2018-01-20
       
      Documentation: http://www.blocklist.de/en/export.html
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://lists.blocklist.de/lists/apache.txt\n  name: Apache\n  provider: Blocklist.de\n  rate_limit: 86400\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.blocklistde.parser\n
      "},{"location":"user/feeds/#bots","title":"Bots","text":"
      Blocklist.DE Bots Collector is the bot responsible to get the report from source of information. All IP addresses which have been reported within the last 48 hours as having run attacks attacks on the RFI-Attacks, REG-Bots, IRC-Bots or BadBots (BadBots = he has posted a Spam-Comment on a open Forum or Wiki).
       
      Public: yes
       
      Revision: 2018-01-20
       
      Documentation: http://www.blocklist.de/en/export.html
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://lists.blocklist.de/lists/bots.txt\n  name: Bots\n  provider: Blocklist.de\n  rate_limit: 86400\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.blocklistde.parser\n
      "},{"location":"user/feeds/#brute-force-logins","title":"Brute-force Logins","text":"
      Blocklist.DE Brute-force Login Collector is the bot responsible to get the report from source of information. All IPs which attacks Joomlas, Wordpress and other Web-Logins with Brute-Force Logins.
       
      Public: yes
       
      Revision: 2018-01-20
       
      Documentation: http://www.blocklist.de/en/export.html
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://lists.blocklist.de/lists/bruteforcelogin.txt\n  name: Brute-force Logins\n  provider: Blocklist.de\n  rate_limit: 86400\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.blocklistde.parser\n
      "},{"location":"user/feeds/#ftp","title":"FTP","text":"
      Blocklist.DE FTP Collector is the bot responsible to get the report from source of information. All IP addresses which have been reported within the last 48 hours for attacks on the Service FTP.
       
      Public: yes
       
      Revision: 2018-01-20
       
      Documentation: http://www.blocklist.de/en/export.html
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://lists.blocklist.de/lists/ftp.txt\n  name: FTP\n  provider: Blocklist.de\n  rate_limit: 86400\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.blocklistde.parser\n
      "},{"location":"user/feeds/#imap","title":"IMAP","text":"
      Blocklist.DE IMAP Collector is the bot responsible to get the report from source of information. All IP addresses which have been reported within the last 48 hours for attacks on the service like IMAP, SASL, POP3, etc.
       
      Public: yes
       
      Revision: 2018-01-20
       
      Documentation: http://www.blocklist.de/en/export.html
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://lists.blocklist.de/lists/imap.txt\n  name: IMAP\n  provider: Blocklist.de\n  rate_limit: 86400\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.blocklistde.parser\n
      "},{"location":"user/feeds/#irc-bots","title":"IRC Bots","text":"
      No description provided by feed provider.
       
      Public: yes
       
      Revision: 2018-01-20
       
      Documentation: http://www.blocklist.de/en/export.html
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://lists.blocklist.de/lists/ircbot.txt\n  name: IRC Bots\n  provider: Blocklist.de\n  rate_limit: 86400\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.blocklistde.parser\n
      "},{"location":"user/feeds/#mail","title":"Mail","text":"
      Blocklist.DE Mail Collector is the bot responsible to get the report from source of information. All IP addresses which have been reported within the last 48 hours as having run attacks on the service Mail, Postfix.
       
      Public: yes
       
      Revision: 2018-01-20
       
      Documentation: http://www.blocklist.de/en/export.html
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://lists.blocklist.de/lists/mail.txt\n  name: Mail\n  provider: Blocklist.de\n  rate_limit: 86400\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.blocklistde.parser\n
      "},{"location":"user/feeds/#sip","title":"SIP","text":"
      Blocklist.DE SIP Collector is the bot responsible to get the report from source of information. All IP addresses that tried to login in a SIP-, VOIP- or Asterisk-Server and are included in the IPs-List from http://www.infiltrated.net/ (Twitter).
       
      Public: yes
       
      Revision: 2018-01-20
       
      Documentation: http://www.blocklist.de/en/export.html
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://lists.blocklist.de/lists/sip.txt\n  name: SIP\n  provider: Blocklist.de\n  rate_limit: 86400\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.blocklistde.parser\n
      "},{"location":"user/feeds/#ssh","title":"SSH","text":"
      Blocklist.DE SSH Collector is the bot responsible to get the report from source of information. All IP addresses which have been reported within the last 48 hours as having run attacks on the service SSH.
       
      Public: yes
       
      Revision: 2018-01-20
       
      Documentation: http://www.blocklist.de/en/export.html
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://lists.blocklist.de/lists/ssh.txt\n  name: SSH\n  provider: Blocklist.de\n  rate_limit: 86400\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.blocklistde.parser\n
      "},{"location":"user/feeds/#strong-ips","title":"Strong IPs","text":"
      Blocklist.DE Strong IPs Collector is the bot responsible to get the report from source of information. All IPs which are older then 2 month and have more then 5.000 attacks.
       
      Public: yes
       
      Revision: 2018-01-20
       
      Documentation: http://www.blocklist.de/en/export.html
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://lists.blocklist.de/lists/strongips.txt\n  name: Strong IPs\n  provider: Blocklist.de\n  rate_limit: 86400\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.blocklistde.parser\n
      "},{"location":"user/feeds/#blueliv","title":"Blueliv","text":""},{"location":"user/feeds/#crimeserver","title":"CrimeServer","text":"
      Blueliv Crimeserver Collector is the bot responsible to get the report through the API.
       
      Public: no
       
      Revision: 2018-01-20
       
      Documentation: https://www.blueliv.com/
       
      Additional Information: The service uses a different API for free users and paying subscribers. In 'CrimeServer' feed the difference lies in the data points present in the feed. The non-free API available from Blueliv contains, for this specific feed, following extra fields not present in the free API; \"_id\" - Internal unique ID \"subType\" - Subtype of the Crime Server \"countryName\" - Country name where the Crime Server is located, in English \"city\" - City where the Crime Server is located \"domain\" - Domain of the Crime Server \"host\" - Host of the Crime Server \"createdAt\" - Date when the Crime Server was added to Blueliv CrimeServer database \"asnCidr\" - Range of IPs that belong to an ISP (registered via Autonomous System Number (ASN)) \"asnId\" - Identifier of an ISP registered via ASN \"asnDesc\" Description of the ISP registered via ASN
       
      Collector configuration
       
      module: intelmq.bots.collectors.blueliv.collector_crimeserver\nparameters:\n  api_key: __APIKEY__\n  name: CrimeServer\n  provider: Blueliv\n  rate_limit: 3600\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.blueliv.parser_crimeserver\n
      "},{"location":"user/feeds/#cert-bund","title":"CERT-Bund","text":""},{"location":"user/feeds/#cb-report-malware-infections-via-imap","title":"CB-Report Malware infections via IMAP","text":"
      CERT-Bund sends reports for the malware-infected hosts.
       
      Public: no
       
      Revision: 2020-08-20
       
      Additional Information: Traffic from malware related hosts contacting command-and-control servers is caught and sent to national CERT teams. There are two e-mail feeds with identical CSV structure -- one reports on general malware infections, the other on the Avalanche botnet.
       
      Collector configuration
       
      module: intelmq.bots.collectors.mail.collector_mail_attach\nparameters:\n  attach_regex: events.csv\n  extract_files: False\n  folder: INBOX\n  mail_host: __HOST__\n  mail_password: __PASSWORD__\n  mail_ssl: True\n  mail_user: __USERNAME__\n  name: CB-Report Malware infections via IMAP\n  provider: CERT-Bund\n  rate_limit: 86400\n  subject_regex: ^\\\\[CB-Report#.* Malware infections (\\\\(Avalanche\\\\) )?in country\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.generic.parser_csv\nparameters:\n  columns: [\"source.asn\", \"source.ip\", \"time.source\", \"classification.type\", \"malware.name\", \"source.port\", \"destination.ip\", \"destination.port\", \"destination.fqdn\", \"protocol.transport\"]\n  default_url_protocol: http://\n  defaults_fields: {'classification.type': 'infected-system'}\n  delimiter: ,\n  skip_header: True\n  time_format: from_format|%Y-%m-%d %H:%M:%S\n
      "},{"location":"user/feeds/#certpl","title":"CERT.PL","text":""},{"location":"user/feeds/#n6-stomp-stream","title":"N6 Stomp Stream","text":"
      N6 Collector - CERT.pl's n6 Stream API feed (via STOMP interface). Note that 'rate_limit' does not apply to this bot, as it is waiting for messages on a stream.
       
      Public: no
       
      Revision: 2023-10-08
       
      Documentation: https://n6.readthedocs.io/usage/streamapi/
       
      Additional Information: Contact CERT.pl to get access to the feed. Note that the configuration parameter values suggested here are suitable for the new n6 Stream API variant (with authentication based on 'username' and 'password'); for this variant, typically you can leave the 'ssl_ca_certificate' parameter's value empty - then the system's default CA certificates will be used; however, if that does not work, you need to set 'ssl_ca_certificate' to the path to a file containing CA certificates eligible to verify \".cert.pl\" server certificates (to be found among the publicly available CA certs distributed with modern web browsers/OSes). Also, note that the 'server' parameter's value (for the new API variant) suggested here, \"n6stream-new.cert.pl\", is a temporary domain; ultimately, it will be changed back to \"stream.cert.pl\". When it comes to the old API variant (turned off in November 2023!), you need to have the 'server' parameter set to the name \"n6stream.cert.pl\", 'auth_by_ssl_client_certificate' set to true, 'ssl_ca_certificate' set to the path to a file containing the n6's legacy self-signed CA certificate (which is stored in file \"intelmq/bots/collectors/stomp/ca.pem\"), and the parameters 'ssl_client_certificate' and 'ssl_client_certificate_key' set to the paths to your-n6*-client-specific certificate and key files (note that the 'username' and 'password' parameters are then irrelevant and can be omitted).
       
      Collector configuration
       
      module: intelmq.bots.collectors.stomp.collector\nparameters:\n  auth_by_ssl_client_certificate: False\n  exchange: {insert your STOMP *destination* to subscribe to, as given by CERT.pl, e.g. /exchange/my.example.org/*.*.*.*}\n  name: N6 Stomp Stream\n  password: {insert your *n6* API key}\n  port: 61614\n  provider: CERT.PL\n  server: n6stream-new.cert.pl\n  ssl_ca_certificate: \n  username: {insert your *n6* login, e.g. someuser@my.example.org}\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.n6.parser_n6stomp\n
      "},{"location":"user/feeds/#cins-army","title":"CINS Army","text":""},{"location":"user/feeds/#cins-army-list","title":"CINS Army List","text":"
      The CINS Army (CIArmy.com) list is a subset of the CINS Active Threat Intelligence ruleset, and consists of IP addresses that meet one of two basic criteria: 1) The IP's recent Rogue Packet score factor is very poor, or 2) The IP has tripped a designated number of 'trusted' alerts across a given number of our Sentinels deployed around the world.
       
      Public: yes
       
      Revision: 2018-01-20
       
      Documentation: https://cinsscore.com/#list
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: http://cinsscore.com/list/ci-badguys.txt\n  name: CINS Army List\n  provider: CINS Army\n  rate_limit: 3600\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.ci_army.parser\n
      "},{"location":"user/feeds/#cznic","title":"CZ.NIC","text":""},{"location":"user/feeds/#haas","title":"HaaS","text":"
      SSH attackers against HaaS (Honeypot as a Service) provided by CZ.NIC, z.s.p.o. The dump is published once a day.
       
      Public: yes
       
      Revision: 2020-07-22
       
      Documentation: https://haas.nic.cz/
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  extract_files: True\n  http_url: https://haas.nic.cz/stats/export/{time[%Y/%m/%Y-%m-%d]}.json.gz\n  http_url_formatting: {'days': -1}\n  rate_limit: 86400\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.cznic.parser_haas\n
      "},{"location":"user/feeds/#proki","title":"Proki","text":"
      Aggregation of various sources on malicious IP addresses (malware spreaders or C&C servers).
       
      Public: no
       
      Revision: 2020-08-17
       
      Documentation: https://csirt.cz/en/proki/
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://proki.csirt.cz/api/1/__APIKEY__/data/day/{time[%Y/%m/%d]}\n  http_url_formatting: {'days': -1}\n  name: Proki\n  provider: CZ.NIC\n  rate_limit: 86400\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.cznic.parser_proki\n
      "},{"location":"user/feeds/#calidog","title":"Calidog","text":""},{"location":"user/feeds/#certstream","title":"CertStream","text":"
      HTTP Websocket Stream from certstream.calidog.io providing data from Certificate Transparency Logs.
       
      Public: yes
       
      Revision: 2018-06-15
       
      Documentation: https://medium.com/cali-dog-security/introducing-certstream-3fc13bb98067
       
      Additional Information: Be aware that this feed provides a lot of data and may overload your system quickly.
       
      Collector configuration
       
      module: intelmq.bots.collectors.calidog.collector_certstream\nparameters:\n  name: CertStream\n  provider: Calidog\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.calidog.parser_certstream\n
      "},{"location":"user/feeds/#cleanmx","title":"CleanMX","text":""},{"location":"user/feeds/#phishing","title":"Phishing","text":"
      In order to download the CleanMX feed you need to use a custom user agent and register that user agent.
       
      Public: no
       
      Revision: 2018-01-20
       
      Documentation: http://clean-mx.de/
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_timeout_sec: 120\n  http_url: http://support.clean-mx.de/clean-mx/xmlphishing?response=alive&domain=\n  http_user_agent: {{ your user agent }}\n  name: Phishing\n  provider: CleanMX\n  rate_limit: 86400\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.cleanmx.parser\n
      "},{"location":"user/feeds/#virus","title":"Virus","text":"
      In order to download the CleanMX feed you need to use a custom user agent and register that user agent.
       
      Public: no
       
      Revision: 2018-01-20
       
      Documentation: http://clean-mx.de/
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_timeout_sec: 120\n  http_url: http://support.clean-mx.de/clean-mx/xmlviruses?response=alive&domain=\n  http_user_agent: {{ your user agent }}\n  name: Virus\n  provider: CleanMX\n  rate_limit: 86400\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.cleanmx.parser\n
      "},{"location":"user/feeds/#cybercrime-tracker","title":"CyberCrime Tracker","text":""},{"location":"user/feeds/#latest","title":"Latest","text":"
      C2 servers
       
      Public: yes
       
      Revision: 2019-03-19
       
      Documentation: https://cybercrime-tracker.net/index.php
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://cybercrime-tracker.net/index.php\n  name: Latest\n  provider: CyberCrime Tracker\n  rate_limit: 86400\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.html_table.parser\nparameters:\n  columns: [\"time.source\", \"source.url\", \"source.ip\", \"malware.name\", \"__IGNORE__\"]\n  default_url_protocol: http://\n  defaults_fields: {'classification.type': 'c2-server'}\n  skip_table_head: True\n
      "},{"location":"user/feeds/#dshield","title":"DShield","text":""},{"location":"user/feeds/#as-details","title":"AS Details","text":"
      No description provided by feed provider.
       
      Public: yes
       
      Revision: 2018-01-20
       
      Documentation: https://www.dshield.org/reports.html
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://dshield.org/asdetailsascii.html?as={{ AS Number }}\n  name: AS Details\n  provider: DShield\n  rate_limit: 86400\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.dshield.parser_asn\n
      "},{"location":"user/feeds/#block","title":"Block","text":"
      This list summarizes the top 20 attacking class C (/24) subnets over the last three days. The number of 'attacks' indicates the number of targets reporting scans from this subnet.
       
      Public: yes
       
      Revision: 2018-01-20
       
      Documentation: https://www.dshield.org/reports.html
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://www.dshield.org/block.txt\n  name: Block\n  provider: DShield\n  rate_limit: 86400\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.dshield.parser_block\n
      "},{"location":"user/feeds/#danger-rulez","title":"Danger Rulez","text":""},{"location":"user/feeds/#bruteforce-blocker","title":"Bruteforce Blocker","text":"
      Its main purpose is to block SSH bruteforce attacks via firewall.
       
      Public: yes
       
      Revision: 2018-01-20
       
      Documentation: http://danger.rulez.sk/index.php/bruteforceblocker/
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: http://danger.rulez.sk/projects/bruteforceblocker/blist.php\n  name: Bruteforce Blocker\n  provider: Danger Rulez\n  rate_limit: 3600\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.danger_rulez.parser\n
      "},{"location":"user/feeds/#dataplane","title":"Dataplane","text":""},{"location":"user/feeds/#dns-recursion-desired","title":"DNS Recursion Desired","text":"
      Entries consist of fields with identifying characteristics of a source IP address that has been seen performing a DNS recursion desired query to a remote host. This report lists hosts that are suspicious of more than just port scanning. The host may be DNS server cataloging or searching for hosts to use for DNS-based DDoS amplification.
       
      Public: yes
       
      Revision: 2021-09-09
       
      Documentation: https://dataplane.org/
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://dataplane.org/dnsrd.txt\n  name: DNS Recursion Desired\n  provider: Dataplane\n  rate_limit: 3600\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.dataplane.parser\n
      "},{"location":"user/feeds/#dns-recursion-desired-any","title":"DNS Recursion Desired ANY","text":"
      Entries consist of fields with identifying characteristics of a source IP address that has been seen performing a DNS recursion desired IN ANY query to a remote host. This report lists hosts that are suspicious of more than just port scanning. The host may be DNS server cataloging or searching for hosts to use for DNS-based DDoS amplification.
       
      Public: yes
       
      Revision: 2021-09-09
       
      Documentation: https://dataplane.org/
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://dataplane.org/dnsrdany.txt\n  name: DNS Recursion Desired ANY\n  provider: Dataplane\n  rate_limit: 3600\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.dataplane.parser\n
      "},{"location":"user/feeds/#dns-version","title":"DNS Version","text":"
      Entries consist of fields with identifying characteristics of a source IP address that has been seen performing a DNS CH TXT version.bind query to a remote host. This report lists hosts that are suspicious of more than just port scanning. The host may be DNS server cataloging or searching for vulnerable DNS servers.
       
      Public: yes
       
      Revision: 2021-09-09
       
      Documentation: https://dataplane.org/
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://dataplane.org/dnsversion.txt\n  name: DNS Version\n  provider: Dataplane\n  rate_limit: 3600\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.dataplane.parser\n
      "},{"location":"user/feeds/#protocol-41","title":"Protocol 41","text":"
      Entries consist of fields with identifying characteristics of a host that has been detected to offer open IPv6 over IPv4 tunneling. This could allow for the host to be used a public proxy against IPv6 hosts.
       
      Public: yes
       
      Revision: 2021-09-09
       
      Documentation: https://dataplane.org/
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://dataplane.org/proto41.txt\n  name: Protocol 41\n  provider: Dataplane\n  rate_limit: 3600\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.dataplane.parser\n
      "},{"location":"user/feeds/#sip-query","title":"SIP Query","text":"
      Entries consist of fields with identifying characteristics of a source IP address that has been seen initiating a SIP OPTIONS query to a remote host. This report lists hosts that are suspicious of more than just port scanning. The hosts may be SIP server cataloging or conducting various forms of telephony abuse. Report is updated hourly.
       
      Public: yes
       
      Revision: 2018-01-20
       
      Documentation: https://dataplane.org/
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://dataplane.org/sipquery.txt\n  name: SIP Query\n  provider: Dataplane\n  rate_limit: 3600\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.dataplane.parser\n
      "},{"location":"user/feeds/#sip-registration","title":"SIP Registration","text":"
      Entries consist of fields with identifying characteristics of a source IP address that has been seen initiating a SIP REGISTER operation to a remote host. This report lists hosts that are suspicious of more than just port scanning. The hosts may be SIP client cataloging or conducting various forms of telephony abuse. Report is updated hourly.
       
      Public: yes
       
      Revision: 2018-01-20
       
      Documentation: https://dataplane.org/
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://dataplane.org/sipregistration.txt\n  name: SIP Registration\n  provider: Dataplane\n  rate_limit: 3600\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.dataplane.parser\n
      "},{"location":"user/feeds/#smtp-data","title":"SMTP Data","text":"
      Entries consist of fields with identifying characteristics of a host that has been seen initiating a SMTP DATA operation to a remote host. The source report lists hosts that are suspicious of more than just port scanning. The host may be SMTP server cataloging or conducting various forms of email abuse.
       
      Public: yes
       
      Revision: 2021-09-09
       
      Documentation: https://dataplane.org/
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://dataplane.org/smtpdata.txt\n  name: SMTP Data\n  provider: Dataplane\n  rate_limit: 3600\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.dataplane.parser\n
      "},{"location":"user/feeds/#smtp-greet","title":"SMTP Greet","text":"
      Entries consist of fields with identifying characteristics of a host that has been seen initiating a SMTP HELO/EHLO operation to a remote host. The source report lists hosts that are suspicious of more than just port scanning. The host may be SMTP server cataloging or conducting various forms of email abuse.
       
      Public: yes
       
      Revision: 2021-09-09
       
      Documentation: https://dataplane.org/
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://dataplane.org/smtpgreet.txt\n  name: SMTP Greet\n  provider: Dataplane\n  rate_limit: 3600\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.dataplane.parser\n
      "},{"location":"user/feeds/#ssh-client-connection","title":"SSH Client Connection","text":"
      Entries below consist of fields with identifying characteristics of a source IP address that has been seen initiating an SSH connection to a remote host. This report lists hosts that are suspicious of more than just port scanning. The hosts may be SSH server cataloging or conducting authentication attack attempts. Report is updated hourly.
       
      Public: yes
       
      Revision: 2018-01-20
       
      Documentation: https://dataplane.org/
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://dataplane.org/sshclient.txt\n  name: SSH Client Connection\n  provider: Dataplane\n  rate_limit: 3600\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.dataplane.parser\n
      "},{"location":"user/feeds/#ssh-password-authentication","title":"SSH Password Authentication","text":"
      Entries below consist of fields with identifying characteristics of a source IP address that has been seen attempting to remotely login to a host using SSH password authentication. The report lists hosts that are highly suspicious and are likely conducting malicious SSH password authentication attacks. Report is updated hourly.
       
      Public: yes
       
      Revision: 2018-01-20
       
      Documentation: https://dataplane.org/
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://dataplane.org/sshpwauth.txt\n  name: SSH Password Authentication\n  provider: Dataplane\n  rate_limit: 3600\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.dataplane.parser\n
      "},{"location":"user/feeds/#telnet-login","title":"Telnet Login","text":"
      Entries consist of fields with identifying characteristics of a host that has been seen initiating a telnet connection to a remote host. The source report lists hosts that are suspicious of more than just port scanning. The host may be telnet server cataloging or conducting authentication attack attempts.
       
      Public: yes
       
      Revision: 2021-09-09
       
      Documentation: https://dataplane.org/
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://dataplane.org/telnetlogin.txt\n  name: Telnet Login\n  provider: Dataplane\n  rate_limit: 3600\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.dataplane.parser\n
      "},{"location":"user/feeds/#vncrfb-login","title":"VNC/RFB Login","text":"
      Entries consist of fields with identifying characteristics of a host that has been seen initiating a VNC remote buffer session to a remote host. The source report lists hosts that are suspicious of more than just port scanning. The host may be VNC/RFB server cataloging or conducting authentication attack attempts.
       
      Public: yes
       
      Revision: 2021-09-09
       
      Documentation: https://dataplane.org/
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://dataplane.org/vncrfb.txt\n  name: VNC/RFB Login\n  provider: Dataplane\n  rate_limit: 3600\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.dataplane.parser\n
      "},{"location":"user/feeds/#eset","title":"ESET","text":""},{"location":"user/feeds/#eti-domains","title":"ETI Domains","text":"
      Domain data from ESET's TAXII API.
       
      Public: no
       
      Revision: 2020-06-30
       
      Documentation: https://www.eset.com/int/business/services/threat-intelligence/
       
      Collector configuration
       
      module: intelmq.bots.collectors.eset.collector\nparameters:\n  collection: ei.domains v2 (json)\n  endpoint: eti.eset.com\n  password: <password>\n  time_delta: 3600\n  username: <username>\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.eset.parser\n
      "},{"location":"user/feeds/#eti-urls","title":"ETI URLs","text":"
      URL data from ESET's TAXII API.
       
      Public: no
       
      Revision: 2020-06-30
       
      Documentation: https://www.eset.com/int/business/services/threat-intelligence/
       
      Collector configuration
       
      module: intelmq.bots.collectors.eset.collector\nparameters:\n  collection: ei.urls (json)\n  endpoint: eti.eset.com\n  password: <password>\n  time_delta: 3600\n  username: <username>\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.eset.parser\n
      "},{"location":"user/feeds/#fireeye","title":"Fireeye","text":""},{"location":"user/feeds/#malware-analysis-system","title":"Malware Analysis System","text":"
      Process data from Fireeye mail and file analysis appliances. SHA1 and MD5 malware hashes are extracted and if there is network communication, also URLs and domains.
       
      Public: no
       
      Revision: 2021-05-03
       
      Documentation: https://www.fireeye.com/products/malware-analysis.html
       
      Collector configuration
       
      module: intelmq.bots.collectors.fireeye.collector_mas\nparameters:\n  host: <hostname of your appliance>\n  http_password: <your password>\n  http_username: <your username>\n  request_duration: <how old date should be fetched eg 24_hours or 48_hours>\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.fireeye.parser\n
      "},{"location":"user/feeds/#fraunhofer","title":"Fraunhofer","text":""},{"location":"user/feeds/#dga-archive","title":"DGA Archive","text":"
      Fraunhofer DGA collector fetches data from Fraunhofer's domain generation archive.
       
      Public: no
       
      Revision: 2018-01-20
       
      Documentation: https://dgarchive.caad.fkie.fraunhofer.de/welcome/
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_password: {{ your password }}\n  http_url: https://dgarchive.caad.fkie.fraunhofer.de/today\n  http_username: {{ your username }}\n  name: DGA Archive\n  provider: Fraunhofer\n  rate_limit: 10800\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.fraunhofer.parser_dga\n
      "},{"location":"user/feeds/#have-i-been-pwned","title":"Have I Been Pwned","text":""},{"location":"user/feeds/#enterprise-callback","title":"Enterprise Callback","text":"
      With the Enterprise Subscription of 'Have I Been Pwned' you are able to provide a callback URL and any new leak data is submitted to it. It is recommended to put a webserver with Authorization check, TLS etc. in front of the API collector.
       
      Public: no
       
      Revision: 2019-09-11
       
      Documentation: https://haveibeenpwned.com/EnterpriseSubscriber/
       
      Additional Information: A minimal nginx configuration could look like: 
      server {\n    listen 443 ssl http2;\n    server_name [your host name];\n    client_max_body_size 50M;\n\n    ssl_certificate [path to your key];\n    ssl_certificate_key [path to your certificate];\n\n    location /[your private url] {\n         if ($http_authorization != '[your private password]') {\n             return 403;\n         }\n         proxy_pass http://localhost:5001/intelmq/push;\n         proxy_read_timeout 30;\n         proxy_connect_timeout 30;\n     }\n}\n
       
      Collector configuration
       
      module: intelmq.bots.collectors.api.collector_api\nparameters:\n  name: Enterprise Callback\n  port: 5001\n  provider: Have I Been Pwned\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.hibp.parser_callback\n
      "},{"location":"user/feeds/#malwarepatrol","title":"MalwarePatrol","text":""},{"location":"user/feeds/#dansguardian","title":"DansGuardian","text":"
      Malware block list with URLs
       
      Public: no
       
      Revision: 2018-01-20
       
      Documentation: https://www.malwarepatrol.net/non-commercial/
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://lists.malwarepatrol.net/cgi/getfile?receipt={{ your API key }}&product=8&list=dansguardian\n  name: DansGuardian\n  provider: MalwarePatrol\n  rate_limit: 180000\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.malwarepatrol.parser_dansguardian\n
      "},{"location":"user/feeds/#malwareurl","title":"MalwareURL","text":""},{"location":"user/feeds/#latest-malicious-activity","title":"Latest malicious activity","text":"
      Latest malicious domains/IPs.
       
      Public: yes
       
      Revision: 2018-02-05
       
      Documentation: https://www.malwareurl.com/
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://www.malwareurl.com/\n  name: Latest malicious activity\n  provider: MalwareURL\n  rate_limit: 86400\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.malwareurl.parser\n
      "},{"location":"user/feeds/#mcafee-advanced-threat-defense","title":"McAfee Advanced Threat Defense","text":""},{"location":"user/feeds/#sandbox-reports","title":"Sandbox Reports","text":"
      Processes reports from McAfee's sandboxing solution via the openDXL API.
       
      Public: no
       
      Revision: 2018-07-05
       
      Documentation: https://www.mcafee.com/enterprise/en-us/products/advanced-threat-defense.html
       
      Collector configuration
       
      module: intelmq.bots.collectors.opendxl.collector\nparameters:\n  dxl_config_file: {{ location of dxl configuration file }}\n  dxl_topic: /mcafee/event/atd/file/report\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.mcafee.parser_atd\nparameters:\n  verdict_severity: 4\n
      "},{"location":"user/feeds/#microsoft","title":"Microsoft","text":""},{"location":"user/feeds/#bingmurls-via-interflow","title":"BingMURLs via Interflow","text":"
      Collects Malicious URLs detected by Bing from the Interflow API. The feed is available via Microsoft\u2019s Government Security Program (GSP).
       
      Public: no
       
      Revision: 2018-05-29
       
      Documentation: https://docs.microsoft.com/en-us/security/gsp/informationsharingandexchange
       
      Additional Information: Depending on the file sizes you may need to increase the parameter 'http_timeout_sec' of the collector.
       
      Collector configuration
       
      module: intelmq.bots.collectors.microsoft.collector_interflow\nparameters:\n  api_key: {{ your API key }}\n  file_match: ^bingmurls_\n  http_timeout_sec: 300\n  name: BingMURLs via Interflow\n  not_older_than: 2 days\n  provider: Microsoft\n  rate_limit: 3600\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.microsoft.parser_bingmurls\n
      "},{"location":"user/feeds/#ctip-c2-via-azure","title":"CTIP C2 via Azure","text":"
      Collects the CTIP C2 feed from a shared Azure Storage. The feed is available via Microsoft\u2019s Government Security Program (GSP).
       
      Public: no
       
      Revision: 2020-05-29
       
      Documentation: https://docs.microsoft.com/en-us/security/gsp/informationsharingandexchange
       
      Additional Information: The cache is needed for memorizing which files have already been processed, the TTL should be higher than the oldest file available in the storage (currently the last three days are available). The connection string contains endpoint as well as authentication information.
       
      Collector configuration
       
      module: intelmq.bots.collectors.microsoft.collector_azure\nparameters:\n  connection_string: {{ your connection string }}\n  container_name: ctip-c2\n  name: CTIP C2 via Azure\n  provider: Microsoft\n  rate_limit: 3600\n  redis_cache_db: 5\n  redis_cache_host: 127.0.0.1\n  redis_cache_port: 6379\n  redis_cache_ttl: 864000\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.microsoft.parser_ctip\n
      "},{"location":"user/feeds/#ctip-infected-via-azure","title":"CTIP Infected via Azure","text":"
      Collects the CTIP (Sinkhole data) from a shared Azure Storage. The feed is available via Microsoft\u2019s Government Security Program (GSP).
       
      Public: no
       
      Revision: 2022-06-01
       
      Documentation: https://docs.microsoft.com/en-us/security/gsp/informationsharingandexchange http://www.dcuctip.com/
       
      Additional Information: The cache is needed for memorizing which files have already been processed, the TTL should be higher than the oldest file available in the storage (currently the last three days are available). The connection string contains endpoint as well as authentication information. As many IPs occur very often in the data, you may want to use a deduplicator specifically for the feed. More information about the feed can be found on www.dcuctip.com after login with your GSP account.
       
      Collector configuration
       
      module: intelmq.bots.collectors.microsoft.collector_azure\nparameters:\n  connection_string: {{ your connection string }}\n  container_name: ctip-infected-summary\n  name: CTIP Infected via Azure\n  provider: Microsoft\n  rate_limit: 3600\n  redis_cache_db: 5\n  redis_cache_host: 127.0.0.1\n  redis_cache_port: 6379\n  redis_cache_ttl: 864000\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.microsoft.parser_ctip\n
      "},{"location":"user/feeds/#ctip-infected-via-interflow","title":"CTIP Infected via Interflow","text":"
      Collects the CTIP Infected feed (Sinkhole data for your country) files from the Interflow API.The feed is available via Microsoft\u2019s Government Security Program (GSP).
       
      Public: no
       
      Revision: 2018-03-06
       
      Documentation: https://docs.microsoft.com/en-us/security/gsp/informationsharingandexchange http://www.dcuctip.com/
       
      Additional Information: Depending on the file sizes you may need to increase the parameter 'http_timeout_sec' of the collector. As many IPs occur very often in the data, you may want to use a deduplicator specifically for the feed. More information about the feed can be found on www.dcuctip.com after login with your GSP account.
       
      Collector configuration
       
      module: intelmq.bots.collectors.microsoft.collector_interflow\nparameters:\n  api_key: {{ your API key }}\n  file_match: ^ctip_\n  http_timeout_sec: 300\n  name: CTIP Infected via Interflow\n  not_older_than: 2 days\n  provider: Microsoft\n  rate_limit: 3600\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.microsoft.parser_ctip\n
      "},{"location":"user/feeds/#openphish","title":"OpenPhish","text":""},{"location":"user/feeds/#premium-feed","title":"Premium Feed","text":"
      OpenPhish is a fully automated self-contained platform for phishing intelligence. It identifies phishing sites and performs intelligence analysis in real time without human intervention and without using any external resources, such as blacklists.
       
      Public: no
       
      Revision: 2018-02-06
       
      Documentation: https://www.openphish.com/phishing_feeds.html
       
      Additional Information: Discounts available for Government and National CERTs a well as for Nonprofit and Not-for-Profit organizations.
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_password: {{ your password }}\n  http_url: https://openphish.com/prvt-intell/\n  http_username: {{ your username }}\n  name: Premium Feed\n  provider: OpenPhish\n  rate_limit: 86400\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.openphish.parser_commercial\n
      "},{"location":"user/feeds/#public-feed","title":"Public feed","text":"
      OpenPhish is a fully automated self-contained platform for phishing intelligence. It identifies phishing sites and performs intelligence analysis in real time without human intervention and without using any external resources, such as blacklists.
       
      Public: yes
       
      Revision: 2018-01-20
       
      Documentation: https://www.openphish.com/
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://www.openphish.com/feed.txt\n  name: Public feed\n  provider: OpenPhish\n  rate_limit: 86400\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.openphish.parser\n
      "},{"location":"user/feeds/#phishtank","title":"PhishTank","text":""},{"location":"user/feeds/#online","title":"Online","text":"
      PhishTank is a collaborative clearing house for data and information about phishing on the Internet.
       
      Public: no
       
      Revision: 2022-11-21
       
      Documentation: https://www.phishtank.com/developer_info.php
       
      Additional Information: Updated hourly as per the documentation. Download is possible without API key, but limited to few downloads per day.
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  extract_files: True\n  http_url: https://data.phishtank.com/data/{{ your API key }}/online-valid.json.gz\n  name: Online\n  provider: PhishTank\n  rate_limit: 3600\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.phishtank.parser\n
      "},{"location":"user/feeds/#precisionsec","title":"PrecisionSec","text":""},{"location":"user/feeds/#agent-tesla","title":"Agent Tesla","text":"
      Agent Tesla IoCs, URLs where the malware is hosted.
       
      Public: yes
       
      Revision: 2019-04-02
       
      Documentation: https://precisionsec.com/threat-intelligence-feeds/agent-tesla/
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://precisionsec.com/threat-intelligence-feeds/agent-tesla/\n  name: Agent Tesla\n  provider: PrecisionSec\n  rate_limit: 86400\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.html_table.parser\nparameters:\n  columns: [\"source.ip|source.url\", \"time.source\"]\n  default_url_protocol: http://\n  defaults_fields: {'classification.type': 'malware-distribution'}\n  skip_table_head: True\n
      "},{"location":"user/feeds/#shadowserver","title":"Shadowserver","text":""},{"location":"user/feeds/#via-api","title":"Via API","text":"
      Shadowserver sends out a variety of reports to subscribers, see documentation.
       
      Public: no
       
      Revision: 2020-01-08
       
      Documentation: https://www.shadowserver.org/what-we-do/network-reporting/api-documentation/
       
      Additional Information: This configuration fetches user-configurable reports from the Shadowserver Reports API. For a list of reports, have a look at the Shadowserver collector and parser documentation.
       
      Collector configuration
       
      module: intelmq.bots.collectors.shadowserver.collector_reports_api\nparameters:\n  api_key: <API key>\n  country: <CC>\n  rate_limit: 86400\n  redis_cache_db: 12\n  redis_cache_host: 127.0.0.1\n  redis_cache_port: 6379\n  redis_cache_ttl: 864000\n  secret: <API secret>\n  types: <single report or list of reports>\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.shadowserver.parser_json\n
      "},{"location":"user/feeds/#via-imap","title":"Via IMAP","text":"
      Shadowserver sends out a variety of reports (see https://www.shadowserver.org/wiki/pmwiki.php/Services/Reports).
       
      Public: no
       
      Revision: 2018-01-20
       
      Documentation: https://www.shadowserver.org/what-we-do/network-reporting/
       
      Additional Information: The configuration retrieves the data from a e-mails via IMAP from the attachments.
       
      Collector configuration
       
      module: intelmq.bots.collectors.mail.collector_mail_attach\nparameters:\n  attach_regex: csv.zip\n  extract_files: True\n  folder: INBOX\n  mail_host: __HOST__\n  mail_password: __PASSWORD__\n  mail_ssl: True\n  mail_user: __USERNAME__\n  name: Via IMAP\n  provider: Shadowserver\n  rate_limit: 86400\n  subject_regex: __REGEX__\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.shadowserver.parser\n
      "},{"location":"user/feeds/#via-request-tracker","title":"Via Request Tracker","text":"
      Shadowserver sends out a variety of reports (see https://www.shadowserver.org/wiki/pmwiki.php/Services/Reports).
       
      Public: no
       
      Revision: 2018-01-20
       
      Documentation: https://www.shadowserver.org/what-we-do/network-reporting/
       
      Additional Information: The configuration retrieves the data from a RT/RTIR ticketing instance via the attachment or an download.
       
      Collector configuration
       
      module: intelmq.bots.collectors.rt.collector_rt\nparameters:\n  attachment_regex: \\\\.csv\\\\.zip$\n  extract_attachment: True\n  extract_download: False\n  http_password: {{ your HTTP Authentication password or null }}\n  http_username: {{ your HTTP Authentication username or null }}\n  password: __PASSWORD__\n  provider: Shadowserver\n  rate_limit: 3600\n  search_not_older_than: {{ relative time or null }}\n  search_owner: nobody\n  search_queue: Incident Reports\n  search_requestor: autoreports@shadowserver.org\n  search_status: new\n  search_subject_like: \\[__COUNTRY__\\] Shadowserver __COUNTRY__\n  set_status: open\n  take_ticket: True\n  uri: http://localhost/rt/REST/1.0\n  url_regex: https://dl.shadowserver.org/[a-zA-Z0-9?_-]*\n  user: __USERNAME__\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.shadowserver.parser\n
      "},{"location":"user/feeds/#shodan","title":"Shodan","text":""},{"location":"user/feeds/#country-stream","title":"Country Stream","text":"
      Collects the Shodan stream for one or multiple countries from the Shodan API.
       
      Public: no
       
      Revision: 2021-03-22
       
      Documentation: https://developer.shodan.io/api/stream
       
      Additional Information: A Shodan account with streaming permissions is needed.
       
      Collector configuration
       
      module: intelmq.bots.collectors.shodan.collector_stream\nparameters:\n  api_key: <API key>\n  countries: <comma-separated list of country codes>\n  error_retry_delay: 0\n  name: Country Stream\n  provider: Shodan\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.shodan.parser\nparameters:\n  error_retry_delay: 0\n  ignore_errors: False\n  minimal_mode: False\n
      "},{"location":"user/feeds/#spamhaus","title":"Spamhaus","text":""},{"location":"user/feeds/#asn-drop","title":"ASN Drop","text":"
      ASN-DROP contains a list of Autonomous System Numbers controlled by spammers or cyber criminals, as well as \"hijacked\" ASNs. ASN-DROP can be used to filter BGP routes which are being used for malicious purposes.
       
      Public: yes
       
      Revision: 2018-01-20
       
      Documentation: https://www.spamhaus.org/drop/
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://www.spamhaus.org/drop/asndrop.txt\n  name: ASN Drop\n  provider: Spamhaus\n  rate_limit: 3600\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.spamhaus.parser_drop\n
      "},{"location":"user/feeds/#cert","title":"CERT","text":"
      Spamhaus CERT Insight Portal. Access limited to CERTs and CSIRTs with national or regional responsibility. .
       
      Public: no
       
      Revision: 2018-01-20
       
      Documentation: https://www.spamhaus.org/news/article/705/spamhaus-launches-cert-insight-portal
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: {{ your CERT portal URL }}\n  name: CERT\n  provider: Spamhaus\n  rate_limit: 3600\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.spamhaus.parser_cert\n
      "},{"location":"user/feeds/#drop","title":"Drop","text":"
      The DROP list will not include any IP address space under the control of any legitimate network - even if being used by \"the spammers from hell\". DROP will only include netblocks allocated directly by an established Regional Internet Registry (RIR) or National Internet Registry (NIR) such as ARIN, RIPE, AFRINIC, APNIC, LACNIC or KRNIC or direct RIR allocations.
       
      Public: yes
       
      Revision: 2018-01-20
       
      Documentation: https://www.spamhaus.org/drop/
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://www.spamhaus.org/drop/drop.txt\n  name: Drop\n  provider: Spamhaus\n  rate_limit: 3600\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.spamhaus.parser_drop\n
      "},{"location":"user/feeds/#dropv6","title":"Dropv6","text":"
      The DROPv6 list includes IPv6 ranges allocated to spammers or cyber criminals. DROPv6 will only include IPv6 netblocks allocated directly by an established Regional Internet Registry (RIR) or National Internet Registry (NIR) such as ARIN, RIPE, AFRINIC, APNIC, LACNIC or KRNIC or direct RIR allocations.
       
      Public: yes
       
      Revision: 2018-01-20
       
      Documentation: https://www.spamhaus.org/drop/
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://www.spamhaus.org/drop/dropv6.txt\n  name: Dropv6\n  provider: Spamhaus\n  rate_limit: 3600\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.spamhaus.parser_drop\n
      "},{"location":"user/feeds/#edrop","title":"EDrop","text":"
      EDROP is an extension of the DROP list that includes sub-allocated netblocks controlled by spammers or cyber criminals. EDROP is meant to be used in addition to the direct allocations on the DROP list.
       
      Public: yes
       
      Revision: 2018-01-20
       
      Documentation: https://www.spamhaus.org/drop/
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://www.spamhaus.org/drop/edrop.txt\n  name: EDrop\n  provider: Spamhaus\n  rate_limit: 3600\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.spamhaus.parser_drop\n
      "},{"location":"user/feeds/#strangereal-intel","title":"Strangereal Intel","text":""},{"location":"user/feeds/#dailyioc","title":"DailyIOC","text":"
      Daily IOC from tweets and articles
       
      Public: yes
       
      Revision: 2019-12-05
       
      Documentation: https://github.com/StrangerealIntel/DailyIOC
       
      Additional Information: collector's extra_fields parameter may be any of fields from the github content API response <https://developer.github.com/v3/repos/contents/>_
       
      Collector configuration
       
      module: intelmq.bots.collectors.github_api.collector_github_contents_api\nparameters:\n  personal_access_token: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token\n  regex: .*.json\n  repository: StrangerealIntel/DailyIOC\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.github_feed\n
      "},{"location":"user/feeds/#surbl","title":"Surbl","text":""},{"location":"user/feeds/#malicious-domains","title":"Malicious Domains","text":"
      Detected malicious domains. Note that you have to opened up Sponsored Datafeed Service (SDS) access to the SURBL data via rsync for your IP address.
       
      Public: no
       
      Revision: 2018-09-04
       
      Collector configuration
       
      module: intelmq.bots.collectors.rsync.collector_rsync\nparameters:\n  file: wild.surbl.org.rbldnsd\n  rsync_path: blacksync.prolocation.net::surbl-wild/\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.surbl.parser\n
      "},{"location":"user/feeds/#team-cymru","title":"Team Cymru","text":""},{"location":"user/feeds/#cap","title":"CAP","text":"
      Team Cymru provides daily lists of compromised or abused devices for the ASNs and/or netblocks with a CSIRT's jurisdiction. This includes such information as bot infected hosts, command and control systems, open resolvers, malware urls, phishing urls, and brute force attacks
       
      Public: no
       
      Revision: 2018-01-20
       
      Documentation: https://www.team-cymru.com/CSIRT-AP.html https://www.cymru.com/$certname/report_info.txt
       
      Additional Information: \"Two feeds types are offered:  * The new https://www.cymru.com/$certname/$certname_{time[%Y%m%d]}.txt  * and the old https://www.cymru.com/$certname/infected_{time[%Y%m%d]}.txt  Both formats are supported by the parser and the new one is recommended.  As of 2019-09-12 the old format will be retired soon.\"
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_password: {{ your password }}\n  http_url: https://www.cymru.com/$certname/$certname_{time[%Y%m%d]}.txt\n  http_url_formatting: True\n  http_username: {{ your username }}\n  name: CAP\n  provider: Team Cymru\n  rate_limit: 86400\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.cymru.parser_cap_program\n
      "},{"location":"user/feeds/#full-bogons-ipv4","title":"Full Bogons IPv4","text":"
      Fullbogons are a larger set which also includes IP space that has been allocated to an RIR, but not assigned by that RIR to an actual ISP or other end-user. IANA maintains a convenient IPv4 summary page listing allocated and reserved netblocks, and each RIR maintains a list of all prefixes that they have assigned to end-users. Our bogon reference pages include additional links and resources to assist those who wish to properly filter bogon prefixes within their networks.
       
      Public: yes
       
      Revision: 2018-01-20
       
      Documentation: https://www.team-cymru.com/bogon-reference-http.html
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://www.team-cymru.org/Services/Bogons/fullbogons-ipv4.txt\n  name: Full Bogons IPv4\n  provider: Team Cymru\n  rate_limit: 86400\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.cymru.parser_full_bogons\n
      "},{"location":"user/feeds/#full-bogons-ipv6","title":"Full Bogons IPv6","text":"
      Fullbogons are a larger set which also includes IP space that has been allocated to an RIR, but not assigned by that RIR to an actual ISP or other end-user. IANA maintains a convenient IPv4 summary page listing allocated and reserved netblocks, and each RIR maintains a list of all prefixes that they have assigned to end-users. Our bogon reference pages include additional links and resources to assist those who wish to properly filter bogon prefixes within their networks.
       
      Public: yes
       
      Revision: 2018-01-20
       
      Documentation: https://www.team-cymru.com/bogon-reference-http.html
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://www.team-cymru.org/Services/Bogons/fullbogons-ipv6.txt\n  name: Full Bogons IPv6\n  provider: Team Cymru\n  rate_limit: 86400\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.cymru.parser_full_bogons\n
      "},{"location":"user/feeds/#threatminer","title":"Threatminer","text":""},{"location":"user/feeds/#recent-domains","title":"Recent domains","text":"
      Latest malicious domains.
       
      Public: yes
       
      Revision: 2018-02-06
       
      Documentation: https://www.threatminer.org/
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://www.threatminer.org/\n  name: Recent domains\n  provider: Threatminer\n  rate_limit: 86400\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.threatminer.parser\n
      "},{"location":"user/feeds/#turris","title":"Turris","text":""},{"location":"user/feeds/#greylist","title":"Greylist","text":"
      The data are processed and classified every week and behaviour of IP addresses that accessed a larger number of Turris routers is evaluated. The result is a list of addresses that have tried to obtain information about services on the router or tried to gain access to them. The list also contains a list of tags for each address which indicate what behaviour of the address was observed.
       
      Public: yes
       
      Revision: 2023-06-13
       
      Documentation: https://project.turris.cz/en/greylist
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://view.sentinel.turris.cz/greylist-data/greylist-latest.csv\n  name: Greylist\n  provider: Turris\n  rate_limit: 43200\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.turris.parser\n
      "},{"location":"user/feeds/#greylist-with-pgp-signature-verification","title":"Greylist with PGP signature verification","text":"
      The data are processed and classified every week and behaviour of IP addresses that accessed a larger number of Turris routers is evaluated. The result is a list of addresses that have tried to obtain information about services on the router or tried to gain access to them. The list also contains a list of tags for each address which indicate what behaviour of the address was observed.
       
      The Turris Greylist feed provides PGP signatures for the provided files. You will need to import the public PGP key from the linked documentation page, currently available at https://pgp.mit.edu/pks/lookup?op=vindex&search=0x10876666 or from below. See the URL Fetcher Collector documentation for more information on PGP signature verification.
       
      PGP Public key:
       
      -----BEGIN PGP PUBLIC KEY BLOCK-----\nVersion: SKS 1.1.6\nComment: Hostname: pgp.mit.edu\n\nmQINBFRl7D8BEADaRFoDa/+r27Gtqrdn8sZL4aSYTU4Q3gDr3TfigK8H26Un/Y79a/DUL1o0\no8SRae3uwVcjJDHZ6KDnxThbqF7URfpuCcCYxOs8p/eu3dSueqEGTODHWF4ChIh2japJDc4t\n3FQHbIh2e3GHotVqJGhvxMmWqBFoZ/mlWvhjs99FFBZ87qbUNk7l1UAGEXeWeECgz9nGox40\n3YpCgEsnJJsKC53y5LD/wBf4z+z0GsLg2GMRejmPRgrkSE/d9VjF/+niifAj2ZVFoINSVjjI\n8wQFc8qLiExdzwLdgc+ggdzk5scY3ugI5IBt1zflxMIOG4BxKj/5IWsnhKMG2NLVGUYOODoG\npKhcY0gCHypw1bmkp2m+BDVyg4KM2fFPgQ554DAX3xdukMCzzZyBxR3UdT4dN7xRVhpph3Y2\nAmh1E/dpde9uwKFk1oRHkRZ3UT1XtpbXtFNY0wCiGXPt6KznJAJcomYFkeLHjJo3nMK0hISV\nGSNetVLfNWlTkeo93E1innbSaDEN70H4jPivjdVjSrLtIGfr2IudUJI84dGmvMxssWuM2qdg\nFSzoTHw9UE9KT3SltKPS+F7u9x3h1J492YaVDncATRjPZUBDhbvo6Pcezhup7XTnI3gbRQc2\noEUDb933nwuobHm3VsUcf9686v6j8TYehsbjk+zdA4BoS/IdCwARAQABtC5UdXJyaXMgR3Jl\neWxpc3QgR2VuZXJhdG9yIDxncmV5bGlzdEB0dXJyaXMuY3o+iQI4BBMBAgAiBQJUZew/AhsD\nBgsJCAcDAgYVCAIJCgsEFgIDAQIeAQIXgAAKCRDAQrU3EIdmZoH4D/9Jo6j9RZxCAPTaQ9WZ\nWOdb1Eqd/206bObEX+xJAago+8vuy+waatHYBM9/+yxh0SIg2g5whd6J7A++7ePpt5XzX6hq\nbzdG8qGtsCRu+CpDJ40UwHep79Ck6O/A9KbZcZW1z/DhbYT3z/ZVWALy4RtgmyC67Vr+j/C7\nKNQ529bs3kP9AzvEIeBC4wdKl8dUSuZIPFbgf565zRNKLtHVgVhiuDPcxKmBEl4/PLYF30a9\n5Tgp8/PNa2qp1DV/EZjcsxvSRIZB3InGBvdKdSzvs4N/wLnKWedj1GGm7tJhSkJa4MLBSOIx\nyamhTS/3A5Cd1qoDhLkp7DGVXSdgEtpoZDC0jR7nTS6pXojcgQaF7SfJ3cjZaLI5rjsx0YLk\nG4PzonQKCAAQG1G9haCDniD8NrrkZ3eFiafoKEECRFETIG0BJHjPdSWcK9jtNCupBYb7JCiz\nQ0hwLh2wrw/wCutQezD8XfsBFFIQC18TsJAVgdHLZnGYkd5dIbV/1scOcm52w6EGIeMBBYlB\nJ2+JNukH5sJDA6zAXNl2I1H1eZsP4+FSNIfB6LdovHVPAjn7qXCw3+IonnQK8+g8YJkbbhKJ\nsPejfg+ndpe5u0zX+GvQCFBFu03muANA0Y/OOeGIQwU93d/akN0P1SRfq+bDXnkRIJQOD6XV\n0ZPKVXlNOjy/z2iN2A==\n=wjkM\n-----END PGP PUBLIC KEY BLOCK-----\n
       
      Public: yes
       
      Revision: 2018-01-20
       
      Documentation: https://project.turris.cz/en/greylist
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://www.turris.cz/greylist-data/greylist-latest.csv\n  name: Greylist\n  provider: Turris\n  rate_limit: 43200\n  signature_url: https://www.turris.cz/greylist-data/greylist-latest.csv.asc\n  verify_pgp_signatures: True\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.turris.parser\n
      "},{"location":"user/feeds/#university-of-toulouse","title":"University of Toulouse","text":""},{"location":"user/feeds/#blacklist","title":"Blacklist","text":"
      Various blacklist feeds
       
      Public: yes
       
      Revision: 2018-01-20
       
      Documentation: https://dsi.ut-capitole.fr/blacklists/
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  extract_files: true\n  http_url: https://dsi.ut-capitole.fr/blacklists/download/{collection name}.tar.gz\n  name: Blacklist\n  provider: University of Toulouse\n  rate_limit: 43200\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.generic.parser_csv\nparameters:\n  columns: {depends on a collection}\n  defaults_fields: {'classification.type': '{depends on a collection}'}\n  delimiter: false\n
      "},{"location":"user/feeds/#vxvault","title":"VXVault","text":""},{"location":"user/feeds/#urls","title":"URLs","text":"
      This feed provides IP addresses hosting Malware.
       
      Public: yes
       
      Revision: 2018-01-20
       
      Documentation: http://vxvault.net/ViriList.php
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: http://vxvault.net/URL_List.php\n  name: URLs\n  provider: VXVault\n  rate_limit: 3600\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.vxvault.parser\n
      "},{"location":"user/feeds/#viriback","title":"ViriBack","text":""},{"location":"user/feeds/#c2-tracker","title":"C2 Tracker","text":"
      Latest detected C2 servers.
       
      Public: yes
       
      Revision: 2022-11-15
       
      Documentation: https://viriback.com/
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://tracker.viriback.com/dump.php\n  name: C2 Tracker\n  provider: ViriBack\n  rate_limit: 86400\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.generic.csv_parser\nparameters:\n  columns: [\"malware.name\", \"source.url\", \"source.ip\", \"time.source\"]\n  defaults_fields: {'classification.type': 'malware-distribution'}\n  skip_header: True\n
      "},{"location":"user/feeds/#zoneh","title":"ZoneH","text":""},{"location":"user/feeds/#defacements","title":"Defacements","text":"
      all the information contained in Zone-H's cybercrime archive were either collected online from public sources or directly notified anonymously to us.
       
      Public: no
       
      Revision: 2018-01-20
       
      Documentation: https://zone-h.org/
       
      Collector configuration
       
      module: intelmq.bots.collectors.mail.collector_mail_attach\nparameters:\n  attach_regex: csv\n  extract_files: False\n  folder: INBOX\n  mail_host: __HOST__\n  mail_password: __PASSWORD__\n  mail_ssl: True\n  mail_user: __USERNAME__\n  name: Defacements\n  provider: ZoneH\n  rate_limit: 3600\n  sent_from: datazh@zone-h.org\n  subject_regex: Report\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.zoneh.parser\n
      "},{"location":"user/feeds/#capture","title":"cAPTure","text":""},{"location":"user/feeds/#ponmocup-domains-cif-format","title":"Ponmocup Domains CIF Format","text":"
      List of ponmocup malware redirection domains and infected web-servers from cAPTure. See also http://security-research.dyndns.org/pub/botnet-links.htm and http://c-apt-ure.blogspot.com/search/label/ponmocup The data in the CIF format is not equal to the Shadowserver CSV format. Reasons are unknown.
       
      Public: yes
       
      Revision: 2018-01-20
       
      Documentation: http://security-research.dyndns.org/pub/malware-feeds/
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: http://security-research.dyndns.org/pub/malware-feeds/ponmocup-infected-domains-CIF-latest.txt\n  name: Infected Domains\n  provider: cAPTure\n  rate_limit: 10800\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.dyn.parser\n
      "},{"location":"user/feeds/#ponmocup-domains-shadowserver-format","title":"Ponmocup Domains Shadowserver Format","text":"
      List of ponmocup malware redirection domains and infected web-servers from cAPTure. See also http://security-research.dyndns.org/pub/botnet-links.htm and http://c-apt-ure.blogspot.com/search/label/ponmocup The data in the Shadowserver CSV is not equal to the CIF format format. Reasons are unknown.
       
      Public: yes
       
      Revision: 2020-07-08
       
      Documentation: http://security-research.dyndns.org/pub/malware-feeds/
       
      Collector configuration
       
      module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: http://security-research.dyndns.org/pub/malware-feeds/ponmocup-infected-domains-shadowserver.csv\n  name: Infected Domains\n  provider: cAPTure\n  rate_limit: 10800\n
       
      Parser configuration
       
      module: intelmq.bots.parsers.generic.parser_csv\nparameters:\n  columns: [\"time.source\", \"source.ip\", \"source.fqdn\", \"source.urlpath\", \"source.port\", \"protocol.application\", \"extra.tag\", \"extra.redirect_target\", \"extra.category\"]\n  compose_fields: {'source.url': 'http://{0}{1}'}\n  defaults_fields: {'classification.type': 'malware-distribution'}\n  delimiter: ,\n  skip_header: True\n
      "},{"location":"user/intro/","title":"Intro","text":""},{"location":"user/intro/#intro","title":"Intro","text":"
      The User Guide provides information on how to use installed IntelMQ and it's components. Let's start with a basic not-so-technical description of how IntelMQ works and the used terminology:
       
         
      • It consists of small (python) programs called bots.
        •  
      • Bots communicate witch each other (using something called message broker) by passing so called events (JSON objects).
        •  
      • An example event can look like this:
        •  
       
      {\n    \"source.geolocation.cc\": \"JO\",\n    \"malware.name\": \"qakbot\",\n    \"source.ip\": \"82.212.115.188\",\n    \"source.asn\": 47887,\n    \"classification.type\": \"c2-server\",\n    \"extra.status\": \"offline\",\n    \"source.port\": 443,\n    \"classification.taxonomy\": \"malicious-code\",\n    \"source.geolocation.latitude\": 31.9522,\n    \"feed.accuracy\": 100,\n    \"extra.last_online\": \"2023-02-16\",\n    \"time.observation\": \"2023-02-16T09:55:12+00:00\",\n    \"source.geolocation.city\": \"amman\",\n    \"source.network\": \"82.212.115.0/24\",\n    \"time.source\": \"2023-02-15T14:19:09+00:00\",\n    \"source.as_name\": \"NEU-AS\",\n    \"source.geolocation.longitude\": 35.939,\n    \"feed.name\": \"abusech-feodo-c2-tracker\"\n  }\n
       
         
      •  
        Bots are divided into following groups:
         
           
        • Collectors - bots that collect data from sources such as website, mailbox, api, etc.
          •  
        • Parsers - bots that split and parse collected data into individual events.
          •  
        • Experts - bots that can do additional processing of events such as enriching, filtering, etc.
          •  
        • Outputs - bots that can output events to files, databases, etc.
          •  
         
        •  
      •  
        Data sources supported by IntelMQ are called feeds.
         
           
        • IntelMQ provides recommended configuration of collector and parser bot combinations for selected feeds.
          •  
         
        •  
      • The collection of all configured bots and their communication paths is called pipeline (or botnet).
        •  
      • Individual bots as well as the complete pipeline can be configured, managed and monitored via:
           
        • Web interface called IntelMQ Manager (best suited for regular users).
          •  
        • Command line tool called intelmqctl (best suited for administrators).
          •  
        • REST API provided by the IntelMQ API extension (best suited for other programs).
          •  
         
        •  
      "},{"location":"user/manager/","title":"Manager","text":""},{"location":"user/manager/#using-intelmq-manager","title":"Using IntelMQ Manager","text":"
      IntelMQ Manager is a graphical interface to manage configurations for IntelMQ. It's goal is to provide an intuitive tool to allow non-programmers to specify the data flow in IntelMQ.
      "},{"location":"user/manager/#configuration-pages","title":"Configuration Pages","text":""},{"location":"user/manager/#pipeline","title":"Pipeline","text":"
      This interface lets you visually configure the whole IntelMQ pipeline and the parameters of every single bot. You will be able to see the pipeline in a graph-like visualisation similar to the following screenshot (click to enlarge):
       
      "},{"location":"user/manager/#named-queues-paths","title":"Named queues / paths","text":"
      With IntelMQ Manager you can set the name of certain paths by double-clicking on the line which connects two bots:
       
       
      The name is then displayed along the edge:
       
      "},{"location":"user/manager/#bots-configuration","title":"Bots Configuration","text":"
      When you add a node or edit one you will be presented with a form with the available parameters for the bot. There you can easily change the parameters as shown in the screenshot:
       
       
      After editing the bot's configuration and pipeline, simply click Save Configuration to automatically write the changes to the correct files. The configurations are now ready to be deployed.
       
      Warning
       
      Without saving the configuration your changes will be lost whenever you reload the web page or move between different tabs within the IntelMQ manager page.
      "},{"location":"user/manager/#botnet-management","title":"Botnet Management","text":"
      When you save a configuration you can go to the Management section to see what bots are running and start/stop the entire botnet, or a single bot.
       
      "},{"location":"user/manager/#botnet-monitoring","title":"Botnet Monitoring","text":"
      You can also monitor the logs of individual bots or see the status of the queues for the entire system or for single bots.
       
      In this next example we can see the number of queued messages for all the queues in the system.
       
       
      The following example we can see the status information of a single bot. Namely, the number of queued messages in the queues that are related to that bot and also the last 20 log lines of that single bot.
       
      "},{"location":"user/manager/#keyboard-shortcuts","title":"Keyboard Shortcuts","text":"
      Any underscored letter denotes access key shortcut. The needed shortcut-keyboard is different per Browser:
       
         
      • Firefox: Ctrl+Alt + Letter
        •  
      • Chrome & Chromium: Alt + Letter
        •  
      "}]}
\ No newline at end of file
+{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Introduction","text":""},{"location":"#introduction","title":"Introduction","text":"
      IntelMQ is a solution for IT security teams (CERTs & CSIRTs, SOCs abuse departments, etc.) for collecting and processing security feeds (such as log files) using a message queuing protocol. It's a community driven initiative called IHAP1 (Incident Handling Automation Project) which was conceptually designed by European CERTs/CSIRTs during several InfoSec events. Its main goal is to give to incident responders an easy way to collect & process threat intelligence thus improving the incident handling processes of CERTs.
       
      IntelMQ is frequently used for:
       
         
      • automated incident handling
        •  
      • situational awareness
        •  
      • automated notifications
        •  
      • as data collector for other tools
        •  
      • and more!
        •  
       
      The design was influenced by AbuseHelper however it was re-written from scratch and aims at:
       
         
      • Reducing the complexity of system administration
        •  
      • Reducing the complexity of writing new bots for new data feeds
        •  
      • Reducing the probability of events lost in all process with persistence functionality (even system crash)
        •  
      • Use and improve the existing Data Harmonization Ontology
        •  
      • Use JSON format for all messages
        •  
      • Provide easy way to store data into databases and log collectors such as PostgreSQL, Elasticsearch and Splunk
        •  
      • Provide easy way to create your own black-lists
        •  
      • Provide easy communication with other systems via HTTP RESTful API
        •  
       
      It follows the following basic meta-guidelines:
       
         
      • Don't break simplicity - KISS
        •  
      • Keep it open source - forever
        •  
      • Strive for perfection while keeping a deadline
        •  
      • Reduce complexity/avoid feature bloat
        •  
      • Embrace unit testing
        •  
      • Code readability: test with inexperienced programmers
        •  
      • Communicate clearly
        •  
      "},{"location":"#contribute","title":"Contribute","text":"
         
      • Subscribe to the IntelMQ Developers mailing list and engage in discussions
        •  
      • Report any errors and suggest improvements via issues
        •  
      • Read the Developer Guide and open a pull request
        •  
       
         
      1.  
        Incident Handling Automation Project, mailing list: ihap@lists.trusted-introducer.org\u00a0\u21a9
         
        2.  
      "},{"location":"changelog/","title":"Changelog","text":""},{"location":"changelog/#changelog","title":"CHANGELOG","text":""},{"location":"changelog/#332-unreleased","title":"3.3.2 (unreleased)","text":""},{"location":"changelog/#configuration","title":"Configuration","text":""},{"location":"changelog/#core","title":"Core","text":"
         
      • Python 3.8 or newer is required (PR#2541 by Sebastian Wagner).
        •  
      "},{"location":"changelog/#development","title":"Development","text":""},{"location":"changelog/#data-format","title":"Data Format","text":""},{"location":"changelog/#bots","title":"Bots","text":""},{"location":"changelog/#collectors","title":"Collectors","text":"
         
      • intelmq.bots.collectors.shadowserver.collector_reports_api.py:
        •  
      • Fixed behaviour if parameter types value is empty string, behave the same way as not set, not like no type.
        •  
      "},{"location":"changelog/#parsers","title":"Parsers","text":"
         
      • intelmq.bots.parsers.shadowserver._config:
        •  
      • fix error message formatting if schema file is absent (PR#2528 by Sebastian Wagner).
        •  
      • intelmq.bots.parsers.shadowserver.parser:
        •  
      • Fix to avoid schema download if not configured #2530.
        •  
      "},{"location":"changelog/#experts","title":"Experts","text":"
         
      • intelmq.bots.experts.securitytxt:
        •  
      • Added new bot (PR#2538 by Frank Westers and Sebastian Wagner)
        •  
      "},{"location":"changelog/#outputs","title":"Outputs","text":"
         
      • intelmq.bots.outputs.cif3.output:
        •  
      • The requirement can only be installed on Python version < 3.12.
        •  
      • Add a check on the Python version and exit if incompatible.
        •  
      • Add a deprecation warning (PR#2544 by Sebastian Wagner)
        •  
      "},{"location":"changelog/#documentation","title":"Documentation","text":""},{"location":"changelog/#packaging","title":"Packaging","text":""},{"location":"changelog/#tests","title":"Tests","text":"
         
      • Install build dependencies for pymssql on Python 3.8 as there are no wheels available for this Python version (PR#2542 by Sebastian Wagner).
        •  
      • Install psql explicitly for workflow support on other platforms such as act (PR#2542 by Sebastian Wagner).
        •  
      • Create intelmq user & group if running privileged to allow dropping privileges (PR#2542 by Sebastian Wagner).
        •  
      • intelmq.tests.lib.test_pipeline.TestAmqp.test_acknowledge: Also skip on Python 3.11 besides on 3.8 when running on CI (PR#2542 by Sebastian Wagner).
        •  
      "},{"location":"changelog/#tools","title":"Tools","text":""},{"location":"changelog/#contrib","title":"Contrib","text":""},{"location":"changelog/#known-issues","title":"Known issues","text":""},{"location":"changelog/#331-2024-09-03","title":"3.3.1 (2024-09-03)","text":""},{"location":"changelog/#core_1","title":"Core","text":"
         
      • intelmq.lib.utils.drop_privileges: When IntelMQ is called as root and dropping the privileges to user intelmq, also set the non-primary groups associated with the intelmq user. Makes the behaviour of running intelmqctl as root closer to the behaviour of sudo -u intelmq ... (PR#2507 by Mikk Margus M\u00f6ll).
        •  
      • intelmq.lib.utils.unzip: Ignore directories themselves when extracting data to prevent the extraction of empty data for a directory entries (PR#2512 by Kamil Mankowski).
        •  
      "},{"location":"changelog/#bots_1","title":"Bots","text":""},{"location":"changelog/#collectors_1","title":"Collectors","text":"
         
      • intelmq.bots.collectors.shadowserver.collector_reports_api.py:
        •  
      • Added support for the types parameter to be either a string or a list (PR#2495 by elsif2).
        •  
      • Refactored to utilize the type field returned by the API to match the requested types instead of a sub-string match on the filename.
        •  
      • Fixed timezone issue for collecting reports (PR#2506 by elsif2).
        •  
      • Fixed behaviour if parameter reports value is empty string, behave the same way as not set, not like no report (PR#2523 by Sebastian Wagner).
        •  
      • intelmq.bots.collectors.shodan.collector_stream (PR#2492 by Mikk Margus M\u00f6ll):
        •  
      • Add alert parameter to Shodan stream collector to allow fetching streams by configured alert ID
        •  
      • intelmq.bots.collectors.mail._lib: Remove deprecated parameter attach_unzip from default parameters (PR#2511 by Sebastian Wagner).
        •  
      "},{"location":"changelog/#parsers_1","title":"Parsers","text":"
         
      • intelmq.bots.parsers.shadowserver._config:
        •  
      • Fetch schema before first run (PR#2482 by elsif2, fixes #2480).
        •  
      • intelmq.bots.parsers.dataplane.parser: Use | as field delimiter, fix parsing of AS names including | (PR#2488 by DigitalTrustCenter).
        •  
      • all parsers: add copy_collector_provided_fields parameter allowing copying additional fields from the report, e.g. extra.file_name.   (PR#2513 by Kamil Mankowski).
        •  
      "},{"location":"changelog/#experts_1","title":"Experts","text":"
         
      • intelmq.bots.experts.sieve.expert:
        •  
      • For :contains, =~ and !~, convert the value to string before matching avoiding an exception. If the value is a dict, convert the value to JSON (PR#2500 by Sebastian Wagner).
        •  
      • Add support for variables in Sieve scripts (PR#2514 by Mikk Margus M\u00f6ll, fixes #2486).
        •  
      • intelmq.bots.experts.filter.expert:
        •  
      • Treat value false for parameter filter_regex as false (PR#2499 by Sebastian Wagner).
        •  
      "},{"location":"changelog/#outputs_1","title":"Outputs","text":"
         
      • intelmq.bots.outputs.misp.output_feed: Handle failures if saved current event wasn't saved or is incorrect (PR by Kamil Mankowski).
        •  
      • intelmq.bots.outputs.smtp_batch.output: Documentation on multiple recipients added (PR#2501 by Edvard Rejthar).
        •  
      "},{"location":"changelog/#documentation_1","title":"Documentation","text":"
         
      • Bots: Clarify some section of Mail collectors and the Generic CSV Parser (PR#2510 by Sebastian Wagner).
        •  
      "},{"location":"changelog/#known-issues_1","title":"Known Issues","text":"
      This is short list of the most important known issues. The full list can be retrieved from GitHub. - intelmq.parsers.html_table may not process invalid URLs in patched Python version due to changes in urllib (#2382). - Breaking changes in 'rt' 3.0 library (#2367). - Type error with SQL output bot's prepare_values returning list instead of tuple (#2255). - intelmq_psql_initdb does not work for SQLite (#2202). - intelmqsetup: should install a default state file (#2175). - Misp Expert - Crash if misp event already exist (#2170). - Spamhaus CERT parser uses wrong field (#2165). - Custom headers ignored in HTTPCollectorBot (#2150). - intelmqctl log: parsing syslog does not work (#2097). - Bash completion scripts depend on old JSON-based configuration files (#2094). - Bots started with IntelMQ-API/Manager stop when the webserver is restarted (#952). - Corrupt dump files when interrupted during writing (#870).
      "},{"location":"changelog/#330-2024-03-01","title":"3.3.0 (2024-03-01)","text":""},{"location":"changelog/#configuration_1","title":"Configuration","text":"
         
      • Add new optional configuration parameters for intelmq.bots.collectors.stomp.collector   and intelmq.bots.outputs.stomp.output (PR#2408 by Jan Kaliszewski):
        •  
      • auth_by_ssl_client_certificate (Boolean, default: true; if false then     ssl_client_certificate and ssl_client_certificate_key will be ignored);
        •  
      • username (STOMP authentication login, default: \"guest\"; to be used only     if auth_by_ssl_client_certificate is false);
        •  
      • password (STOMP authentication passcode, default: \"guest\"; to be used only     if auth_by_ssl_client_certificate is false).
        •  
      • Add the possibility to set the ssl_ca_certificate configuration parameter for   intelmq.bots.collectors.stomp.collector and/or intelmq.bots.outputs.stomp.output   to an empty string - which means that the SSL machinery used for STOMP communication   will attempt to load the system\u2019s default CA certificates (PR#2414 by Jan Kaliszewski).
        •  
      "},{"location":"changelog/#core_2","title":"Core","text":"
         
      • intelmq.lib.message: For invalid message keys, add a hint on the failure to the exception: not allowed by configuration or not matching regular expression (PR#2398 by Sebastian Wagner).
        •  
      • intelmq.lib.exceptions.InvalidKey: Add optional parameter additional_text (PR#2398 by Sebastian Wagner).
        •  
      • Change the way we discover bots to allow easy extending based on the entry point name. (PR#2413 by Kamil Mankowski)
        •  
      • intelmq.lib.mixins: Add a new class, StompMixin (defined in a new submodule: stomp),   which provides certain common STOMP-bot-specific operations, factored out from   intelmq.bots.collectors.stomp.collector and intelmq.bots.outputs.stomp.output   (PR#2408 and PR#2414 by Jan Kaliszewski).
        •  
      • intelmq.lib.upgrades: Replace deprecated instances of url2fqdn experts by the new url expert in runtime configuration (PR#2432 by Sebastian Wagner).
        •  
      • intelmq.lib.bot: Ensure closing log files on reloading (PR#2435 by Kamil Mankowski).
        •  
      • AMQP Pipeline: fix SSL context to pointing to create a client-side connection that verifies the server (PR by Kamil Mankowski).
        •  
      • Only load the config once when starting intelmqctl (which makes IntelMQ API calls take less time) (PR#2444 by DigitalTrustCenter).
        •  
      "},{"location":"changelog/#development_1","title":"Development","text":"
         
      • Makefile: Add codespell and test commands (PR#2425 by Sebastian Wagner).
        •  
      "},{"location":"changelog/#data-format_1","title":"Data Format","text":""},{"location":"changelog/#bots_2","title":"Bots","text":""},{"location":"changelog/#collectors_2","title":"Collectors","text":"
         
      • intelmq.bots.collectors.stomp.collector (PR#2408 and PR#2414 by Jan Kaliszewski):
        •  
      • Drop support for versions of stomp.py older than 4.1.12.
        •  
      • Update the code to support new versions of stomp.py, including the latest (8.1.0);     fixes #2342.
        •  
      • Add support for authentication based on STOMP login and passcode, introducing three     new configuration parameters (see above: Configuration).
        •  
      • Add support for loading the system\u2019s default CA certificates, as an alternative to     specifying the CA certificate(s) file path explicitly (see above: Configuration).
        •  
      • Fix (by carefully targeted monkey patching) certain security problems caused by     SSL-related weaknesses that some versions of stomp.py suffer from.
        •  
      • Fix the reconnection behavior: do not attempt to reconnect after shutdown. Also,     never attempt to reconnect if the version of stomp.py is older than 4.1.21 (it     did not work properly anyway).
        •  
      • Add coercion of the port config parameter to int.
        •  
      • Add implementation of the check hook (verifying, in particular, accessibility     of necessary file(s)).
        •  
      • Remove undocumented and unused attributes of StompCollectorBot instances:     ssl_ca_cert, ssl_cl_cert, ssl_cl_cert_key.
        •  
      • Minor fixes/improvements and some refactoring (see also above: Core...).
        •  
      • intelmq.bots.collectors.amqp: fix SSL context to pointing to create a client-side connection that verifies the server (PR by Kamil Mankowski).
        •  
      • intelmq.bots.collectors.shadowserver.collector_reports_api:
        •  
      • The 'json' option is no longer supported as the 'csv' option provides better performance (PR#2372 by elsif2).
        •  
      • intelmq.bots.collectors.alienvault_otx.collector (PR#2449 by qux-bbb):
        •  
      • Fix modified_pulses_only is always False.
        •  
      "},{"location":"changelog/#parsers_2","title":"Parsers","text":"
         
      • intelmq.bots.parsers.netlab_360.parser: Removed as the feed is discontinued. (#2442 by Filip Pokorn\u00fd)
        •  
      • intelmq.bots.parsers.webinspektor.parser: Removed as the feed is discontinued. (#2442 by Filip Pokorn\u00fd)
        •  
      • intelmq.bots.parsers.sucuri.parser: Removed as the feed is discontinued. (#2442 by Filip Pokorn\u00fd)
        •  
      • intelmq.bots.parsers.shadowserver._config:
        •  
      • Switch to dynamic configuration to decouple report schema changes from IntelMQ releases by regularly downloading them from the Shadowserver server (PR#2372 by elsif2).
        •  
      • intelmq.bots.parsers.cymru: Save current line. (PR by Kamil Mankowski)
        •  
      "},{"location":"changelog/#experts_2","title":"Experts","text":"
         
      • intelmq.bots.experts.jinja (PR#2417 by Mikk Margus M\u00f6ll):
        •  
      • Add optional socket_perms and socket_group parameters to change     file permissions on socket file, if it is in use.
        •  
      • intelmq.bots.experts.ripe (PR#2461 by Mikk Margus M\u00f6ll):
        •  
      • Handle \"No abuse contact found for\" messages for non-ASN resources
        •  
      "},{"location":"changelog/#outputs_2","title":"Outputs","text":"
         
      • intelmq.bots.outputs.stomp.output (PR#2408 and PR#2414 by Jan Kaliszewski):
        •  
      • Drop support for versions of stomp.py older than 4.1.12.
        •  
      • Update the code to support new versions of stomp.py, including the latest (8.1.0).
        •  
      • Add support for authentication based on STOMP login and passcode, introducing three     new configuration parameters (see above: Configuration).
        •  
      • Add support for loading the system\u2019s default CA certificates, as an alternative to     specifying the CA certificate(s) file path explicitly (see above: Configuration).
        •  
      • Fix (by carefully targeted monkey patching) certain security problems caused by     SSL-related weaknesses that some versions of stomp.py suffer from.
        •  
      • Fix AttributeError caused by attempts to get unset attributes of StompOutputBot     (ssl_ca_cert et consortes).
        •  
      • Add coercion of the port config parameter to int.
        •  
      • Add implementation of the check hook (verifying, in particular, accessibility     of necessary file(s)).
        •  
      • Add stomp.py version check (raise MissingDependencyError if not >=4.1.12).
        •  
      • Minor fixes/improvements and some refactoring (see also above: Core...).
        •  
      • intelmq.bots.outputs.stomp.output (PR#2423 by Kamil Mankowski):
        •  
      • Try to reconnect on NotConnectedException.
        •  
      • intelmq.bots.outputs.smtp_batch.output (PR #2439 by Edvard Rejthar):
        •  
      • Fix ability to send with the default bcc
        •  
      • intelmq.bots.outputs.amqp: fix SSL context to pointing to create a client-side connection that verifies the server (PR by Kamil Mankowski).
        •  
      "},{"location":"changelog/#documentation_2","title":"Documentation","text":"
         
      • Add a readthedocs configuration file to fix the build fail (PR#2403 by Sebastian Wagner).
        •  
      • Add a guide of developing extensions packages (PR#2413 by Kamil Mankowski)
        •  
      • Update/fix/improve the stuff related to the STOMP bots and integration with the n6's   Stream API (PR#2408 and PR#2414 by Jan Kaliszewski).
        •  
      • Complete documentation overhaul. Change to markdown format. Uses the mkdocs-material (PR#2419 by Filip Pokorn\u00fd).
        •  
      • Adds warning banner if not browsing the latest version of the docs (PR#2445 by Filip Pokorn\u00fd).
        •  
      • Fix logo path in index.md when building the docs (PR#2445 by Filip Pokorn\u00fd).
        •  
      "},{"location":"changelog/#packaging_1","title":"Packaging","text":"
         
      • Add pendulum to suggested packages, as it is required for the sieve bot (PR#2424 by Sebastian Wagner).
        •  
      • debian/control: in Suggests field, replace python3-stomp.py (>= 4.1.9) with   python3-stomp (>= 4.1.12), i.e., fix the package name by removing the .py   suffix and bump the minimum version to 4.1.12 (PR#2414 by Jan Kaliszewski).
        •  
      "},{"location":"changelog/#tests_1","title":"Tests","text":""},{"location":"changelog/#tools_1","title":"Tools","text":"
         
      • intelmq_psql_initdb:
        •  
      • got support for providing custom harmonization file, generating view for storing raw fields separately, and adding IF NOT EXISTS/OR REPLACE clauses (PR#2404 by Kamil Mankowski).
        •  
      • got support for generating JSONB fields for PostgreSQL schema (PR#2436 by Kamil Mankowski).
        •  
      "},{"location":"changelog/#321-2023-08-28","title":"3.2.1 (2023-08-28)","text":""},{"location":"changelog/#core_3","title":"Core","text":"
         
      • Fixed issue preventing bots from stopping after reloading (PR by Kamil Mankowski).
        •  
      "},{"location":"changelog/#bots_3","title":"Bots","text":""},{"location":"changelog/#experts_3","title":"Experts","text":"
         
      • intelmq.bots.experts.reverse_dns.expert:
        •  
      • Fix the cache key to not cache results for /24 (IPv4) and /128 (IPv6) networks but for single IP-Adresses (PR#2395 by Sebastian Wagner, fixes #2394).
        •  
      "},{"location":"changelog/#320-2023-07-18","title":"3.2.0 (2023-07-18)","text":""},{"location":"changelog/#core_4","title":"Core","text":"
         
      • intelmq.lib.utils:
        •  
      • resolve_dns: Deprecate dnspython versions pre-2.0.0 and disable search domains (PR#2352)
        •  
      • Fixed not resetting destination path statistics in the stats cache after restarting bot (Fixes #2331)
        •  
      • Force flushing statistics if bot will sleep longer than flushing delay (Fixes #2336)
        •  
      • intelmq.lib.upgrages: Fix a bug in the upgrade function for version 3.1.0 which caused an exception if a generic csv parser instance had no parameter type (PR#2319 by Filip Pokorn\u00fd).
        •  
      • intelmq.lib.datatypes: Adds TimeFormat class to be used for the time_format bot parameter (PR#2329 by Filip Pokorn\u00fd).
        •  
      • intelmq.lib.exceptions: Fixes a bug in InvalidArgument exception (PR#2329 by Filip Pokorn\u00fd).
        •  
      • intelmq.lib.harmonization:
        •  
      • Changes signature and names of DateTime conversion functions for consistency, backwards compatible (PR#2329 by Filip Pokorn\u00fd).
        •  
      • Ensure rejecting URLs with leading whitespaces after changes in CPython (fixes #2377)
        •  
      • intelmq.lib.bot.Bot: Allow setting the parameters via parameter on bot initialization.
        •  
      "},{"location":"changelog/#development_2","title":"Development","text":"
         
      • CI: pin the Codespell version to omit troubles caused by its new releases (PR #2379).
        •  
      • CI: Updated the versions of the github actions in the CI workflows. (PR#2392 by Sebastian Kufner)
        •  
      "},{"location":"changelog/#bots_4","title":"Bots","text":""},{"location":"changelog/#collectors_3","title":"Collectors","text":"
         
      • intelmq.bots.collector.rt:
        •  
      • restrict python-rt to be below version 3.0 due to introduced breaking changes,
        •  
      • added support for Subject NOT LIKE queries,
        •  
      • added support for multiple values in ticket subject queries.
        •  
      • intelmq.bots.collectors.rsync: Support for optional private key, relative time parsing for the source path, extra rsync parameters and strict host key checking (PR#2241 by Mateo Durante).
        •  
      "},{"location":"changelog/#parsers_3","title":"Parsers","text":"
         
      • intelmq.bots.parsers.shadowserver._config:
        •  
      • Reset detected feedname at shutdown to re-detect the feedname on reloads (PR#2361 by @elsif2, fixes #2360).
        •  
      • Switch to dynamic configuration to decouple report schema changes from IntelMQ releases.
        •  
      • Added 'IPv6-Vulnerable-Exchange' alias and 'Accessible-WS-Discovery-Service' report. (PR#2338)
        •  
      • Removed unused p0f_genre and p0f_detail from the 'DNS-Open-Resolvers' report. (PR#2338)
        •  
      • Added 'Accessible-SIP' report. (PR#2348)
        •  
      • Added 'IPv6-Open-HTTP-Proxy' and 'IPv6-Accessible-HTTP-Proxy' aliases. (PR#2348)
        •  
      • Removed  duplicate mappings from the 'Spam-URL' report. (PR#2348)
        •  
      • intelmq.bots.parsers.generic.parser_csv: Changes time_format parameter to use new TimeFormat class (PR#2329 by Filip Pokorn\u00fd).
        •  
      • intelmq.bots.parsers.html_table.parser: Changes time_format parameter to use new TimeFormat class (PR#2329 by Filip Pokorn\u00fd).
        •  
      • intelmq.bots.parsers.turris.parser.py Updated to the latest data format (issue #2167). (PR#2373 by Filip Pokorn\u00fd).
        •  
      "},{"location":"changelog/#experts_4","title":"Experts","text":"
         
      • intelmq.bots.experts.sieve:
        •  
      • Allow empty lists in sieve rule files (PR#2341 by Mikk Margus M\u00f6ll).
        •  
      • intelmq.bots.experts.cymru_whois:
        •  
      • Ignore AS names with unexpected unicode characters (PR#2352, fixes #2132)
        •  
      • Avoid extraneous search domain-based queries on NXDOMAIN result (PR#2352)
        •  
      • intelmq.bots.experts.sieve:
        •  
      • Added :before and :after keywords (PR#2374)
        •  
      "},{"location":"changelog/#outputs_3","title":"Outputs","text":"
         
      • intelmq.bots.outputs.cif3.output: Added (PR#2244 by Michael Davis).
        •  
      • intelmq.bots.outputs.sql.output: New parameter fail_on_errors (PR#2362 by Sebastian Wagner).
        •  
      • intelmq.bots.outputs.smtp_batch.output: Added a bot to gathering the events and sending them by e-mails at a stroke as CSV files (PR#2253 by Edvard Rejthar)
        •  
      "},{"location":"changelog/#documentation_3","title":"Documentation","text":"
         
      • API: update API installation to be aligned with the rewritten API, and clarify some missing steps.
        •  
      "},{"location":"changelog/#tests_2","title":"Tests","text":"
         
      • New decorator skip_installation and environment variable INTELMQ_TEST_INSTALLATION to skip tests requiring an IntelMQ installation on the test host by default (PR#2370 by Sebastian Wagner, fixes #2369)
        •  
      "},{"location":"changelog/#tools_2","title":"Tools","text":"
         
      • intelmqsetup:
        •  
      • SECURITY: fixed a low-risk bug causing the tool to change owner of / if run with the INTELMQ_PATHS_NO_OPT environment variable set. This affects only the PIP package as the DEB/RPM packages don't contain this tool. (PR#2355 by Kamil Ma\u0144kowski, fixes #2354)
        •  
      • contrib.eventdb.separate-raws-table.sql: Added the missing commas to complete the sql syntax. (PR#2386, fixes #2125 by Sebastian Kufner)
        •  
      • intelmq_psql_initdb:
        •  
      • Added parameter -o to set the output file destination. (by Sebastian Kufner)
        •  
      • intelmqctl:
        •  
      • Increased the performance through removing unnecessary reads. (by Sebastian Kufner)
        •  
      "},{"location":"changelog/#known-issues_2","title":"Known Issues","text":"
      This is short list of the most important known issues. The full list can be retrieved from GitHub. - intelmq.parsers.html_table may not process invalid URLs in patched Python version due to changes in urllib (#2382). - Breaking changes in 'rt' library (#2367). - Stomp collector failed (#2342). - Type error with SQL output bot's prepare_values returning list instead of tuple (#2255). - intelmq_psql_initdb does not work for SQLite (#2202). - intelmqsetup: should install a default state file (#2175). - Misp Expert - Crash if misp event already exist (#2170). - Turris greylist has been updated (#2167). - Spamhaus CERT parser uses wrong field (#2165). - Custom headers ignored in HTTPCollectorBot (#2150). - intelmqctl log: parsing syslog does not work (#2097). - Bash completion scripts depend on old JSON-based configuration files (#2094). - Bot configuration examples use JSON instead of YAML (#2066). - Bots started with IntelMQ-API/Manager stop when the webserver is restarted (#952). - Corrupt dump files when interrupted during writing (#870).
      "},{"location":"changelog/#310-2023-02-10","title":"3.1.0 (2023-02-10)","text":"
         
      • Upgraded syntax to Python 3.6 (mostly Format-Strings) using pyuprade (PR#2136 by Sebastian Wagner).
        •  
      "},{"location":"changelog/#core_5","title":"Core","text":"
         
      • intelmq.lib.upgrades:
        •  
      • Refactor upgrade functions global configuration handling removing the old-style defaults configuration (PR#2058 by Sebastian Wagner).
        •  
      • Pass version history as parameter to upgrade functions (PR#2058 by Sebastian Wagner).
        •  
      • intelmq.lib.message:
        •  
      • Fix and pre-compile the regular expression for harmonization key names and also check keys in the extra. namespace (PR#2059 by Sebastian Wagner, fixes #1807).
        •  
      • intelmq.lib.bot.SQLBot was replaced by an SQLMixin in intelmq.lib.mixins.SQLMixin. The Generic DB Lookup Expert bot and the SQLOutput bot were updated accordingly.
        •  
      • Added support for MSSQL (PR#2171 by Karl-Johan Karlsson).
        •  
      • Added optional reconnect delay parameter (PR#2171 by Karl-Johan Karlsson).
        •  
      • Added an ExpertBot class - it should be used by all expert bots as a parent class
        •  
      • Introduced a module for IntelMQ related datatypes intelmq.lib.datatypes which for now only contains an Enum listing the four bot types
        •  
      • Added a bottype attribute to CollectorBot, ParserBot, ExpertBot, OutputBot
        •  
      • Introduces a module for IntelMQ processmanagers. The processmanagers were up until now part of the intelmqct script.   They now reside in intelmq.lib.processmanager which also contains an interface definition the processmanager implementations must adhere to.   Both the processmanagers and the intelmqctl script were cleaned up a bit.   The LogLevel and ReturnType Enums were added to intelmq.lib.datatypes.
        •  
      • intelmq.lib.bot:
        •  
      • Enhance behaviour if an unconfigured bot is started (PR#2054 by Sebastian Wagner).
        •  
      • Fix line recovery and message dumping of the ParserBot (PR#2192 by Sebastian Wagner).
           
        • Previously the dumped message was always the last message of a report if the report contained multiple lines leading to data-loss.
          •  
         
        •  
      • Fix crashing at start in multithreaded bots (PR#2236 by DigitalTrustCenter).
        •  
      • Added default_fields parameter to ParserBot (PR#2293 by Filip Pokorn\u00fd)
        •  
      • intelmq.lib.pipeline:
        •  
      • Changed BRPOPLPUSH to BLMOVE, because BRPOPLPUSH has been marked as deprecated by redis in favor of BLMOVE (PR#2149 and PR#2240 by Sebastian Waldbauer and Sebastian Wagner, fixes #1827, #2233).
        •  
      • intelmq.lib.utils:
        •  
      • Added wrapper resolve_dns for querying DNS, with the support for recommended methods from dnspython package in versions 1 and 2.
        •  
      • Moved line filtering inside RewindableFileHandle for easier handling and limiting number of temporary objects.
        •  
      • intelmq.lib.harmonization:
        •  
      • Fixed DateTime handling of naive time strings (previously assumed local timezone, now assumes UTC) (PR#2279 by Filip Pokorn\u00fd, fixes #2278)
        •  
      • Removes tzone argument from DateTime.from_timestamp and DateTime.from_epoch_millis
        •  
      • DateTime.from_timstamp now also allows string argument
        •  
      • Removes pytz global dependency
        •  
      • Removed support for Python 3.6, including removing conditional dependencies and updating syntax to use features from newest versions. (fixes #2272)
        •  
      "},{"location":"changelog/#development_3","title":"Development","text":"
         
      • Removed Python 3.6 from CI.
        •  
      • Enabled tests against Python 3.11.
        •  
      "},{"location":"changelog/#bots_5","title":"Bots","text":"
         
      • Set the parent class of all bots to the correct bot class
        •  
      "},{"location":"changelog/#collectors_4","title":"Collectors","text":"
         
      • intelmq.bots.collectors.mail._lib:
        •  
      • Add support for unverified SSL/STARTTLS connections (PR#2055 by Sebastian Wagner).
        •  
      • Fix exception handling for aborted IMAP connections (PR#2187 by Sebastian Wagner).
        •  
      • intelmq.bots.collectors.blueliv: Fix Blueliv collector requirements (PR#2161 by Gethvi).
        •  
      • intelmq.bots.collectors.github_api._collector_github_api: Added personal access token support (PR#2145 by Sebastian Waldbauer, fixes #1549).
        •  
      • intelmq.bots.collectors.file.collector_file: Added file lock support, no more race conditions (PR#2147 by Sebastian Waldbauer, fixes #2128)
        •  
      • intelmq.bots.collectors.shadowserver.collector_reports_api.py: Added file_format option to download reports in CSV format for better performance (PR#2246 by elsif2)
        •  
      "},{"location":"changelog/#parsers_4","title":"Parsers","text":"
         
      • intelmq.bots.parsers.alienvault.parser_otx: Save CVE data in extra.cve instead of extra.CVE due to the field name restriction on lower-case characters  (PR#2059 by Sebastian Wagner).
        •  
      • intelmq.bots.parsers.anubisnetworks.parser: Changed field name format from extra.communication.http.x_forwarded_for_#1 to extra.communication.http.x_forwarded_for_1 due to the field name restriction on alphanumeric characters (PR#2059 by Sebastian Wagner).
        •  
      • intelmq.bots.parsers.dataplane.parser:
        •  
      • Add support for additional feeds (PR#2102 by Mikk Margus M\u00f6ll).
           
        • DNS Recursion Desired
          •  
        • DNS Recursion Desired ANY
          •  
        • DNS Version
          •  
        • Protocol 41
          •  
        • SMTP Greet
          •  
        • SMTP Data
          •  
        • Telnet Login
          •  
        • VNC/RFB Login
          •  
         
        •  
      • Fix event object creation (PR#2298 by DigitalTrustCenter).
        •  
      • Removed intelmq.bots.parsers.malc0de: this bot was marked as deprecated and removed from feed due to offline status (PR#2184 by Tamas Gutsohn, fixes #2178).
        •  
      • intelmq.bots.parsers.microsoft.parser_ctip:
        •  
      • New parameter overwrite (PR#2112 by Sebastian Wagner, fixes #2022).
        •  
      • Fix handling of field Payload.domain if it contains the same IP address as Payload.serverIp (PR#2144 by Mikk Margus M\u00f6ll and Sebastian Wagner).
        •  
      • Handle Payload field with non-base64-encoded JSON content and numbered dictionaries (PR#2193 by Sebastian Wagner)
        •  
      • intelmq.bots.parsers.shodan.parser (PR#2117 by Mikk Margus M\u00f6ll):
        •  
      • Instead of keeping track of extra.ftp.<something>.parameters, FTP parameters are collected together into extra.ftp.features as a list of said features, reducing field count.
        •  
      • Shodan field rsync.modules is collected.
        •  
      • Conversion functions can raise NoValueException with a string argument to signify that the conversion would not succeed, such as in the case of a single IP address being given in hostnames, which would then be passed into source.reverse_dns and fail to validate as a FQDN.
        •  
      • Variable _common_keys is moved out of the class.
        •  
      • _dict_dict_to_obj_list is introduced, for converting a string-to-dict mapping into a list of dicts with the previous key as an attribute of the dict; this can be useful for preventing issues where, when feeding the data into aggregating tools, you'd end up with many more fields than necessary, e.g vulns.CVE-2010-0001.cvss, CVE-2010-0002.cvss etc.
        •  
      • _get_first to get the first item from a list, with NoValueException raised on empty lists.
        •  
      • _get_first_hostname to handle the first valid FQDN from a list of hostnames for hostnames in the Shodan banner, if there is one, and gives NoValueException otherwise.
        •  
      • ssl.cert.serial and ssl.dhparams.generator, which may return both integers and strings, are converted to strings.
        •  
      • Changes to method apply_mapping, such as reducing needless loop iterations, removing a big try-except, and adding the NoValueException handling described above.
        •  
      • Stops falsy values (False, 0) besides None from being filtered out.
        •  
      • intelmq.bots.parsers.shadowserver._config:
        •  
      • Added support for Accessible AMQP, Device Identification Report (IPv4 and IPv6) (PR#2134 by Mateo Durante).
        •  
      • Added file name mapping for SSL-POODLE-Vulnerable-Servers IPv6 (file name scan6_ssl_poodle) (PR#2134 by Mateo Durante).
        •  
      • Added Malware-URL, Sandbox-Connection, Sandbox-DNS, Accessible-AMQP, Open-AnonymouIs-MQTT, Accessible-QUIC, Accessible-SSH, SYNful-Knock, and Special (PR#2227 by elsif2)
        •  
      • Removed legacy reports Amplification-DDoS-Victim, CAIDA-IP-Spoofer, Darknet, Drone, Drone-Brute-Force, IPv6-Sinkhole-HTTP-Drone, Microsoft-Sinkhole, and Sinkhole-HTTP-Drone (PR#2227 by elsif2).
        •  
      • Users storing events in a database should be aware that field names and types have been updated (PR#2227 by elsif2).
        •  
      • Corrected \"Accessible-AMQP\" message_length type (int) and added \"STUN\" support (PR#2235 by elsif2).
        •  
      • Added amplification factor to UDP scan reports (PR#2238 by elsif2).
        •  
      • Added version and build_date to \"Vulnerable-HTTP\" report (PR#2238 by elsif2).
        •  
      • The following field types have been standardized across all Shadowserver reports (PR#2246 by elsif2):       destination.fqdn (validate_fqdn)       destination.url (convert_http_host_and_url)       extra.browser_trusted (convert_bool)       extra.duration (convert_int)       extra.end_time (convert_date_utc)       extra.freak_vulnerable (convert_bool)       extra.ok (convert_bool)       extra.password (validate_to_none)       extra.ssl_poodle (convert_bool)       extra.status (convert_int)       extra.uptime (convert_int)       extra.version (convert_to_none)       source.network (validate_network)
        •  
      • The following report field names have changed to better represent their values:       scan_rsync:extra.password renamed to extra.has_password       scan_elasticsearch:status renamed to http_code
        •  
      • Added Accessible-HTTP-proxy and Open-HTTP-proxy (PR#2246 by elsif2).
        •  
      • Added http_agent to the Honeypot-DDoS report and added the DDoS-Participant report (PR#2303 by elsif2)
        •  
      • Added Accessible-SLP, IPv6 Accesssible-SLP, IPv6-DNS-Open-Resolvers, and IPv6-Open-LDAP-TCP reports (PR#2311 by elsif2)
        •  
      •  
        Standardized response_length to response_size in Accessible-ICS and Open-MSSQL (PR#2311 by elsif2)
         
        •  
      •  
        intelmq.bots.parsers.cymru.parser_cap_program: The parser mapped the hostname into source.fqdn which is not allowed by the IntelMQ Data Format. Added a check (PR#2215 by Sebastian Waldbauer, fixes #2169)
         
        •  
      • intelmq.bots.parsers.generic.parser_csv:
        •  
      • Use RewindableFileHandle to use the original current line for line recovery (PR#2192 by Sebastian Wagner).
        •  
      • Recovering CSV lines preserves the original line ending (PR#2280 by Kamil Mankowski, fixes #1597)
        •  
      • intelmq.bots.parsers.autoshun.parser: Removed, as the feed is discontinued (PR#2214 by Sebastian Waldbauer, fixes #2162).
        •  
      • intelmq.bots.parsers.openphish.parser_commercial: Refactored complete code (PR#2160 by Filip Pokorn\u00fd).
        •  
      • Fixes wrong mapping of host field to source.fqdn when the content was an IP address.
        •  
      • Adds newly added fields in the feed.
        •  
      • intelmq.bots.parsers.phishtank.parser: Refactored code (PR#2270 by Filip Pokorn\u00fd)
        •  
      • Changes feed URL to JSON format (contains more information). The URL needs to by manually updated in the configuration!
        •  
      • Adds fields from the JSON feed.
        •  
      • intelmq.bots.parsers.dshield.parser_domain: Has been removed, due to the feed is discontinued. (PR#2276 by Sebastian Waldbauer)
        •  
      • intelmq.bots.parsers.abusech.parser_ip: Removed (PR#2268 by Filip Pokorn\u00fd).
        •  
      • intelmq.bots.parsers.abusech.parser_domain: Removed (PR#2268 by Filip Pokorn\u00fd).
        •  
      • intelmq.bots.parsers.abusech.parser_feodotracker: Added new parser bot (PR#2268 by Filip Pokorn\u00fd)
        •  
      • Changes feed URL to JSON format (contains more information).
        •  
      • Adds fields from the JSON feed.
        •  
      • intelmq.bots.parsers.generic.parser_csv: Parameter type is deprecated, default_fields should be used. (PR#2293 by Filip Pokorn\u00fd)
        •  
      • intelmq.bots.parsers.generic.parser_csv: Parameter skip_header now allows also integer as a fixed number of lines to skip. (PR#2313 by Filip Pokorn\u00fd)
        •  
      • intelmq.bots.parsers.taichung.parser: Removed (PR#2266 by Filip Pokorn\u00fd)
        •  
      "},{"location":"changelog/#experts_5","title":"Experts","text":"
         
      • intelmq.bots.experts.domain_valid: New bot for checking domain's validity (PR#1966 by Marius Karotkis).
        •  
      • intelmq.bots.experts.truncate_by_delimiter.expert: Cut string if its length is higher than a maximum length (PR#1967 by Marius Karotkis).
        •  
      • intelmq.bots.experts.remove_affix: Remove prefix or postfix strings from a field (PR#1965 by Marius Karotkis).
        •  
      • intelmq.bots.experts.asn_lookup.expert: Fixes update-database script on the last few days of a month (PR#2121 by Filip Pokorn\u00fd, fixes #2088).
        •  
      • intelmq.bots.experts.threshold.expert: Correctly use the standard parameter redis_cache_ttl instead of the previously used parameter timeout (PR#2155 by Karl-Johan Karlsson).
        •  
      • intelmq.bots.experts.jinja2.expert: Lift restriction on requirement jinja2 < 3 (PR#2158 by Sebastian Wagner).
        •  
      • intelmq.bots.experts.asn_lookup.expert, intelmq.bots.experts.domain_suffix.expert, intelmq.bots.experts.maxmind_geoip.expert, intelmq.bots.experts.recordedfuture_iprisk.expert, intelmq.bots.experts.tor_nodes.expert: New parameter autoupdate_cached_database to disable automatic updates (downloads) of cached databases (PR#2180 by Sebastian Wagner).
        •  
      • intelmq.bots.experts.url.expert: New bot for extracting additional information from source.url and/or destination.url (PR#2315 by Filip Pokorn\u00fd).
        •  
      "},{"location":"changelog/#outputs_4","title":"Outputs","text":"
         
      • Removed intelmq.bots.outputs.postgresql: this bot was marked as deprecated in 2019 announced to be removed in version 3 of IntelMQ (PR#2045 by Birger Schacht).
        •  
      • Added intelmq.bots.outputs.rpz_file.output to create RPZ files (PR#1962 by Marius Karotkis).
        •  
      • Added intelmq.bots.outputs.bro_file.output to create Bro intel formatted files (PR#1963 by Marius Karotkis).
        •  
      • intelmq.bots.outputs.templated_smtp.output:
        •  
      • Add new function from_json() (which just calls json.loads() in the standard Python environment), meaning the Templated SMTP output bot can take strings containing JSON documents and do the formatting itself (PR#2120 by Karl-Johan Karlsson).
        •  
      • Lift restriction on requirement jinja2 < 3 (PR#2158 by Sebastian Wagner).
        •  
      • intelmq.bots.outputs.sql:
        •  
      • For PostgreSQL, escape Nullbytes in text to prevent \"unsupported Unicode escape sequence\" issues (PR#2223 by Sebastian Wagner, fixes #2203).
        •  
      "},{"location":"changelog/#documentation_4","title":"Documentation","text":"
         
      • Feeds: Add documentation for newly supported dataplane feeds, see above (PR#2102 by Mikk Margus M\u00f6ll).
        •  
      • Installation: Restructured the whole document to make it clearer and straight-forward (PR#2113 by Sebastian Wagner).
        •  
      • Add workaround for https://github.com/sphinx-doc/sphinx/issues/10701 (PR#2225 by Sebastian Wagner, kudos @yarikoptic, fixes #2224).
        •  
      • Fix wrong operator for list-contains-value operation in sieve expert documentation (PR#2256 by Filip Pokorn\u00fd).
        •  
      • Added documentation on default_fields parameter (PR#2293 by Filip Pokorn\u00fd).
        •  
      • Updated documentation on skip_header parameter (PR#2313 by Filip Pokorn\u00fd).
        •  
      • Viriback Unsafe Sites feed replaced with Viriback C2 Tracker. (PR#2266 by Filip Pokorn\u00fd)
        •  
      • Netlab 360 Mirai Scanner feed removed as it is discontinued. (PR#2266 by Filip Pokorn\u00fd)
        •  
      • Benkow Malware Panels Tracker feed changed parser configuration. (PR#2266 by Filip Pokorn\u00fd)
        •  
      • Taichung feed removed as it is discontinued. (PR#2266 by Filip Pokorn\u00fd)
        •  
      • Added new URL Expert bot. (PR#2315 by Filip Pokorn\u00fd)
        •  
      "},{"location":"changelog/#packaging_2","title":"Packaging","text":"
         
      • Remove deleted intelmq.bots.experts.sieve.validator from executables in setup.py (PR#2256 by Filip Pokorn\u00fd).
        •  
      • Run the geoip database cron-job twice a week (PR#2285 by Filip Pokorn\u00fd).
        •  
      "},{"location":"changelog/#tests_3","title":"Tests","text":"
         
      • Add GitHub Action to run regexploit on all Python, JSON and YAML files (PR#2059 by Sebastian Wagner).
        •  
      • intelmq.lib.test:
        •  
      • Decorator skip_ci also detects dpkg-buildpackage environments by checking the environment variable DEB_BUILD_ARCH (PR#2123 by Sebastian Wagner).
        •  
      • Fixing regex to catchall after python version and process ID, add tests for it (PR#2216 by Sebastian Waldbauer and Sebastian Wagner, fixes #2185)
        •  
      • Also test on Python 3.10 (PR#2140 by Sebastian Wagner).
        •  
      • Switch from nosetests to pytest, as the former does not support Python 3.10 (PR#2140 by Sebastian Wagner).
        •  
      • CodeQL Github Actions exponential backtracking on strings fixed. (PR#2148 by Sebastian Waldbauer, fixes #2138)
        •  
      • Reverse DNS expert tests: remove outdated failing test test_invalid_ptr (PR#2208 by Sebastian Wagner, fixes #2206).
        •  
      • Add test dependency requests_mock to the development extra requirements in setup.py (PR#2210 by Sebastian Wagner).
        •  
      • Threshold Expert tests: Use environment variable INTELMQ_PIPELINE_HOST as redis host, analogous to other tests (PR#2209 by Sebastian Wagner, fixes #2207).
        •  
      • Remove codecov action as it failed regularly (PR#2237 by Sebastian Wagner, fixes #2229).
        •  
      • intelmq.lib.test.BotTestCase: Adds skip_checks variable to not fail on non-empty messages from calling check function (PR#2315 by Filip Pokorn\u00fd).
        •  
      "},{"location":"changelog/#tools_3","title":"Tools","text":"
         
      • intelmqctl:
        •  
      • fix process manager initialization if run non-interactively, as intelmqdump does it (PR#2189 by Sebastian Wagner, fixes 2188).
        •  
      • check: handle SyntaxError in bot modules and report it without breaking execution (fixes #2177)
        •  
      • Privilege drop before logfile creation (PR#2277 by Sebastian Waldbauer, fixes 2176)
        •  
      • intelmqsetup: Revised installation of manager by building the static files at setup, not build time, making it behave more meaningful. Requires intelmq-manager >= 3.1.0 (PR#2198 by Sebastian Wagner, fixes #2197).
        •  
      • intelmqdump: Respected global and per-bot custom settings of logging_path (fix #1605).
        •  
      "},{"location":"changelog/#contrib_1","title":"Contrib","text":"
         
      • logrotate: Move compress and ownership rules to the IntelMQ-blocks to prevent that they apply to other files (PR#2111 by Sebastian Wagner, fixes #2110).
        •  
      "},{"location":"changelog/#known-issues_3","title":"Known issues","text":"
      This is short list of the most important known issues. The full list can be retrieved from GitHub. - intelmq_psql_initdb does not work for SQLite (#2202). - intelmqsetup: should install a default state file (#2175). - Misp Expert - Crash if misp event already exist (#2170). - Turris greylist has been updated (#2167). - Spamhaus CERT parser uses wrong field (#2165). - Custom headers ignored in HTTPCollectorBot (#2150). - Missing commas in SQL query for separate Events table (#2125). - intelmqctl log: parsing syslog does not work (#2097). - Bash completion scripts depend on old JSON-based configuration files (#2094). - Bot configuration examples use JSON instead of YAML (#2066). - Bots started with IntelMQ-API/Manager stop when the webserver is restarted (#952). - Corrupt dump files when interrupted during writing (#870).
      "},{"location":"changelog/#302-2021-09-10","title":"3.0.2 (2021-09-10)","text":""},{"location":"changelog/#core_6","title":"Core","text":"
         
      • intelmq.lib.bot.CollectorBot: Fixed an issue with within the new_report function, which re-loads the harmonization file after a new incoming dataset, which leads to CPU drain and decreased performance (PR#2106 by Sebastian Waldbauer, fixes #2098).
        •  
      • intelmq.lib.bot.Bot: Make private members __is_multithreadable and __collector_empty_process protected members _is_multithreadable and _collector_empty_process to make them easily modifiable by Bot classes (PR#2109 by Sebastian Wagner, fixes #2108).   Also affected and adapted bots by this change are:
        •  
      • intelmq.bots.collectors.api.collector_api
        •  
      • intelmq.bots.collectors.stomp.collector
        •  
      • intelmq.bots.experts.splunk_saved_search.expert
        •  
      • intelmq.bots.experts.threshold.expert
        •  
      • intelmq.bots.outputs.file.output
        •  
      • intelmq.bots.outputs.misp.output_api
        •  
      • intelmq.bots.outputs.misp.output_feed
        •  
      • intelmq.bots.outputs.tcp.output
        •  
      • intelmq.bots.outputs.udp.output
        •  
      • intelmq.lib.cache: Do not create the Cache class if the host is null, allows deactivating the bot statistics (PR#2104 by Sebastian Waldbauer, fixes #2103).
        •  
      "},{"location":"changelog/#bots_6","title":"Bots","text":""},{"location":"changelog/#experts_6","title":"Experts","text":"
         
      • intelmq.bots.experts.domain_suffix.expert: Only print skipped database update message if verbose mode is active (PR#2107 by Sebastian Wagner, fixes #2016).
        •  
      "},{"location":"changelog/#documentation_5","title":"Documentation","text":"
         
      • Add configuration upgrade steps for 3.0 to NEWS (PR#2101 by Sebastian Wagner).
        •  
      "},{"location":"changelog/#known-issues_4","title":"Known issues","text":"
      See open bug reports for a more detailed list. - ParserBot: erroneous raw line recovery in error handling (#1850).
      "},{"location":"changelog/#301-2021-09-02","title":"3.0.1 (2021-09-02)","text":""},{"location":"changelog/#configuration_2","title":"Configuration","text":""},{"location":"changelog/#core_7","title":"Core","text":"
         
      • intelmq.lib.bot_debugger: Fix accessing the bot's destination queues (PR#2027 by Mikk Margus M\u00f6ll).
        •  
      • intelmq.lib.pipeline: Fix handling of load_balance parameter (PR#2027 by Mikk Margus M\u00f6ll).
        •  
      • intelmq.lib.bot: Fix handling of parameter destination_queues if value is an empty dictionary (PR#2051 by Sebastian Wagner, fixes #2034).
        •  
      "},{"location":"changelog/#bots_7","title":"Bots","text":""},{"location":"changelog/#collectors_5","title":"Collectors","text":"
         
      • intelmq.bots.collectors.shodan.collector_stream: Fix access to parameters, the bot wrongly used self.parameters (PR#2020 by Mikk Margus M\u00f6ll).
        •  
      • intelmq.bots.collectors.mail.collector_mail_attach: Add attachment file name as extra.file_name also if the attachment is not compressed (PR#2021 by Alex Kaplan).
        •  
      • intelmq.bots.collectors.http.collector_http_stream: Fix access to parameters, the bot wrongly used self.parameters (by Sebastian Wagner).
        •  
      "},{"location":"changelog/#parsers_5","title":"Parsers","text":"
         
      • intelmq.bots.parsers.microsoft.parser_ctip: Map Payload.domain to destination.fqdn instead of extra.payload.domain as it matches to destination.ip from DestinationIp (PR#2023 by Sebastian Wagner).
        •  
      • Removed intelmq.bots.parsers.malwaredomains because the upstream data source (malwaredomains.com) does not exist anymore (PR#2026 by Birger Schacht, fixes #2024).
        •  
      • intelmq.bots.parsers.shadowserver._config:
        •  
      • Add support for feed \"Vulnerable SMTP Server\" (PR#2037 by Mikk Margus M\u00f6ll).
        •  
      • Fix differentiation between feeds \"Accessible HTTP\" and \"Vulnerable HTTP\" (PR#2037 by Mikk Margus M\u00f6ll, fixes #1984).
        •  
      • Add support for the new feeds Microsoft Sinkhole Events Report, Microsoft Sinkhole HTTP Events Report (PR#2036 by Birger Schacht).
        •  
      • Complement feed mappings and documentation for feeds with IPv4 and IPv6 variants (PR#2046 by Mikk Margus M\u00f6ll and Sebastian Wagner).
        •  
      • Feed names with and without the optional IPv4/IPv6 postfix can be used now consistently.
        •  
      • Add support for feed \"Honeypot HTTP Scan\" (PR#2047 by Mikk Margus M\u00f6ll).
        •  
      • Update filename mapping for changed filename of feed \"Accessible-MSRDPUDP\" (PR#2060 by abr4xc).
        •  
      "},{"location":"changelog/#experts_7","title":"Experts","text":"
         
      • intelmq.bots.experts.gethostbyname.expert: Handle numeric values for the gaierrors_to_ignore parameter (PR#2073 by Sebastian Wagner, fixes #2072).
        •  
      • intelmq.bots.experts.filter.expert: Fix handling of empty-string parameters not_after and not_before (PR#2075 by Sebastian Wagner, fixes #2074).
        •  
      "},{"location":"changelog/#outputs_5","title":"Outputs","text":"
         
      • intelmq.bots.outputs.mcafee.output_esm_ip: Fix access to parameters, the bot wrongly used self.parameters (by Sebastian Wagner).
        •  
      • intelmq.bots.outputs.misp.output_api: Fix access to parameters, the bot wrongly used self.parameters (by Sebastian Wagner).
        •  
      • intelmq.bots.outputs.smtp.output: Add Content-Disposition-header to the attachment, fixing the display in Mail Clients as actual attachment (PR#2052 by Sebastian Wagner, fixes #2018).
        •  
      "},{"location":"changelog/#documentation_6","title":"Documentation","text":"
         
      • Various formatting fixes (by Sebastian Wagner).
        •  
      • Removed the malwaredomains feed from the feeds list because the upstream data source (malwaredomains.com) does not exist anymore (PR#2026 by Birger Schacht, fixes #2024).
        •  
      • Update Docker installation instructions (PR#2035 by Sebastian Waldbauer).
        •  
      "},{"location":"changelog/#packaging_3","title":"Packaging","text":"
         
      • intelmq-update-database crontab: Add missing recordedfuture_iprisk update call (by Sebastian Wagner).
        •  
      "},{"location":"changelog/#tests_4","title":"Tests","text":"
         
      • Replace calls to deprecated/undocumented logging.warn with logging.warning (by Sebastian Wagner, fixes #2013).
        •  
      • intelmq.tests.bots.experts.rdap.test_expert: Declare cache use, fixes build failures (by Sebastian Wagner, fixes #2014).
        •  
      • intelmq.tests.bots.collectors.mail.test_collector_attach: Test text attachment (by Sebastian Wagner).
        •  
      "},{"location":"changelog/#tools_4","title":"Tools","text":"
         
      • intelmqctl:
        •  
      • Also honour parameters from environment variables (PR#2068 by Sebastian Wagner, fixes #2063).
        •  
      • Fix management actions (start/stop/status/reload/restart) for groups (PR#2086 by Sebastian Wagner, fixes #2085).
        •  
      • Do not use hardcoded logging path in /opt/intelmq, use the internal default instead (PR#2092 by Sebastian Wagner, fixes #2091).
        •  
      "},{"location":"changelog/#known-issues_5","title":"Known issues","text":"
      See open bug reports for a more detailed list. - ParserBot: erroneous raw line recovery in error handling (#1850).
      "},{"location":"changelog/#300-2021-07-02","title":"3.0.0 (2021-07-02)","text":""},{"location":"changelog/#configuration_3","title":"Configuration","text":"
         
      • The BOTS file is no longer used and has been removed (by Sebastian Wagner).
        •  
      • The defaults.conf file is no longer used and has been removed (PR#1814 by Birger Schacht).
        •  
      • The pipeline.conf file is no longer used and has been removed (PR#1849 by Birger Schacht).
        •  
      • The runtime.conf was renamed to runtime.yaml and is now in YAML format (PR#1812 by Birger Schacht).
        •  
      "},{"location":"changelog/#core_8","title":"Core","text":"
         
      • intelmq.lib.harmonization:
        •  
      • New class ClassificationTaxonomy with fixed list of taxonomies and sanitiation (by Sebastian Wagner).
        •  
      • intelmq.lib.bot:
        •  
      • Handle InvalidValue exceptions upon message retrieval by dumping the message instead of repeating endlessly (#1765, PR#1766 by Filip Pokorn\u00fd).
        •  
      • Rewrite of the parameter loading and handling, getting rid of the parameters member (PR#1729 by Birger Schacht).
        •  
      • The pipeline is now initialized before the call of init to allow bots accessing data directly on startup/initialization for cleanup or maintenance tasks (PR#1982 by Sebastian Wagner).
        •  
      • intelmq.lib.exceptions:
        •  
      • InvalidValue: Add optional parameter object (PR#1766 by Filip Pokorn\u00fd).
        •  
      • intelmq.lib.utils:
        •  
      • New function list_all_bots to list all available/installed bots as replacement for the BOTS file (#368, #552, #644, #757, #1069, #1750, PR#1751 by Sebastian Waldbauer).
        •  
      • New function get_bots_settings to return the effective bot parameters, with global parameters applied (PR#1928 by Sebastian Wagner, #1927).
        •  
      • Removed deprecated function create_request_session_from_bot (PR#1997 by Sebastian Wagner, #1404).
        •  
      • parse_relative: Add support for parsing minutes and seconds (PR#1857 by Sebastian Wagner).
        •  
      • intelmq.lib.bot_debugger:
        •  
      • Set bot's logging_level directly in __init__ before the bot's initialization by changing the default value (by Sebastian Wagner).
        •  
      • Rewrite load_configuration_patch by adapting it to the parameter and configuration rewrite (by Sebastian Wagner).
        •  
      • Do not rely on the runtime configuration's group setting of bots to determine the required message type of messages given on the command line (PR#1949 by Sebastian Wagner).
        •  
      "},{"location":"changelog/#development_4","title":"Development","text":"
         
      • rewrite_config_files.py: Removed obsolete BOTS-file-related rewriting functionality (by Sebastian Wagner, #1543).
        •  
      • A GitHub Action that checks for reuse compliance of all the license and copyright headers was added (PR#1976 by Birger Schacht).
        •  
      • PyYAML is no longer a required dependency for development environments, all calls to it have been replaced by ruamel.yaml (by Sebastian Wagner).
        •  
      "},{"location":"changelog/#data-format_2","title":"Data Format","text":"
      The IntelMQ Data Harmonization (\"DHO\") is renamed to IntelMQ Data Format (\"IDF\"). Internal files remain and work the same as before (PR#1818 by Sebastian Waldbauer, fixes 1810). Update allowed classification fields to version 1.3 (2021-05-18) (by Sebastian Wagner, fixes #1409, #1476). - The taxonomy abusive content has been renamed to abusive-content. - The taxonomy information content security has been renamed to information-content-security.   - The validation of type unauthorised-information-access has been fixed, a bug prevented the use of it.   - The validation of type unauthorised-information-modification has been fixed, a bug prevented the use of it.   - The type leak has been renamed to data-leak.   - The type dropzone has been removed. Taxonomy other with type other and identifier dropzone can be used instead. Ongoing discussion in the RSIT WG. - The taxonomy intrusion attempts has been renamed to intrusion-attempts. - For the taxonomy intrusions (PR#1993 by Sebastian Wagner, addresses #1409):   - The type compromised has been renamed to system-compromise.   - The type unauthorized-command has been merged into system-compromise.   - The type unauthorized-login has been merged into system-compromise.   - The type backdoor has been merged into system-compromise (PR#1995 by Sebastian Wagner, addresses #1409).   - The type defacement has been merged into taxonomy information-content-security, type unauthorised-information-modification (PR#1994 by Sebastian Wagner, addresses #1409). - The taxonomy information gathering has been rename to information-gathering. - The taxonomy malicious code has been renamed to malicious-code.   - The type c2server has been renamed to c2-server.   - The type malware has been integrated into infected-system and malware-distribution, respectively (PR#1917 by Sebastian Wagner addresses #1409).   - The type ransomware has been integrated into infected-system.   - The type dga domain has been moved to the taxonomy other renamed dga-domain (PR#1992 by Sebastian Wagner fixes #1613). - For the taxonomy 'availability', the type misconfiguration is new. - For the taxonomy 'other', the type unknown has been renamed to undetermined. - For the taxonomy 'vulnerable':   - The type vulnerable client has been renamed to vulnerable-system.   - The type vulnerable service has been renamed to vulnerable-system.
      "},{"location":"changelog/#bots_8","title":"Bots","text":"
         
      • The parameters handling of numerous bots has been refactored (PR#1751, PR#1729, by Birger Schacht, Sebastian Wagner, Sebastian Waldbauer).
        •  
      "},{"location":"changelog/#collectors_6","title":"Collectors","text":"
         
      • Remove intelmq.bots.collectors.xmpp: one of the dependencies of the bot was deprecated and according to a short survey on the IntelMQ   users mailinglist, the bot is not used by anyone. (https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html, PR#1761 by Birger Schacht, closes #1614).
        •  
      • intelmq.bots.collectors.mail._lib: Added parameter mail_starttls for STARTTLS in all mail collector bots (PR#1831 by Marius Karotkis, fixes #1128).
        •  
      • Added intelmq.bots.collectors.fireeye: A bot that collects indicators from Fireeye MAS appliances (PR#1745 by Christopher Schappelwein).
        •  
      • intelmq.bots.collectors.api.collector_api (PR#1987 by Mikk Margus M\u00f6ll, fixes #1986):
        •  
      • Added UNIX socket capability.
        •  
      • Correctly close the IOLoop in the shutdown method to fix reload.
        •  
      • intelmq.bots.collectors.rt.collector_rt (PR#1997 by Sebastian Wagner, #1404):
        •  
      • compatibility with the deprecated parameter unzip_attachment (removed in 2.1.0) was removed.
        •  
      "},{"location":"changelog/#parsers_6","title":"Parsers","text":"
         
      • Added intelmq.bots.parsers.fireeye: A bot that parses hashes and URLs from Fireeye MAS indicators (PR#1745 by Christopher Schappelwein).
        •  
      • intelmq.bots.parsers.shadowserver._config:
        •  
      • Improved the feed-mapping and all conversion functions (PR#1971 by Mikk Margus M\u00f6ll).
        •  
      • intelmq.bots.parsers.generic.parser_csv:
        •  
      • Fix handling of empty string values for parameter time_format (by Sebastian Wagner).
        •  
      "},{"location":"changelog/#experts_8","title":"Experts","text":"
         
      • intelmq.bots.experts.domain_suffix.expert:
        •  
      • Added --update-database option to update domain suffix database (by Sebastian Wagner).
        •  
      • Fix check method: load database with UTF-8 encoding explicitly (by Sebastian Wagner).
        •  
      • Added intelmq.bots.experts.http.expert_status: A bot that fetches the HTTP Status for a given URI and adds it to the message (PR#1789 by Birger Schacht, fixes #1047 partly).
        •  
      • Added intelmq.bots.experts.http.expert_content: A bot that fetches an HTTP resource and checks if it contains a specific string (PR#1811 by Birger Schacht).
        •  
      • Added intelmq.bots.experts.lookyloo.expert: A bot that sends requests to a lookyloo instance & adds screenshot_url to the event (PR#1844 by Sebastian Waldbauer, fixes #1048).
        •  
      • Added intelmq.bots.experts.rdap.expert: A bot that checks the rdap protocol for an abuse contact for a given domain (PR#1881 by Sebastian Waldbauer and Sebastian Wagner).
        •  
      • intelmq.bots.experts.sieve.expert:
        •  
      • Add operators for comparing lists and sets (PR#1895 by Mikk Margus M\u00f6ll):
           
        • :equals
          •  
        • :overlaps
          •  
        • :supersetof
          •  
        • :subsetof
          •  
        • :equals
          •  
         
        •  
      • Add support for comparing boolean values (PR#1895 by Mikk Margus M\u00f6ll).
        •  
      • Add support for rule negation with ! (PR#1895, PR#1923 by Mikk Margus M\u00f6ll).
        •  
      • Add support for values types float, int, bool and string for all lists items (PR#1895 by Mikk Margus M\u00f6ll).
        •  
      • Add actions for lists (PR#1895 by Mikk Margus M\u00f6ll).
           
        • append
          •  
        • append! (forced/overwriting)
          •  
         
        •  
      • Rewrite the rule-processing and operator-handling code to make it more comprehensible and extensible (PR#1895, PR#1923 by Mikk Margus M\u00f6ll).
        •  
      • Nested if statements, plus mixed actions and actions in the same scope (PR #1923 by Mikk Margus M\u00f6ll).
        •  
      • The attribute manipulation actions add, add! and update support non-string (bool/int/float) values (PR #1923 by Mikk Margus M\u00f6ll).
        •  
      • Drop the :notcontains operator, as it made is redundant by generic negation: ! foo :contains 'x' instead of foo :notcontains 'x' (PR#1957 by Mikk Margus M\u00f6ll).
        •  
      • Split string and numeric matches into single- and multivalued variants, with the relevant new operators :in, :containsany and :regexin for string lists, and :in for numeric value lists (PR#1957 by Mikk Margus M\u00f6ll).
           
        • Removed the == operator for lists, with the previous meaning of :in. Have a look at the NEWS.md for more information.
          •  
         
        •  
      • Added intelmq.bots.experts.uwhoisd: A bot that fetches the whois entry from a uwhois-instance (PR#1918 by Rapha\u00ebl Vinot).
        •  
      • Removed deprecated intelmq.bots.experts.ripencc_abuse_contact.expert. It was replaced by intelmq.bots.experts.ripe.expert and marked as deprecated in 2.0.0.beta1 (PR#1997 by Sebastian Wagner, #1404).
        •  
      • intelmq.bots.experts.modify.expert:
        •  
      • Removed compatibility with deprecated configuration format before 1.0.0.dev7 (PR#1997 by Sebastian Wagner, #1404).
        •  
      • Added intelmq.bots.experts.aggregate: A bot that aggregate events based upon given fields & a timespan (PR#1959 by Sebastian Waldbauer).
        •  
      • Added intelmq.bots.experts.tuency: A bot that queries the IntelMQ API of a tuency instance (PR#1857 by Sebastian Wagner, fixes #1856).
        •  
      "},{"location":"changelog/#outputs_6","title":"Outputs","text":"
         
      • Remove intelmq.bots.outputs.xmpp: one of the dependencies of the bot was deprecated and according to a short survey on the IntelMQ   users mailinglist, the bot is not used by anyone. (https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html, PR#1761 by Birger Schacht, closes #1614)
        •  
      • intelmq.bots.outputs.smtp: Add more debug logging (PR#1949 by Sebastian Wagner).
        •  
      • Added new bot intelmq.bots.outputs.templated_smtp (PR#1901 by Karl-Johan Karlsson).
        •  
      "},{"location":"changelog/#documentation_7","title":"Documentation","text":"
         
      • Updated user and developer documentation to reflect the removal of the BOTS file (PR#1780 by Birger Schacht).
        •  
      • Bots documentation:
        •  
      • Added anchors to all bot sections derived from the module names for easier linking (PR#1943 by Sebastian Wagner fixes part of certtools/intelmq-api#4).
        •  
      • License and copyright information was added to all the bots (PR#1976 by Birger Schacht).
        •  
      • Added documentation on the EventDB (PR#1955 by Birger Schacht, PR#1985 by Sebastian Wagner).
        •  
      • Added TimescaleDB for time-series documentation (PR#1990 by Sebastian Waldbauer).
        •  
      • Improved n6 interoperability documentation by adding more graphs and illustrations (PR#1991 by Sebastian Wagner).
        •  
      • Feed documentation generation: fix and simplify formatting of parameters of types lists, non-string values have been ill-treated (by Sebastian Wagner).
        •  
      • Added documentation on abuse-contact look-ups (PR#2021 by Sebastian Waldbauer and Sebastian Wagner).
        •  
      "},{"location":"changelog/#packaging_4","title":"Packaging","text":"
         
      • Docker images tagged with certat/intelmq-full:develop are built and published on every push to the develop branch (PR#1753 by Sebastian Waldbauer).
        •  
      • Adapt packaging to IntelMQ 3.0 changes: ruamel.yaml dependency, changed configuration, updated database-update scripts (by Birger Schacht and Sebastian Wagner).
        •  
      "},{"location":"changelog/#tests_5","title":"Tests","text":"
         
      • intelmq.tests.lib.test_bot:
        •  
      • Add test case for a raised InvalidValue exception upon message retrieval (#1765, PR#1766 by Filip Pokorn\u00fd and Sebastian Wagner).
        •  
      • intelmq.lib.test:
        •  
      • Compare content of the output field as dictionaries, not as string in assertMessageEqual (PR#1975 by Karl-Johan Karlsson).
        •  
      • Support multiple calls to run_bot from test cases (PR#1989 by Sebastian Wagner).
           
        • Split prepare_source_queue out of prepare_bot.
          •  
        • Added new optional parameter stop_bot to run_bot.
          •  
         
        •  
      "},{"location":"changelog/#tools_5","title":"Tools","text":"
         
      • intelmqdump (PR#1997 by Sebastian Wagner, #1404):
        •  
      • The command e for deleting single entries by given IDs has been merged into the command d (\"delete\"), which can now delete either entries by ID or the whole file.
        •  
      • The command v for editing entries has been renamed to e (\"edit\").
        •  
      "},{"location":"changelog/#contrib_2","title":"Contrib","text":"
         
      • eventdb:
        •  
      • Added separate-raws-table.sql (PR#1985 by Sebastian Wagner).
        •  
      • cron-jobs: Removed the deprecated update scripts (PR#1997 by Sebastian Wagner, #1404):
        •  
      • update-asn-data
        •  
      • update-geoip-data
        •  
      • update-tor-nodes
        •  
      • update-rfiprisk-data   in favor of the built-in update-mechanisms (see the bots' documentation). A crontab file for calling all new update command can be found in contrib/cron-jobs/intelmq-update-database.
        •  
      "},{"location":"changelog/#known-issues_6","title":"Known issues","text":"
         
      • ParserBot: erroneous raw line recovery in error handling (#1850).
        •  
      • ruamel.yaml loader and dumper: human readability bug / support for comments (#2003).
        •  
      "},{"location":"changelog/#233-2021-05-31","title":"2.3.3 (2021-05-31)","text":""},{"location":"changelog/#core_9","title":"Core","text":"
         
      • intelmq.lib.upgrade:
        •  
      • Added v233_feodotracker_browse for Abuse.ch Feodotracker Browse parser configuration adaption (PR#1941 by Sebastian Wagner).
        •  
      "},{"location":"changelog/#bots_9","title":"Bots","text":""},{"location":"changelog/#parsers_7","title":"Parsers","text":"
         
      • intelmq.bots.parsers.microsoft.parser_ctip:
        •  
      • Add support for new field SourceIpInfo.SourceIpv4Int (PR#1940 by Sebastian Wagner).
        •  
      • Fix mapping of \"ConnectionType\" fields, this is not protocol.application. Now mapped to extra.*.connection_type (PR#1940 by Sebastian Wagner).
        •  
      • intelmq.bots.parsers.shadowserver._config:
        •  
      • Add support for the new feeds Honeypot-Amplification-DDoS-Events, Honeypot-Brute-Force-Events, Honeypot-Darknet, IP-Spoofer-Events, Sinkhole-Events, Sinkhole-HTTP-Events, Vulnerable-Exchange-Server, Sinkhole-Events-HTTP-Referer (PR#1950, PR#1952, PR#1953, PR#1954, PR#1970 by Birger Schacht and Sebastian Wagner, PR#1971 by Mikk Margus M\u00f6ll).
        •  
      "},{"location":"changelog/#experts_9","title":"Experts","text":"
         
      • intelmq.bots.experts.splunk_saved_search.expert:
        •  
      • fixed erroneous string formatting (PR#1960 by Karl-Johan Karlsson).
        •  
      "},{"location":"changelog/#outputs_7","title":"Outputs","text":"
         
      • intelmq.bots.outputs.smtp.output:
        •  
      • Handle empty \"fieldnames\" parameter by sending no attachment (PR#1932 by Sebastian Wagner).
        •  
      "},{"location":"changelog/#documentation_8","title":"Documentation","text":"
         
      • dev/data-harmonization renamed to dev/data-format (by Sebastian Waldbauer)
        •  
      • Feeds:
        •  
      • Fixed Abuse.ch Feodotracker Browse parser configuration (PR#1941 by Sebastian Wagner fixes #1938).
        •  
      "},{"location":"changelog/#tests_6","title":"Tests","text":"
         
      • intelmq.bots.parsers.html_table:
        •  
      • Added testcase for Abuse.ch Feodotracker Browse (PR#1941 by Sebastian Wagner).
        •  
      "},{"location":"changelog/#tools_6","title":"Tools","text":"
         
      • intelmqsetup:
        •  
      • Set ownershop of state file path and its parent directory (PR#1911 by Sebastian Wagner).
        •  
      "},{"location":"changelog/#known-issues_7","title":"Known issues","text":"
         
      • ParserBot: erroneous raw line recovery in error handling (#1850).
        •  
      "},{"location":"changelog/#232-2021-04-27","title":"2.3.2 (2021-04-27)","text":""},{"location":"changelog/#core_10","title":"Core","text":"
         
      • intelmq.lib.harmonization:
        •  
      • TLP type: accept value \"yellow\" for TLP level AMBER.
        •  
      "},{"location":"changelog/#bots_10","title":"Bots","text":""},{"location":"changelog/#collectors_7","title":"Collectors","text":"
         
      • intelmq.bots.collectors.shadowserver.collector_reports_api:
        •  
      • Handle timeouts by logging the error and continuing to next report (PR#1852 by Marius Karotkis and Sebastian Wagner, fixes #1823).
        •  
      "},{"location":"changelog/#parsers_8","title":"Parsers","text":"
         
      • intelmq.bots.parsers.shadowserver.config:
        •  
      • Parse and harmonize field end_time as date in Feeds \"Drone-Brute-Force\" and \"Amplification-DDoS-Victim\" (PR#1833 by Mikk Margus M\u00f6ll).
        •  
      • Add conversion function convert_date_utc which assumes UTC and sanitizes the data to datetime (by Sebastian Wagner, fixes #1848).
        •  
      • intelmq.bots.parsers.shadowserver.parser_json:
        •  
      • Use the overwrite parameter for optionally overwriting the \"feed.name\" field (by Sebastian Wagner).
        •  
      • intelmq.bots.parsers.microsoft.parser_ctip:
        •  
      • Handle fields timestamp, timestamp_utc, source_ip, source_port, destination_ip, destination_port, computer_name, bot_id, asn, geo in Payload of CTIP Azure format (PR#1841, PR#1851 and PR#1879 by Sebastian Wagner).
        •  
      • intelmq.bots.parsers.shodan.parser:
        •  
      • Added support for unique keys and verified vulns (PR#1835 by Mikk Margus M\u00f6ll).
        •  
      • intelmq.bots.parsers.cymru.parser_cap_program:
        •  
      • Fix parsing in whitespace edge case in comments (PR#1870 by Alex Kaplan, fixes #1862).
        •  
      "},{"location":"changelog/#experts_10","title":"Experts","text":"
         
      • intelmq.bots.experts.modify:
        •  
      • Add a new rule to the example configuration to change the type of malicious-code events to c2server if the malware name indicates c2 (PR#1854 by Sebastian Wagner).
        •  
      • intelmq.bots.experts.gethostbyname.expert:
        •  
      • Fix handling of parameter gaierrors_to_ignore with value None (PR#1890 by Sebastian Wagner, fixes #1886).
        •  
      "},{"location":"changelog/#outputs_8","title":"Outputs","text":"
         
      • intelmq.bots.outputs.elasticsearch: Fix log message on required elasticsearch library message (by Sebastian Wagner).
        •  
      "},{"location":"changelog/#documentation_9","title":"Documentation","text":"
         
      • dev/data-harmonization: Fix taxonomy name \"information gathering\" should be \"information-gathering\" (by Sebastian Wagner).
        •  
      "},{"location":"changelog/#tests_7","title":"Tests","text":"
         
      • intelmq.tests.bots.parsers.microsoft.test_parser_ctip_azure:
        •  
      • Add test case for TLP level \"YELLOW\".
        •  
      "},{"location":"changelog/#known-issues_8","title":"Known issues","text":"
         
      • ParserBot: erroneous raw line recovery in error handling (#1850).
        •  
      "},{"location":"changelog/#231-2021-03-25","title":"2.3.1 (2021-03-25)","text":""},{"location":"changelog/#configuration_4","title":"Configuration","text":""},{"location":"changelog/#core_11","title":"Core","text":"
         
      • intelmq.lib.utils:
        •  
      • log: Handle null value for logging parameter logging_max_size (PR#1786 by Sebastian Wagner, fixes #1778).
        •  
      • intelmq.lib.pipeline:
        •  
      • Amqp._get_queues: Check virtual host when retrieving queue sizes. Fixes output of intelmqctl check for orphaned queues if AMQP is used and the AMQP user has access to more virtual hosts (PR#1830 by Sebastian Wagner, fixes #1746).
        •  
      "},{"location":"changelog/#bots_11","title":"Bots","text":""},{"location":"changelog/#collectors_8","title":"Collectors","text":"
         
      • intelmq.bots.collectors.shadowserver.collector_reports_api: Added debug logging to show number of downloaded reports and download size (PR#1826 by Sebastian Wagner, partly addresses #1688 and #1823).
        •  
      "},{"location":"changelog/#parsers_9","title":"Parsers","text":"
         
      • intelmq.bots.parsers.cymru.parser_cap_program:
        •  
      • Adapt parser to new upstream format for events of category \"bruteforce\" (PR#1795 by Sebastian Wagner, fixes 1794).
        •  
      • intelmq.bots.parsers.shodan.parser:
        •  
      • Support nested conversions, improved protocol detection and extended Shodan parser mappings (PR#1821 by Mikk Margus M\u00f6ll).
        •  
      "},{"location":"changelog/#documentation_10","title":"Documentation","text":"
         
      • Add missing newlines at end of docs/_static/intelmq-manager/*.png.license files (PR#1785 by Sebastian Wagner, fixes #1777).
        •  
      • Ecosystem: Revise sections on intelmq-cb-mailgen and fody (PR#1792 by Bernhard Reiter).
        •  
      • intelmq-api: Add documentation about necessary write permission for the session database file (PR#1798 by Birger Schacht, fixes intelmq-api#23).
        •  
      • FAQ: Section on redis socket permissions: set only minimal necessary permissions (PR#1809 by Sebastian Wagner).
        •  
      • Add document on hardware requirements (PR#1811 by Sebastian Wagner).
        •  
      • Feeds: Added Shodan Country Stream (by Sebastian Wagner).
        •  
      "},{"location":"changelog/#tests_8","title":"Tests","text":"
         
      • Add missing newlines at end of various test input files (PR#1785 by Sebastian Wagner, fixes #1777).
        •  
      • intelmq.tests.bots.parsers.shodan.test_parser: Add test cases for new code (PR#1821 by Mikk Margus M\u00f6ll).
        •  
      • intelmq.tests.lib.test_harmonization.test_datetime_convert: Only run this test in timezone UTC (PR#1825 by Sebastian Wagner).
        •  
      "},{"location":"changelog/#tools_7","title":"Tools","text":"
         
      • intelmqsetup:
        •  
      • Also cover required directory layout and file permissions for intelmq-api (PR#1787 by Sebastian Wagner, fixes #1783).
        •  
      • Also cover webserver and sudoers configuration for intelmq-api and intelmq-manger (PR#1805 by Sebastian Wagner, fixes #1803).
        •  
      • intelmqctl:
        •  
      • Do not log an error message if logging to file is explicitly disabled, e.g. in calls from intelmsetup. The error message would not be useful for the user and is not necessary.
        •  
      "},{"location":"changelog/#known-issues_9","title":"Known issues","text":"
         
      • Bots started with IntelMQ-API/Manager stop when the webserver is restarted (#952).
        •  
      • Corrupt dump files when interrupted during writing (#870).
        •  
      • CSV line recovery forces Windows line endings (#1597).
        •  
      • intelmqdump: Honor logging_path variable (#1605).
        •  
      • Timeout error in mail URL fetcher (#1621).
        •  
      • Shadowserver Parser: Drone feed has (also?) application protocol in type field (mapped to transport protocol) (#1763).
        •  
      "},{"location":"changelog/#230-2021-03-04","title":"2.3.0 (2021-03-04)","text":"
      IntelMQ no longer supports Python 3.5 (and thus Debian 9 and Ubuntu 16.04), the minimum supported Python version is 3.6.
      "},{"location":"changelog/#configuration_5","title":"Configuration","text":""},{"location":"changelog/#core_12","title":"Core","text":"
         
      • intelmq.lib.bot:
        •  
      • ParserBot.recover_line_json_stream: Make line parameter optional, as it is not needed for this method (by Sebastian Wagner).
        •  
      • Bot.argparser: Added class method _create_argparser (returns argparse.ArgumentParser) for easy command line arguments parsing (PR#1586 by Filip Pokorn\u00fd).
        •  
      • Runtime configuration does not necessarily need a parameter entry for each block. Previously at least an empty block was required (PR#1604 by Filip Pokorn\u00fd).
        •  
      • Allow setting the pipeline host and the Redis cache host by environment variables for docker usage (PR#1669 by Sebastian Waldbauer).
        •  
      • Better logging message for SIGHUP handling if the handling of the signal is not delayed (by Sebastian Wagner).
        •  
      • intelmq.lib.upgrades:
        •  
      • Add upgrade function for removal of HPHosts Hosts file feed and intelmq.bots.parsers.hphosts parser (#1559, by Sebastian Wagner).
        •  
      • intelmq.lib.exceptions:
        •  
      • PipelineError: Remove unused code to format exceptions (by Sebastian Wagner).
        •  
      • intelmq.lib.utils:
        •  
      • create_request_session_from_bot:
           
        • Changed bot argument to optional, uses defaults.conf as fallback, renamed to create_request_session. Name create_request_session_from_bot will be removed in version 3.0.0 (PR#1524 by Filip Pokorn\u00fd).
          •  
        • Fixed setting of http_verify_cert from defaults configuration (PR#1758 by Birger Schacht).
          •  
         
        •  
      • log: Use RotatingFileHandler for allow log file rotation without external tools (PR#1637 by Vasek Bruzek).
        •  
      • intelmq.lib.harmonization:
        •  
      • The IPAddress type sanitation now accepts integer IP addresses and converts them to the string representation (by Sebastian Wagner).
        •  
      • DateTime.parse_utc_isoformat: Add parameter return_datetime to return datetime object instead of string ISO format (by Sebastian Wagner).
        •  
      • DateTime.convert: Fix utc_isoformat format, it pointed to a string and not a function, causing an exception when used (by Sebastian Wagner).
        •  
      • DateTime.from_timestamp: Ensure that time zone information (+00:00) is always present (by Sebastian Wagner).
        •  
      • DateTime.__parse now handles OverflowError exceptions from the dateutil library, happens for large numbers, e.g. telehpone numbers (by Sebastian Wagner).
        •  
      • intelmq.lib.upgrades:
        •  
      • Added upgrade function for CSV parser parameter misspelling (by Sebastian Wagner).
        •  
      • Check for existence of collector and parser for the obsolete Malware Domain List feed and raise warning if found (#1762, PR#1771 by Birger Schacht).
        •  
      "},{"location":"changelog/#development_5","title":"Development","text":"
         
      • intelmq.bin.intelmq_gen_docs:
        •  
      • Add bot name to the resulting feed documentation (PR#1617 by Birger Schacht).
        •  
      • Merged into docs/autogen.py (PR#1622 by Birger Schacht).
        •  
      "},{"location":"changelog/#bots_12","title":"Bots","text":""},{"location":"changelog/#collectors_9","title":"Collectors","text":"
         
      • intelmq.bots.collectors.eset.collector: Added (PR#1554 by Mikk Margus M\u00f6ll).
        •  
      • intelmq.bots.collectors.http.collector_http:
        •  
      • Added PGP signature check functionality (PR#1602 by sinus-x).
        •  
      • If status code is not 2xx, the request's and response's headers and body are logged in debug logging level (#1615, by Sebastian Wagner).
        •  
      • intelmq.bots.collectors.kafka.collector: Added (PR#1654 by Birger Schacht, closes #1634).
        •  
      • intelmq.bots.collectors.xmpp.collector: Marked as deprecated, see https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html (#1614, PR#1685 by Birger Schacht).
        •  
      • intelmq.bots.collectors.shadowserver.collector_api:
        •  
      • Added (#1683, PR#1700 by Birger Schacht).
        •  
      • Change file names in the report to .json instead of the original and wrong .csv (PR#1769 by Sebastian Wagner).
        •  
      • intelmq.bots.collectors.mail: Add content of the email's Date header as extra.email_date to the report in all email collectors (PR#1749 by aleksejsv and Sebastian Wagner).
        •  
      • intelmq.bots.collectors.http.collector_http_stream: Retry on common connection issues without raising exceptions (#1435, PR#1747 by Sebastian Waldbauer and Sebastian Wagner).
        •  
      • intelmq.bots.collectors.shodan.collector_stream: Retry on common connection issues without raising exceptions (#1435, PR#1747 by Sebastian Waldbauer and Sebastian Wagner).
        •  
      • intelmq.bots.collectors.twitter.collector_twitter:
        •  
      • Proper input validation in URLs using urllib. CWE-20, found by GitHub's CodeQL (PR#1754 by Sebastian Wagner).
        •  
      • Limit replacement (\"pastebin.com\", \"pastebin.com/raw\") to a maximum of one (PR#1754 by Sebastian Wagner).
        •  
      "},{"location":"changelog/#parsers_10","title":"Parsers","text":"
         
      • intelmq.bots.parsers.eset.parser: Added (PR#1554 by Mikk Margus M\u00f6ll).
        •  
      • Ignore invalid \"NXDOMAIN\" IP addresses (PR#1573 by Mikk Margus M\u00f6ll).
        •  
      • intelmq.bots.parsers.hphosts: Removed, feed is unavailable (#1559, by Sebastian Wagner).
        •  
      • intelmq.bots.parsers.cznic.parser_haas: Added (PR#1560 by Filip Pokorn\u00fd and Edvard Rejthar).
        •  
      • intelmq.bots.parsers.cznic.parser_proki: Added (PR#1599 by sinus-x).
        •  
      • intelmq.bots.parsers.key_value.parser: Added (PR#1607 by Karl-Johan Karlsson).
        •  
      • intelmq.bots.parsers.generic.parser_csv: Added new parameter compose_fields (by Sebastian Wagner).
        •  
      • intelmq.bots.parsers.shadowserver.parser_json: Added (PR#1700 by Birger Schacht).
        •  
      • intelmq.bots.parsers.shadowserver.config:
        •  
      • Fixed mapping for Block list feed to accept network ranges in CIDR notation (#1720, PR#1728 by Sebastian Waldbauer).
        •  
      • Added mapping for new feed MSRDPUDP, Vulnerable-HTTP, Sinkhole DNS (#1716, #1726, #1733, PR#1732, PR#1735, PR#1736 by Sebastian Waldbauer).
        •  
      • Ignore value 0 for source.asn and destination.asn in all mappings to avoid parsing errors (PR#1769 by Sebastian Wagner).
        •  
      • intelmq.bots.parsers.abusech.parser_ip: Adapt to changes in the Feodo Tracker Botnet C2 IP Blocklist feed (PR#1741 by Thomas Bellus).
        •  
      • intelmq.bots.parsers.malwaredomainlist: Removed, as the feed is obsolete (#1762, PR#1771 by Birger Schacht).
        •  
      "},{"location":"changelog/#experts_11","title":"Experts","text":"
         
      • intelmq.bots.experts.rfc1918.expert:
        •  
      • Add support for ASNs (PR#1557 by Mladen Markovic).
        •  
      • Speed improvements.
        •  
      • More output in debug logging mode (by Sebastian Wagner).
        •  
      • Checks parameter length on initialization and in check method (by Sebastian Wagner).
        •  
      • intelmq.bots.experts.gethostbyname.expert:
        •  
      • Added parameter fallback_to_url and set to True (PR#1586 by Edvard Rejthar).
        •  
      • Added parameter gaierrors_to_ignore to optionally ignore other gethostbyname errors (#1553).
        •  
      • Added parameter overwrite to optionally overwrite existing IP addresses (by Sebastian Wagner).
        •  
      • intelmq.bots.experts.asn_lookup.expert:
        •  
      • Added --update-database option (PR#1524 by Filip Pokorn\u00fd).
        •  
      • The script update-asn-data is now deprecated and will be removed in version 3.0.
        •  
      • intelmq.bots.experts.maxmind_geoip.expert:
        •  
      • Added --update-database option (PR#1524 by Filip Pokorn\u00fd).
        •  
      • Added license_key parameter (PR#1524 by Filip Pokorn\u00fd).
        •  
      • The script update-geoip-data is now deprecated and will be removed in version 3.0.
        •  
      • intelmq.bots.experts.tor_nodes.expert:
        •  
      • Added --update-database option (PR#1524 by Filip Pokorn\u00fd).
        •  
      • The script update-tor-nodes is now deprecated and will be removed in version 3.0.
        •  
      • intelmq.bots.experts.recordedfuture_iprisk.expert:
        •  
      • Added --update-database option (PR#1524 by Filip Pokorn\u00fd).
        •  
      • Added api_token parameter (PR#1524 by Filip Pokorn\u00fd).
        •  
      • The script update-rfiprisk-data is now deprecated and will be removed in version 3.0.
        •  
      • Added intelmq.bots.experts.threshold (PR#1608 by Karl-Johan Karlsson).
        •  
      • Added intelmq.bots.experts.splunk_saved_search.expert (PR#1666 by Karl-Johan Karlsson).
        •  
      • intelmq.bots.experts.sieve.expert:
        •  
      • Added possibility to give multiple queue names for the path directive (#1462, by Sebastian Wagner).
        •  
      • Added possibility to run actions without filtering expression (#1706, PR#1708 by Sebastian Waldbauer).
        •  
      • Added datetime math operations (#1680, PR#1696 by Sebastian Waldbauer).
        •  
      • intelmq.bots.experts.maxmind_geoip.expert:
        •  
      • Fixed handing over of overwrite parameter to event.add (PR#1743 by Birger Schacht).
        •  
      "},{"location":"changelog/#outputs_9","title":"Outputs","text":"
         
      • intelmq.bots.outputs.rt: Added Request Tracker output bot (PR#1589 by Marius Urkis).
        •  
      • intelmq.bots.outputs.xmpp.output: Marked as deprecated, see https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html (#1614, PR#1685 by Birger Schacht).
        •  
      • intelmq.bots.outputs.smtp.output: Fix sending to multiple recipients when recipients are defined by event-data (#1759, PR#1760 by Sebastian Waldbauer and Sebastian Wagner).
        •  
      "},{"location":"changelog/#documentation_11","title":"Documentation","text":"
         
      • Feeds:
        •  
      • Add ESET URL and Domain feeds (by Sebastian Wagner).
        •  
      • Remove unavailable HPHosts Hosts file feed (#1559 by Sebastian Wagner).
        •  
      • Added CZ.NIC HaaS feed (PR#1560 by Filip Pokorn\u00fd and Edvard Rejthar).
        •  
      • Added CZ.NIC Proki feed (PR#1599 by sinus-x).
        •  
      • Updated Abuse.ch URLhaus feed (PT#1572 by Filip Pokorn\u00fd).
        •  
      • Added CERT-BUND CB-Report Malware infections feed (PR#1598 by sinus-x and Sebastian Wagner).
        •  
      • Updated Turris Greylist feed with PGP verification information (by Sebastian Wagner).
        •  
      • Fixed parsing of the public field in the generated feeds documentation (PR#1641 by Birger Schacht).
        •  
      • Change the rate_limit parameter of some feeds from 2 days (129600 seconds) to one day (86400 seconds).
        •  
      • Update the cAPTure Ponmocup Domains feed documentation (PR#1574 by Filip Pokorn\u00fd and Sebastian Wagner).
        •  
      • Added Shadowserver Reports API (by Sebastian Wagner).
        •  
      • Change the rate_limit parameter for many feeds from 2 days to the default one day (by Sebastian Wagner).
        •  
      • Removed Malware Domain List feed, as the feed is obsolete (#1762, PR#1771 by Birger Schacht).
        •  
      • Bots:
        •  
      • Enhanced documentation of RFC1918 Expert (PR#1557 by Mladen Markovic and Sebastian Wagner).
        •  
      • Enhanced documentation of SQL Output (PR#1620 by Edvard Rejthar).
        •  
      • Updated documentation for MaxMind GeoIP, ASN Lookup, TOR Nodes and Recorded Future experts to reflect new --update-database option (PR#1524 by Filip Pokorn\u00fd).
        •  
      • Added documentation for Shadowserver API collector and parser (PR#1700 by Birger Schacht and Sebastian Wagner).
        •  
      • Add n6 integration documentation (by Sebastian Wagner).
        •  
      • Moved 'Orphaned Queues' section from the FAQ to the intelmqctl documentation (by Sebastian Wagner).
        •  
      • Generate documentation using Sphinx (PR#1622 by Birger Schacht).
        •  
      • The documentation is now available at https://intelmq.readthedocs.io/en/latest/
        •  
      • Refactor documentation and fix broken syntax (#1639, PRs #1638 #1640 #1642 by Birger Schacht).
        •  
      • Integrate intelmq-manager and intelmq-api user documentation to provide unified documentation place (PR#1714 & PR#1714 by Birger Schacht).
        •  
      "},{"location":"changelog/#packaging_5","title":"Packaging","text":"
         
      • Fix paths in the packaged logcheck rules (by Sebastian Wagner).
        •  
      • Build the sphinx documentation on package build (PR#1701 by Birger Schacht).
        •  
      • Ignore non-zero exit-codes for the intelmqctl check call in postinst (#1748, by Sebastian Wagner).
        •  
      "},{"location":"changelog/#tests_9","title":"Tests","text":"
         
      • Added tests for intelmq.lib.exceptions.PipelineError (by Sebastian Wagner).
        •  
      • intelmq.tests.bots.collectors.http_collector.test_collector: Use requests_mock to mock all requests and do not require a local webserver (by Sebastian Wagner).
        •  
      • intelmq.tests.bots.outputs.restapi.test_output:
        •  
      • Use requests_mock to mock all requests and do not require a local webserver (by Sebastian Wagner).
        •  
      • Add a test for checking the response status code (by Sebastian Wagner).
        •  
      • intelmq.tests.bots.collectors.mail.test_collector_url: Use requests_mock to mock all requests and do not require a local webserver (by Sebastian Wagner).
        •  
      • intelmq.tests.bots.experts.ripe.test_expert: Use requests_mock to mock all requests and do not require a local webserver (by Sebastian Wagner).
        •  
      • The test flag (environment variable) INTELMQ_TEST_LOCAL_WEB is no longer used (by Sebastian Wagner).
        •  
      • Added tests for intelmq.harmonization.DateTime.parse_utc_isoformat and convert_fuzzy (by Sebastian Wagner).
        •  
      • Move from Travis to GitHub Actions (PR#1707 by Birger Schacht).
        •  
      • intelmq.lib.test:
        •  
      • test_static_bot_check_method checks the bot's static check(parameters) method for any exceptions, and a valid formatted return value (#1505, by Sebastian Wagner).
        •  
      • setUpClass: Skip tests if cache was requests with use_cache member, but Redis is deactivated with the environment variable INTELMQ_SKIP_REDIS (by Sebastian Wagner).
        •  
      • intelmq.tests.bots.experts.cymru_whois.test_expert:
        •  
      • Switch from example.com to ns2.univie.ac.at for hopefully more stable responses (#1730, PR#1731 by Sebastian Waldbauer).
        •  
      • Do not test for exact expected values in the 6to4 network test, as the values are changing regularly (by Sebastian Wagner).
        •  
      • intelmq.tests.bots.parsers.abusech: Remove tests cases of discontinued feeds (PR#1741 by Thomas Bellus).
        •  
      • Activate GitHub's CodeQL Code Analyzing tool as GitHub Action (by Sebastian Wagner).
        •  
      "},{"location":"changelog/#tools_8","title":"Tools","text":"
         
      • intelmqdump:
           
        • Check if given queue is configured upon recovery (#1433, PR#1587 by Mladen Markovic).
          •  
         
        •  
      • intelmqctl:
        •  
      • intelmq list queues: --sum, --count, -s flag for showing total count of messages (#1408, PR#1581 by Mladen Markovic).
        •  
      • intelmq check: Added a possibility to ignore queues from the orphaned queues check (by Sebastian Wagner).
        •  
      • Allow setting the pipeline host by environment variables for docker usage (PR#1669 by Sebastian Waldbauer).
        •  
      "},{"location":"changelog/#contrib_3","title":"Contrib","text":"
         
      • EventDB:
        •  
      • Add SQL script for keeping track of the oldest inserted/update \"time.source\" information (by Sebastian Wagner).
        •  
      • Cron Jobs: The script intelmq-update-data has been renamed to intelmq-update-database (by Filip Pokorn\u00fd).
        •  
      • Dropped utterly outdated contrib modules (by Sebastian Wagner):
        •  
      • ansible
        •  
      • vagrant
        •  
      • vagrant-ansible
        •  
      • logrotate:
        •  
      • Do not use the deprecated \"copytruncate\" option as intelmq re-opens the log anyways (by Sebastian Wagner).
        •  
      • Set file permissions to 0644 (by Sebastian Wagner).
        •  
      "},{"location":"changelog/#known-issues_10","title":"Known issues","text":"
         
      • Bots started with IntelMQ-API/Manager stop when the webserver is restarted (#952).
        •  
      • Corrupt dump files when interrupted during writing (#870).
        •  
      • CSV line recovery forces Windows line endings (#1597).
        •  
      • Timeout error in mail URL fetcher (#1621).
        •  
      • AMQP pipeline: get_queues needs to check vhost of response (#1746).
        •  
      "},{"location":"changelog/#223-2020-12-23","title":"2.2.3 (2020-12-23)","text":""},{"location":"changelog/#documentation_12","title":"Documentation","text":"
         
      • Bots/Sieve expert: Add information about parenthesis in if-expressions (#1681, PR#1687 by Birger Schacht).
        •  
      "},{"location":"changelog/#harmonization","title":"Harmonization","text":"
         
      • See NEWS.md for information on a fixed bug in the taxonomy expert.
        •  
      "},{"location":"changelog/#bots_13","title":"Bots","text":""},{"location":"changelog/#collectors_10","title":"Collectors","text":"
         
      • intelmq.bots.rt.collector_rt: Log the size of the downloaded file in bytes on debug logging level.
        •  
      "},{"location":"changelog/#parsers_11","title":"Parsers","text":"
         
      • intelmq.bots.parsers.cymru.parser_cap_program:
        •  
      • Add support for protocols 47 (GRE) and 59 (IPv6-NoNxt).
        •  
      • Add support for field additional_asns in optional information column.
        •  
      • intelmq.bots.parsers.microsoft.parser_ctip:
        •  
      • Fix mapping of DestinationIpInfo.DestinationIpConnectionType field (contained a typo).
        •  
      • Explicitly ignore field DestinationIpInfo.DestinationIpv4Int as the data is already in another field.
        •  
      • intelmq.bots.parsers.generic.parser_csv:
        •  
      • Ignore line having spaces or tabs only or comment having leading tabs or spaces (PR#1669 by Brajneesh).
        •  
      • Data fields containing - are now ignored and do not raise an exception anymore (#1651, PR#74 by Sebastian Waldbauer).
        •  
      "},{"location":"changelog/#experts_12","title":"Experts","text":"
         
      • intelmq.bots.experts.taxonomy.expert: Map type scanner to information-gathering instead of information gathering. See NEWS file for more information.
        •  
      "},{"location":"changelog/#tests_10","title":"Tests","text":"
         
      • Travis: Deactivate tests with optional requirements on Python 3.5, as the build fails because of abusix/querycontacts version conflicts on dnspython.
        •  
      "},{"location":"changelog/#known-issues_11","title":"Known issues","text":"
         
      • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952).
        •  
      • Corrupt dump files when interrupted during writing (#870).
        •  
      "},{"location":"changelog/#222-2020-10-28","title":"2.2.2 (2020-10-28)","text":""},{"location":"changelog/#core_13","title":"Core","text":"
         
      • intelmq.lib.upgrades:
        •  
      • Add upgrade function for renamed Shadowserver feed name \"Blacklisted-IP\"/\"Blocklist\".
        •  
      "},{"location":"changelog/#bots_14","title":"Bots","text":""},{"location":"changelog/#parsers_12","title":"Parsers","text":"
         
      • intelmq.bots.parsers.shadowserver:
        •  
      • Rename \"Blacklisted-IP\" feed to \"Blocklist\", old name is still valid until IntelMQ version 3.0 (PR#1588 by Thomas Hungenberg).
        •  
      • Added support for the feeds Accessible Radmin and CAIDA IP Spoofer (PR#1600 by sinus-x).
        •  
      • intelmq.bots.parsers.anubisnetworks.parser: Fix parsing error where dst.ip was not equal to comm.http.host.
        •  
      • intelmq/bots/parsers/danger_rulez/parser: correctly skip malformed rows by defining variables before referencing (PR#1601 by Tomas Bellus).
        •  
      • `intelmq.bots.parsers.misp.parser: Fix MISP Event URL (#1619, PR#1618 by Nedfire23).
        •  
      • intelmq.bots.parsers.microsoft.parser_ctip:
        •  
      • Add support for DestinationIpInfo.* and Signatures.Sha256 fields, used by the ctip-c2 feed (PR#1623 by Mikk Margus M\u00f6ll).
        •  
      • Use extra.payload.text for the feed's field Payload if the content cannot be decoded (PR#1610 by Giedrius Ramas).
        •  
      "},{"location":"changelog/#experts_13","title":"Experts","text":"
         
      • intelmq.bots.experts.cymru_whois:
        •  
      • Fix cache key calculation which previously led to duplicate keys and therefore wrong results in rare cases. The cache key calculation is intentionally not backwards-compatible (#1592, PR#1606).
        •  
      • The bot now caches and logs (as level INFO) empty responses from Cymru (PR#1606).
        •  
      "},{"location":"changelog/#documentation_13","title":"Documentation","text":"
         
      • README:
        •  
      • Add Core Infrastructure Initiative Best Practices Badge.
        •  
      • Bots:
        •  
      • Generic CSV Parser: Add note on escaping backslashes (#1579).
        •  
      • Remove section of non-existing \"Copy Extra\" Bot.
        •  
      • Explain taxonomy expert.
        •  
      • Add documentation on n6 parser.
        •  
      • Gethostbyname expert: Add documentation how errors are treated.
        •  
      • Feeds:
        •  
      • Fixed bot modules of Calidog CertStream feed.
        •  
      • Add information on Microsoft CTIP C2 feed.
        •  
      "},{"location":"changelog/#packaging_6","title":"Packaging","text":"
         
      • In Debian packages, intelmqctl check and intelmqctl upgrade-config are executed in the \"postinst\" step (#1551, PR#1624 by Birger Schacht).
        •  
      • Require requests<2.26 for Python 3.5, as 2.25.x will be the last release series of the requests library with support for Python 3.5.
        •  
      "},{"location":"changelog/#tests_11","title":"Tests","text":"
         
      • intelmq.tests.lib.test_pipeline: Skip TestAmqp.test_acknowledge on Travis with Python 3.8.
        •  
      • intelmq.tests.bots.outputs.elasticsearch.test_output: Refresh index intelmq manually to fix random test failures (#1593, PR#1595 by Zach Stone).
        •  
      "},{"location":"changelog/#tools_9","title":"Tools","text":"
         
      • intelmqctl check:
        •  
      • For disabled bots which do not have any pipeline connections, do not raise an error, but only warning.
        •  
      • Fix check on source/destination queues for bots as well the orphaned queues.
        •  
      "},{"location":"changelog/#contrib_4","title":"Contrib","text":"
         
      • Bash completion scripts: Check both /opt/intelmq/ as well as LSB-paths (/etc/intelmq/ and /var/log/intelmq/) for loading bot information (#1561, PR#1628 by Birger Schacht).
        •  
      "},{"location":"changelog/#known-issues_12","title":"Known issues","text":"
         
      • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952).
        •  
      • Corrupt dump files when interrupted during writing (#870).
        •  
      "},{"location":"changelog/#221-2020-07-30","title":"2.2.1 (2020-07-30)","text":""},{"location":"changelog/#core_14","title":"Core","text":"
         
      • intelmq.lib.upgrades:
        •  
      • Add upgrade function for changed configuration of the feed \"Abuse.ch URLhaus\" (#1571, PR#1572 by Filip Pokorn\u00fd).
        •  
      • Add upgrade function for removal of HPHosts Hosts file feed and intelmq.bots.parsers.hphosts parser (#1559).
        •  
      • intelmq.lib.harmonization:
           
        • For IP Addresses, explicitly reject IPv6 addresses with scope ID (due to changed behavior in Python 3.9, #1550).
          •  
         
        •  
      "},{"location":"changelog/#development_6","title":"Development","text":"
         
      • Ignore line length (E501) in code-style checks altogether.
        •  
      "},{"location":"changelog/#bots_15","title":"Bots","text":""},{"location":"changelog/#collectors_11","title":"Collectors","text":"
         
      • intelmq.bots.collectors.misp: Fix access to actual MISP object (PR#1548 by Tomas Bellus @tomas321)
        •  
      • intelmq.bots.collectors.stomp: Remove empty client.pem file.
        •  
      "},{"location":"changelog/#parsers_13","title":"Parsers","text":"
         
      • intelmq.bots.parsers.shadowserver.config:
        •  
      • Add support for Accessible-CoAP feed (PR #1555 by Thomas Hungenberg).
        •  
      • Add support for Accessible-ARD feed (PR #1584 by Tomas Bellus @tomas321).
        •  
      • intelmq.bots.parser.anubisnetworks.parser: Ignore \"TestSinkholingLoss\" events, these are not intended to be sent out at all.
        •  
      • intelmq.bots.parsers.generic.parser_csv: Allow values of type dictionary for parameter type_translation.
        •  
      • intelmq.bots.parsers.hphosts: Removed, feed is unavailable (#1559).
        •  
      • intelmq.bots.parsers.cymru.parser_cap_program: Add support for comment \"username\" for \"scanner\" category.
        •  
      • intelmq.bots.parsers.malwareurl.parser: Check for valid FQDN and IP address in URL and IP address columns (PR#1585 by Marius Urkis).
        •  
      "},{"location":"changelog/#experts_14","title":"Experts","text":"
         
      • intelmq.bots.experts.maxmind_geoip: On Python < 3.6, require maxminddb < 2, as that version does no longer support Python 3.5.
        •  
      "},{"location":"changelog/#outputs_10","title":"Outputs","text":"
         
      • intelmq.bots.outputs.udp: Fix error handling on sending, had a bug itself.
        •  
      "},{"location":"changelog/#documentation_14","title":"Documentation","text":"
         
      • Feeds:
        •  
      • Update documentation of feed \"Abuse.ch URLhaus\" (#1571, PR#1572 by Filip Pokorn\u00fd).
        •  
      • Bots:
        •  
      • Overhaul of all bots' description fields (#1570).
        •  
      • User-Guide:
        •  
      • Overhaul pipeline configuration section and explain named queues better (#1577).
        •  
      "},{"location":"changelog/#tests_12","title":"Tests","text":"
         
      • intelmq.tests.bots.experts.cymru: Adapt test_empty_result, remove test_unicode_as_name and test_country_question_mark (#1576).
        •  
      "},{"location":"changelog/#tools_10","title":"Tools","text":"
         
      • intelmq.bin.intelmq_gen_docs: Format parameters of types lists with double quotes around values to produce conform JSON, ready to copy and paste the value into the IntelMQ Manager's bot parameter form.
        •  
      • intelmq.bin.intelmqctl:
        •  
      • debug: In JSON mode, use dictionaries instead of lists.
        •  
      • debug: Add PATH to the paths shown.
        •  
      • check: Show $PATH environment variable if executable cannot be found.
        •  
      "},{"location":"changelog/#contrib_5","title":"Contrib","text":"
         
      • malware_name_mapping: Change MISP Threat Actors URL to new URL (branch master -> main) in download script.
        •  
      "},{"location":"changelog/#known-issues_13","title":"Known issues","text":"
         
      • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952).
        •  
      • Corrupt dump files when interrupted during writing (#870).
        •  
      • Bash completion scripts search in wrong directory in packages (#1561).
        •  
      • Cymru Expert: Wrong Cache-Key Calculation (#1592).
        •  
      "},{"location":"changelog/#220-2020-06-18","title":"2.2.0 (2020-06-18)","text":"
      Dropped support for Python 3.4.
      "},{"location":"changelog/#core_15","title":"Core","text":"
         
      • __init__: Changes to the path-handling, see User Guide, section /opt and LSB paths for more information
        •  
      • The environment variable INTELMQ_ROOT_DIR can be used to set custom root directories instead of /opt/intelmq/ (#805) in case of non LSB-path installations.
        •  
      • The environment variable ROOT_DIR can be used to set custom root directories instead of / (#805) in case of LSB-path installations.
        •  
      • intelmq.lib.exceptions: Added MissingDependencyError for show error messages about a missing library and how to install it (#1471).
        •  
      • Added optional parameter installed to show the installed version.
        •  
      • Added optional parameter additional_text to show arbitrary text.
        •  
      • Adding more type annotations for core libraries.
        •  
      • intelmq.lib.pipeline.Pythonlist.sleep: Drop deprecated method.
        •  
      • intelmq.lib.utils: write_configuration: Append a newline at end of configuration/file to allow proper comparisons & diffs.
        •  
      • intelmq.lib.test: BotTestCase drops privileges upon initialization (#1489).
        •  
      • intelmq.lib.bot:
        •  
      • New class OutputBot:
           
        • Method export_event to format/export events according to the parameters given by the user.
          •  
         
        •  
      • ParserBot: New methods parse_json_stream and recover_line_json_stream.
        •  
      • ParserBot.recover_line_json: Fix format by adding a list around the line data.
        •  
      • Bot.send_message: In debugging log level, the path to which the message is sent is now logged too.
        •  
      "},{"location":"changelog/#bots_16","title":"Bots","text":"
         
      • Bots with dependencies: Use of intelmq.lib.exceptions.MissingDependencyError.
        •  
      "},{"location":"changelog/#collectors_12","title":"Collectors","text":"
         
      • intelmq.bots.collectors.misp.collector: Deprecate parameter misp_verify in favor of generic parameter http_verify_cert.
        •  
      • intelmq.bots.collectors.tcp.collector: Drop compatibility with Python 3.4.
        •  
      • intelmq.bots.collectors.stomp.collector:
        •  
      • Check the stomp.py version and show an error message if it does not match.
        •  
      • For stomp.py versions >= 5.0.0 redirect the stomp.PrintingListener output to debug logging.
        •  
      • intelmq.bots.collectors.microsoft.collector_azure: Support current Python library azure-storage-blob>= 12.0.0, configuration is incompatible and needs manual change. See NEWS file and bot's documentation for more details.
        •  
      • intelmq.bots.collectors.amqp.collector_amqp: Require pika minimum version 1.0.
        •  
      • intelmq.bots.collectors.github_api.collector_github_contents_api: Added (PR#1481).
        •  
      "},{"location":"changelog/#parsers_14","title":"Parsers","text":"
         
      • intelmq.bots.parsers.autoshun.parser: Drop compatibility with Python 3.4.
        •  
      • intelmq.bots.parsers.html_table.parser: Drop compatibility with Python 3.4.
        •  
      • intelmq.bots.parsers.shadowserver.parser: Add support for MQTT and Open-IPP feeds (PR#1512, PR#1544).
        •  
      • intelmq.bots.parsers.taichung.parser:
        •  
      • Migrate to ParserBot.
        •  
      • Also parse geolocation information if available.
        •  
      • intelmq.bots.parsers.cymru.parser_full_bogons:
        •  
      • Migrate to ParserBot.
        •  
      • Add last updated information in raw.
        •  
      • intelmq.bots.parsers.anubisnetworks.parser: Add new parameter use_malware_familiy_as_classification_identifier.
        •  
      • intelmq.bots.parsers.microsoft.parser_ctip: Compatibility for new CTIP data format used provided by the Azure interface.
        •  
      • intelmq.bots.parsers.cymru.parser_cap_program: Support for openresolver type.
        •  
      • intelmq.bots.parsers.github_feed.parser: Added (PR#1481).
        •  
      • intelmq.bots.parsers.urlvir.parser: Removed, as the feed is discontinued (#1537).
        •  
      "},{"location":"changelog/#experts_15","title":"Experts","text":"
         
      • intelmq.bots.experts.csv_converter: Added as converter to CSV.
        •  
      • intelmq.bots.experts.misp: Added (PR#1475).
        •  
      • intelmq.bots.experts.modify: New parameter maximum_matches.
        •  
      "},{"location":"changelog/#outputs_11","title":"Outputs","text":"
         
      • intelmq.bots.outputs.amqptopic:
        •  
      • Use OutputBot and export_event.
        •  
      • Allow formatting the routing key with event data by the new parameter format_routing_key (boolean).
        •  
      • intelmq.bots.outputs.file: Use OutputBot and export_event.
        •  
      • intelmq.bots.outputs.files: Use OutputBot and export_event.
        •  
      • intelmq.bots.outputs.misp.output_feed: Added, creates a MISP Feed (PR#1473).
        •  
      • intelmq.bots.outputs.misp.output_api: Added, pushes to MISP via the API (PR#1506, PR#1536).
        •  
      • intelmq.bots.outputs.elasticsearch.output: Dropped ElasticSearch version 5 compatibility, added version 7 compatibility (#1513).
        •  
      "},{"location":"changelog/#documentation_15","title":"Documentation","text":"
         
      • Document usage of the INTELMQ_ROOT_DIR environment variable.
        •  
      • Added document on MISP integration possibilities.
        •  
      • Feeds:
        •  
      • Added \"Full Bogons IPv6\" feed.
        •  
      • Remove discontinued URLVir Feeds (#1537).
        •  
      "},{"location":"changelog/#packaging_7","title":"Packaging","text":"
         
      • setup.py do not try to install any data to /opt/intelmq/ as the behavior is inconsistent on various systems and with intelmqsetup we have a tool to create the structure and files anyway.
        •  
      • debian/rules:
        •  
      • Provide a blank state file in the package.
        •  
      • Patches:
        •  
      • Updated fix-intelmq-paths.patch.
        •  
      "},{"location":"changelog/#tests_13","title":"Tests","text":"
         
      • Travis: Use intelmqsetup here too.
        •  
      • Install required build dependencies for the Debian package build test.
        •  
      • This version is no longer automatically tested on Python < 3.5.
        •  
      • Also run the tests on Python 3.8.
        •  
      • Run the Debian packaging tests on Python 3.5 and the code-style test on 3.8.
        •  
      • Added tests for the new bot intelmq.bots.outputs.misp.output_feed (#1473).
        •  
      • Added tests for the new bot intelmq.bots.experts.misp.expert (#1473).
        •  
      • Added tests for intelmq.lib.exceptions.
        •  
      • Added tests for intelmq.lib.bot.OutputBot and intelmq.lib.bot.OutputBot.export_event.
        •  
      • Added IPv6 tests for intelmq.bots.parsers.cymru.parser_full_bogons.
        •  
      • Added tests for intelmq.lib.bot.ParserBot's new methods parse_json_stream and recover_line_json_stream.
        •  
      • intelmq.tests.test_conf: Set encoding to UTF-8 for reading the feeds.yaml file.
        •  
      "},{"location":"changelog/#tools_11","title":"Tools","text":"
         
      • intelmqctl:
        •  
      • upgrade-config:
           
        • Allow setting the state file location with the --state-file parameter.
          •  
        • Do not require a second run anymore, if the state file is newly created (#1491).
          •  
        • New parameter no_backup/--no-backup to skip creation of .bak files for state and configuration files.
          •  
         
        •  
      • Only require psutil for the IntelMQProcessManager, not for process manager independent calls like upgrade-config or check.
        •  
      • Add new command debug to output some information for debugging. Currently implemented:
           
        • paths
          •  
        • environment variables
          •  
         
        •  
      • IntelMQController: New argument --no-file-logging to disable logging to file.
        •  
      • If dropping privileges does not work, intelmqctl will now abort (#1489).
        •  
      • intelmqsetup:
        •  
      • Add argument parsing and an option to skip setting file ownership, possibly not requiring root permissions.
        •  
      • Call intelmqctl upgrade-config and add argument for the state file path (#1491).
        •  
      • intelmq_generate_misp_objects_templates.py: Tool to create a MISP object template (#1470).
        •  
      • intelmqdump: New parameter -t or --truncate to optionally give the maximum length of raw data to show, 0 for no truncating.
        •  
      "},{"location":"changelog/#contrib_6","title":"Contrib","text":"
         
      • Added development-tools.
        •  
      • ElasticSearch: Dropped version 5 compatibility, added version 7 compatibility (#1513).
        •  
      • Malware Name Mapping Downloader:
        •  
      • New parameter --mwnmp-ignore-adware.
        •  
      • The parameter --add-default supports an optional parameter to define the default value.
        •  
      "},{"location":"changelog/#known-issues_14","title":"Known issues","text":"
         
      • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952).
        •  
      • Corrupt dump files when interrupted during writing (#870).
        •  
      "},{"location":"changelog/#213-2020-05-26","title":"2.1.3 (2020-05-26)","text":""},{"location":"changelog/#requirements","title":"Requirements","text":"
         
      • The python library requests is (again) listed as dependency of the core (#1519).
        •  
      "},{"location":"changelog/#core_16","title":"Core","text":"
         
      • intelmq.lib.upgrades:
        •  
      • Harmonization upgrade: Also check and update regular expressions.
        •  
      • Add function to migrate the deprecated parameter attach_unzip to extract_files for the mail attachment collector.
        •  
      • Add function to migrate changed Taichung URL feed.
        •  
      • Check for discontinued Abuse.CH Zeus Tracker feed.
        •  
      • intelmq.lib.bot:
        •  
      • ParserBot.recover_line: Parameter line needs to be optional, fix usage of fallback value self.current_line.
        •  
      • start: Handle decoding errors in the pipeline different so that the bot is not stuck in an endless loop (#1494).
        •  
      • start: Only acknowledge a message in case of errors, if we actually had a message to dump, which is not the case for collectors.
        •  
      • _dump_message: Dump messages with encoding errors base64 encoded, not in JSON format as it's not possible to decode them (#1494).
        •  
      • intelmq.lib.test:
        •  
      • BotTestCase.run_bot: Add parameters allowed_error_count and allowed_warning_count to allow set the number per run, not per test class.
        •  
      • Set source_pipeline_broker and destination_pipeline_broker to pythonlist instead of the old broker, fixes intelmq.tests.lib.test_bot.TestBot.test_pipeline_raising.
        •  
      • Fix test for (allowed) errors and warnings.
        •  
      • intelmq.lib.exceptions:
        •  
      • InvalidKey: Add KeyError as parent class.
        •  
      • DecodingError: Added, string representation has all relevant information on the decoding error, including encoding, reason and the affected string (#1494).
        •  
      • intelmq.lib.pipeline:
        •  
      • Decode messages in Pipeline.receive not in the implementation's _receive so that the internal counter is correct in case of decoding errors (#1494).
        •  
      • intelmq.lib.utils:
        •  
      • decode: Raise new DecodingError if decoding fails.
        •  
      "},{"location":"changelog/#harmonization_1","title":"Harmonization","text":"
         
      • protocol.transport: Adapt regular expression to allow the value nvp-ii (protocol 11).
        •  
      "},{"location":"changelog/#bots_17","title":"Bots","text":""},{"location":"changelog/#collectors_13","title":"Collectors","text":"
         
      • intelmq.bots.collectors.mail.collector_mail_attach:
        •  
      • Fix handling of deprecated parameter name attach_unzip.
        •  
      • Fix handling of attachments without filenames (#1538).
        •  
      • intelmq.bots.collectors.stomp.collector: Fix compatibility with stomp.py versions > 4.1.20 and catch errors on shutdown.
        •  
      • intelmq.bots.collectors.microsoft:
        •  
      • Update REQUIREMENTS.txt temporarily fixing deprecated Azure library (#1530, PR#1532).
        •  
      • intelmq.bots.collectors.microsoft.collector_interflow: Add method for printing the file list.
        •  
      "},{"location":"changelog/#parsers_15","title":"Parsers","text":"
         
      • intelmq.bots.parsers.cymru.parser_cap_program: Support for protocol 11 (nvp-ii) and conficker type.
        •  
      • intelmq.bots.parsers.taichung.parser: Support more types/classifications:
        •  
      • Application Compromise: Apache vulnerability & SQL injections
        •  
      • Brute-force: MSSQL & SSH password guess attacks; Office 365, SSH & SIP attacks
        •  
      • C2 Sever: Attack controller
        •  
      • DDoS
        •  
      • DoS: DNS, DoS, Excess connection
        •  
      • IDS Alert / known vulnerability exploitation: backdoor
        •  
      • Malware: Malware Proxy
        •  
      • Warn on new unknown types.
        •  
      • intelmq.bots.parsers.bitcash.parser: Removed as feed is discontinued.
        •  
      • intelmq.bots.parsers.fraunhofer.parser_ddosattack_cnc and intelmq.bots.parsers.fraunhofer.parser_ddosattack_target: Removed as feed is discontinued.
        •  
      • intelmq.bots.parsers.malwaredomains.parser: Correctly classify C&C and phishing events.
        •  
      • intelmq.bots.parsers.shadowserver.parser: More verbose error message for missing report specification (#1507).
        •  
      • intelmq.bots.parsers.n6.parser_n6stomp: Always add n6 field name as malware.name independent of category.
        •  
      • intelmq.bots.parsers.anubisnetworks: Update parser with new data format.
        •  
      • intelmq.bots.parsers.bambenek: Add new feed URLs with Host faf.bambenekconsulting.com (#1525, PR#1526).
        •  
      • intelmq.bots.parsers.abusech.parser_ransomware: Removed, as the feed is discontinued (#1537).
        •  
      • intelmq.bots.parsers.nothink.parser: Removed, as the feed is discontinued (#1537).
        •  
      • intelmq.bots.parsers.n6.parser: Remove not allowed characters in the name field for malware.name and write original value to event_description.text instead.
        •  
      "},{"location":"changelog/#experts_16","title":"Experts","text":"
         
      • intelmq.bots.experts.cymru_whois.lib: Fix parsing of AS names with Unicode characters.
        •  
      "},{"location":"changelog/#outputs_12","title":"Outputs","text":"
         
      • intelmq.bots.outputs.mongodb:
        •  
      • Set default port 27017.
        •  
      • Use different authentication mechanisms per MongoDB server version to fix compatibility with server version >= 3.4 (#1439).
        •  
      "},{"location":"changelog/#documentation_16","title":"Documentation","text":"
         
      • Feeds:
        •  
      • Remove unavailable feed Abuse.CH Zeus Tracker.
        •  
      • Remove the field status, offline feeds should be removed.
        •  
      • Add a new field public to differentiate between private and public feeds.
        •  
      • Adding documentation URLs to nearly all feeds.
        •  
      • Remove unavailable Bitcash.cz feed.
        •  
      • Remove unavailable Fraunhofer DDos Attack feeds.
        •  
      • Remove unavailable feed Abuse.CH Ransomware Tracker (#1537).
        •  
      • Update information on Bambenek Feeds, many require a license now (#1525).
        •  
      • Remove discontinued Nothink Honeypot Feeds (#1537).
        •  
      • Developers Guide: Fix the instructions for /opt/intelmq file permissions.
        •  
      "},{"location":"changelog/#packaging_8","title":"Packaging","text":"
         
      • Patches: fix-logrotate-path.patch: also include path to rotated file in patch.
        •  
      • Fix paths from /opt to LSB for setup.py and contrib/logrotate/intelmq in build process (#1500).
        •  
      • Add runtime dependency debianutils for the program which, which is required for intelmqctl.
        •  
      "},{"location":"changelog/#tests_14","title":"Tests","text":"
         
      • Dropping Travis tests for 3.4 as required libraries dropped 3.4 support.
        •  
      • intelmq.tests.bots.experts.cymru_whois:
        •  
      • Drop missing ASN test, does not work anymore.
        •  
      • IPv6 to IPv4 test: Test for two possible results.
        •  
      • intelmq.lib.test: Fix compatibility of logging capture with Python >= 3.7 by reworking the whole process (#1342).
        •  
      • intelmq.bots.collectors.tcp.test_collector: Removing custom mocking and bot starting, not necessary anymore.
        •  
      • Added tests for intelmq.bin.intelmqctl.IntelMQProcessManager._interpret_commandline.
        •  
      • Fix and split tests.bots.experts.ripe.test_expert.test_ripe_stat_error_json.
        •  
      • Added tests for invalid encodings in input messages in intelmq.tests.lib.test_bot and intelmq.tests.lib.test_pipeline (#1494).
        •  
      • Travis: Explicitly enable RabbitMQ management plugin.
        •  
      • intelmq.tests.lib.test_message: Fix usage of the parameter blacklist for Message hash tests (#1539).
        •  
      "},{"location":"changelog/#tools_12","title":"Tools","text":"
         
      • intelmqsetup: Copy missing BOTS file to IntelMQ's root directory (#1498).
        •  
      • intelmq_gen_docs: Feed documentation generation: Handle missing/empty parameters.
        •  
      • intelmqctl:
        •  
      • IntelMQProcessManager: For the status of running bots also check the bot ID of the commandline and ignore the path of the executable (#1492).
        •  
      • IntelMQController: Fix exit codes of check command for JSON output (now 0 on success and 1 on error, was swapped, #1520).
        •  
      • intelmqdump:
        •  
      • Handle base64-type messages for show, editor and recovery actions.
        •  
      "},{"location":"changelog/#contrib_7","title":"Contrib","text":"
         
      • intelmq/bots/experts/asn_lookup/update-asn-data: Use pyasn_util_download.py to download the data instead from RIPE, which cannot be parsed currently (#1517, PR#1518, https://github.com/hadiasghari/pyasn/issues/62).
        •  
      "},{"location":"changelog/#known-issues_15","title":"Known issues","text":"
         
      • HTTP stream collector: retry on regular connection problems? (#1435).
        •  
      • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952).
        •  
      • Reverse DNS: Only first record is used (#877).
        •  
      • Corrupt dump files when interrupted during writing (#870).
        •  
      "},{"location":"changelog/#212-2020-01-28","title":"2.1.2 (2020-01-28)","text":""},{"location":"changelog/#core_17","title":"Core","text":"
         
      • __init__: Resolve absolute path for STATE_FILE_PATH variable (resolves ..).
        •  
      • intelmq.lib.utils:
        •  
      • log: Do not raise an exception if logging to neither file nor syslog is requested.
        •  
      • logging StreamHandler: Colorize all warning and error messages red.
        •  
      • logging FileHandler: Strip all shell colorizations from the messages (#1436).
        •  
      • intelmq.lib.message:
        •  
      • Message.to_json: Set sort_keys=True to get reproducible results.
        •  
      • drop_privileges: Handle situations where the user or group intelmq does not exist.
        •  
      • intelmq.lib.pipeline:
        •  
      • Amqp._send and Amqp._acknowledge: Log traceback in debug mode in case of errors and necessary re-connections.
        •  
      • Amqp._acknowledge: Reset delivery tag if acknowledge was successful.
        •  
      "},{"location":"changelog/#bots_18","title":"Bots","text":""},{"location":"changelog/#collectors_14","title":"Collectors","text":"
         
      • intelmq.bots.collectors.misp.collector:
        •  
      • Add compatibility with current pymisp versions and versions released after January 2020 (PR #1468).
        •  
      "},{"location":"changelog/#parsers_16","title":"Parsers","text":"
         
      • intelmq.bots.parsers.shadowserver.config: Add some missing fields for the feed accessible-rdp (#1463).
        •  
      • intelmq.bots.parsers.shadowserver.parser:
        •  
      • Feed-detection based on file names: The prefixed date is optional now.
        •  
      • Feed-detection based on file names: Re-detect feed for every report received (#1493).
        •  
      "},{"location":"changelog/#experts_17","title":"Experts","text":"
         
      • intelmq.bots.experts.national_cert_contact_certat: Handle empty responses by server (#1467).
        •  
      • intelmq.bots.experts.maxmind_geoip: The script update-geoip-data now requires a license key as second parameter because of upstream changes (#1484)).
        •  
      "},{"location":"changelog/#outputs_13","title":"Outputs","text":"
         
      • intelmq.bots.outputs.restapi.output: Fix logging of response body if response status code was not ok.
        •  
      "},{"location":"changelog/#documentation_17","title":"Documentation","text":"
         
      • Remove some hardcoded /opt/intelmq/ paths from code comments and program outputs.
        •  
      "},{"location":"changelog/#packaging_9","title":"Packaging","text":"
         
      • debian/rules: Only replace /opt/intelmq/ with LSB-paths in some certain files, not the whole tree, avoiding wrong replacements.
        •  
      • debian/rules and debian/intelmq.install: Do install the examples configuration directly instead of working around the abandoned examples directory.
        •  
      "},{"location":"changelog/#tests_15","title":"Tests","text":"
         
      • lib/test_utils: Skip some tests on Python 3.4 because contextlib.redirect_stdout and contextlib.redirect_sterr are not supported on this version.
        •  
      • Travis: Stop running tests with all optional dependencies on Python 3.4, as more and more libraries are dropping support for it. Tests on the core and code without non-optional requirements are not affected.
        •  
      • tests.bots.parsers.html_table: Make tests independent of current year.
        •  
      "},{"location":"changelog/#tools_13","title":"Tools","text":"
         
      • intelmqctl upgrade-config: Fix missing substitution in error message \"State file %r is not writable.\".
        •  
      "},{"location":"changelog/#known-issues_16","title":"Known issues","text":"
         
      • bots trapped in endless loop if decoding of raw message fails (#1494)
        •  
      • intelmqctl status of processes: need to check bot id too (#1492)
        •  
      • MongoDB authentication: compatibility on different MongoDB and pymongo versions (#1439)
        •  
      • ctl: shell colorizations are logged (#1436)
        •  
      • http stream collector: retry on regular connection problems? (#1435)
        •  
      • tests: capture logging with context manager (#1342)
        •  
      • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
        •  
      • n6 parser: mapping is modified within each run (#905)
        •  
      • reverse DNS: Only first record is used (#877)
        •  
      • Corrupt dump files when interrupted during writing (#870)
        •  
      "},{"location":"changelog/#211-2019-11-11","title":"2.1.1 (2019-11-11)","text":""},{"location":"changelog/#configuration_6","title":"Configuration","text":"
         
      • Default configuration:
        •  
      • Remove discontinued feed \"Feodo Tracker Domains\" from default configuration.
        •  
      • Add \"Feodo Tracker Browse\" feed to default configuration.
        •  
      "},{"location":"changelog/#core_18","title":"Core","text":"
         
      • intelmq.lib.pipeline: AMQP: using port 15672 as default (like RabbitMQ's defaults) for the monitoring interface for getting statistical data (intelmqctl_rabbitmq_monitoring_url).
        •  
      • intelmq.lib.upgrades: Added a generic upgrade function for harmonization, checking of all message types, it's fields and their types.
        •  
      • intelmq.lib.utils:
        •  
      • TimeoutHTTPAdapter: A subclass of requests.adapters.HTTPAdapter with the possibility to set the timeout per adapter.
        •  
      • create_request_session_from_bot: Use the TimeoutHTTPAdapter with the user-defined timeout. Previously the timeout was not functional.
        •  
      "},{"location":"changelog/#bots_19","title":"Bots","text":""},{"location":"changelog/#parsers_17","title":"Parsers","text":"
         
      • intelmq.bots.parsers.shadowserver.parser: Fix logging message if the parameter feedname is not present.
        •  
      • intelmq.bots.parsers.shodan.parser: Also add field classification.identifier ('network-scan') in minimal mode.
        •  
      • intelmq.bots.parsers.spamhaus.parser_cert: Add support for category 'misc'.
        •  
      • intelmq.bots.parsers.cymru.parser_cap_program:
        •  
      • Add support for phishing events without URL.
        •  
      • Add support for protocols >= 143 (unassigned, experiments, testing, reserved), saving the number to extra, as the data would be bogus.
        •  
      • intelmq.bots.parsers.microsoft.parser_bingmurls:
        •  
      • Save the Tags data as source.geolocation.cc.
        •  
      "},{"location":"changelog/#experts_18","title":"Experts","text":"
         
      • intelmq.bots.experts.modify.expert: Fix bug with setting non-string values (#1460).
        •  
      "},{"location":"changelog/#outputs_14","title":"Outputs","text":"
         
      • intelmq.bots.outputs.smtp:
        •  
      • Allow non-existent field in text formatting by using a default value None instead of throwing errors.
        •  
      • Fix Authentication (#1464).
        •  
      • Fix sending to multiple recipients (#1464).
        •  
      "},{"location":"changelog/#documentation_18","title":"Documentation","text":"
         
      • Feeds:
        •  
      • Fix configuration of Feodo Tracker Browse feed.
        •  
      • Bots:
        •  
      • Sieve expert: Document behavior of != with lists.
        •  
      "},{"location":"changelog/#tests_16","title":"Tests","text":"
         
      • Adaption and extension of the test cases to the changes.
        •  
      "},{"location":"changelog/#tools_14","title":"Tools","text":"
         
      • intelmq.bin.intelmqctl:
        •  
      • check: Check if running the upgrade function for harmonization is necessary.
        •  
      • upgrade-config: Run the upgrade function for harmonization.
        •  
      • intelmqctl restart did throw an error as the message for restarting was not defined (#1465).
        •  
      "},{"location":"changelog/#known-issues_17","title":"Known issues","text":"
         
      • MongoDB authentication: compatibility on different MongoDB and pymongo versions (#1439)
        •  
      • ctl: shell colorizations are logged (#1436)
        •  
      • http stream collector: retry on regular connection problems? (#1435)
        •  
      • tests: capture logging with context manager (#1342)
        •  
      • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
        •  
      • n6 parser: mapping is modified within each run (#905)
        •  
      • reverse DNS: Only first record is used (#877)
        •  
      • Corrupt dump files when interrupted during writing (#870)
        •  
      "},{"location":"changelog/#210-2019-10-15","title":"2.1.0 (2019-10-15)","text":""},{"location":"changelog/#core_19","title":"Core","text":"
         
      • intelmq.lib.harmonization:
        •  
      • Use correct parent classes.
        •  
      • Add DateTime.convert as interface for all existing conversion functions.
        •  
      • add DateTime.convert_from_format.
        •  
      • add DateTime.convert_from_format_midnight.
        •  
      • add DateTime.convert_fuzzy.
        •  
      • intelmq.lib.pipeline:
        •  
      • Redis: Use single connection client if calling bot is not multithreaded. Gives a small speed advantage.
        •  
      • Require the bot instance as parameter for all pipeline classes.
        •  
      • New internal variable _has_message to keep the state of the pipeline.
        •  
      • Split receive and acknowledge into public-facing and private methods.
        •  
      • Add reject_message method to the Pipeline class for explicit re-queue of messages.
        •  
      • AMQP:
           
        • Make exchange configurable.
          •  
        • If exchange is set, the queues are not declared, the queue name is for routing used by exchanges.
          •  
         
        •  
      • intelmq.lib.bot:
        •  
      • Log message after successful bot initialization, no log message anymore for ready pipeline.
        •  
      • Use existing current message if receive is called and the current message still exists.
        •  
      • Fix handling of received messaged after a SIGHUP that happened during a blocking receiving connection using explicit rejection (#1438).
        •  
      • New method _parse_common_parameters called before init to parse commonly used argument. Currently supported: extract_files.
        •  
      • intelmq.lib.test:
        •  
      • Fix the tests broker by providing the testing pipeline.
        •  
      • intelmq.lib.utils:
        •  
      • unzip:
           
        • new parameter return_names to optionally return the file names.
          •  
        • support for zip
          •  
        • new parameters try_zip, try_gzip and try_tar to control which compressions are tried.
          •  
        • rewritten to an iterative approach
          •  
         
        •  
      • add file_name_from_response to extract a file name from a Response object for downloaded files.
        •  
      • intelmq.lib.upgrades: Added v210_deprecations for deprecated parameters.
        •  
      "},{"location":"changelog/#harmonization_2","title":"Harmonization","text":"
         
      • Add extra to reports.
        •  
      "},{"location":"changelog/#bots_20","title":"Bots","text":""},{"location":"changelog/#collectors_15","title":"Collectors","text":"
         
      • intelmq.bots.collectors.http.collector_http:
        •  
      • More extensive usage of intelmq.lib.utils.unzip.
        •  
      • Save the file names in the report if files have been extracted form an archive.
        •  
      • intelmq.bots.collectors.rt.collector_rt:
        •  
      • Save ticket information/metadata in the extra fields of the report.
        •  
      • Support for RT 3.8 and RT 4.4.
        •  
      • New parameters extract_attachment and extract_download for generic archive extraction and consistency. The parameter unzip_attachment is deprecated.
        •  
      • intelmq.bots.collectors.mail.*: Save email information/metadata in the extra fields of the report. See the bots documentation for a complete list of provided data.
        •  
      • intelmq.bots.collectors.mail.collector_mail_attach: Check for existence/validity of the attach_regex parameter.
        •  
      • Use the lib's unzip function for uncompressing attachments and use the .
        •  
      • intelmq.bots.collectors.mail.collector_mail_url: Save the file name of the downloaded file as extra.file_name.
        •  
      • intelmq.bots.collectors.amqp.collector_amqp: New collector to collect data from (remote) AMQP servers, for bot IntelMQ as well as external data.
        •  
      • use default SSL context for client purposes, fixes compatibility with python < 3.6 if TLS is used.
        •  
      "},{"location":"changelog/#parsers_18","title":"Parsers","text":"
         
      • intelmq.bots.parsers.html_table.parser:
        •  
      • New parameter \"html_parser\".
        •  
      • Use time conversion functions directly from intelmq.lib.harmonization.DateTime.convert.
        •  
      • Limit lxml dependency on 3.4 to < 4.4.0 (incompatibility).
        •  
      • intelmq.bots.parsers.netlab_360.parser: Add support for hajime scanners.
        •  
      • intelmq.bots.parsers.hibp.parser_callback: A new parser to parse data retrieved from a HIBP Enterprise Subscription.
        •  
      • intelmq.bots.parsers.shadowserver.parser:
        •  
      • Ability to detect the feed base on the reports's field extra.file_name, so the parameter feedname is no longer required and one configured parser can parse any feed (#1442).
        •  
      "},{"location":"changelog/#experts_19","title":"Experts","text":"
         
      • Add geohash expert.
        •  
      • intelmq.bots.experts.generic_db_lookup.expert
        •  
      • new optional parameter engine with postgresql (default) and sqlite (new) as possible values.
        •  
      "},{"location":"changelog/#outputs_15","title":"Outputs","text":"
         
      • Add intelmq.bots.outputs.touch.output.
        •  
      • intelmq.bots.outputs.postgresql.output:
        •  
      • deprecated in favor of intelmq.bots.outputs.sql.output
        •  
      • Compatibility shim will be available in the 2.x series.
        •  
      • intelmq.bots.outputs.sql.output added generic SQL output bot. Comparted to
        •  
      • new optional parameter engine with postgresql (default) and sqlite (new) as possible values.
        •  
      • intelmq.bots.outputs.stomp.output: New parameters message_hierarchical, message_jsondict_as_string, message_with_type, single_key.
        •  
      "},{"location":"changelog/#documentation_19","title":"Documentation","text":"
         
      • Feeds:
        •  
      • Add ViriBack feed.
        •  
      • Add Have I Been Pwned Enterprise Callback.
        •  
      • intelmq.tests.bots.outputs.amqptopic.test_output: Added.
        •  
      • Move the documentation of most bots from separate README files to the central Bots.md and feeds.yaml files.
        •  
      "},{"location":"changelog/#tests_17","title":"Tests","text":"
         
      • Travis:
        •  
      • Use UTC timezone.
        •  
      • Tests for utils.unzip.
        •  
      • Add a new asset: Zip archive with two files, same as with .tar.gz archive.
        •  
      • Added tests for the Mail Attachment & Mail URL collectors.
        •  
      • Ignore logging-tests on Python 3.7 temporarily (#1342).
        •  
      "},{"location":"changelog/#tools_15","title":"Tools","text":"
         
      • intelmqctl:
        •  
      • Use green and red text color for some interactive output to indicate obvious errors or the absence of them.
        •  
      • intelmqdump:
        •  
      • New edit action v to modify a message saved in the dump (#1284).
        •  
      "},{"location":"changelog/#contrib_8","title":"Contrib","text":"
         
      • malware name mapping:
        •  
      • Add support for MISP treat actors data, see it's README for more information.
           
        • And handle empty synonyms in misp's galxies data.
          •  
         
        •  
      • Move apply-Script to the new EventDB directory
        •  
      • EventDB: Scripts for applying malware name mapping and domain suffixes to an EventDB.
        •  
      "},{"location":"changelog/#known-issues_18","title":"Known issues","text":"
         
      • MongoDB authentication: compatibility on different MongoDB and pymongo versions (#1439)
        •  
      • ctl: shell colorizations are logged (#1436)
        •  
      • http stream collector: retry on regular connection problems? (#1435)
        •  
      • tests: capture logging with context manager (#1342)
        •  
      • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
        •  
      • n6 parser: mapping is modified within each run (#905)
        •  
      • reverse DNS: Only first record is used (#877)
        •  
      • Corrupt dump files when interrupted during writing (#870)
        •  
      "},{"location":"changelog/#202-2019-10-14","title":"2.0.2 (2019-10-14)","text":""},{"location":"changelog/#core_20","title":"Core","text":"
         
      • intelmq.lib.bot.CollectorBot: Support the deprecated parameter feed until version 2.2 as the documentation was not properly updated (#1445).
        •  
      • intelmq.lib.bot.Bot:
        •  
      • _dump_message: Wait for up to 60 seconds instead of 50 if the dump file is locked (the log message was said 60, but the code was for only 50).
        •  
      • intelmq.lib.upgrades.v202_fixes
        •  
      • Migration of deprecated parameter feed for Collectors.
        •  
      • Ripe expert parameter query_ripe_stat_ip was not correctly configured in v110_deprecations, now use query_ripe_stat_asn as default if it does not exist.
        •  
      • intelmq.lib.upgrades.v110_deprecations: Fix upgrade of ripe expert configuration.
        •  
      • intelmq.lib.bot_debugger:
        •  
      • Fix handling of empty messages generated by parser when user wanted to show the result by \"--show-sent\" flag.
        •  
      • Fix handling of sent messages for bots using the path_permissive parameter (#1453).
        •  
      • intelmq.lib.pipeline.Amqp:
        •  
      • use default SSL context for client purposes, fixes compatibility with python < 3.6 if TLS is used.
        •  
      • Reconnect once on sending messages if disconnect detected.
        •  
      "},{"location":"changelog/#bots_21","title":"Bots","text":""},{"location":"changelog/#collectors_16","title":"Collectors","text":"
         
      • intelmq.bots.collectors.api.collector_api:
        •  
      • Handle non-existing IO loop in shutdown.
        •  
      • Close socket on shutdown, fixes reloading.
        •  
      • Marked as non-threadable.
        •  
      • intelmq.bots.collectors.rt.collector_rt: Check for matching URLs if no attachment_regex is given.
        •  
      • intelmq.bots.collectors.stomp.collector_stomp: Handle disconnects by actively reconnecting.
        •  
      "},{"location":"changelog/#parsers_19","title":"Parsers","text":"
         
      • intelmq.bots.cymru.parser_cap_program: Fix parsing of the new $certname_$date.txt report format (#1443):
        •  
      • Support protocol ICMP.
        •  
      • Fix error message for unsupported protocols.
        •  
      • Support fields destination_port_numbers, port.
        •  
      • Support for all proxy types without ports.
        •  
      • Use Country Code of AS as source.geolocation.cc.
        •  
      • Support for 'scanner' and 'spam' categories.
        •  
      • Handle bogus lines with missing separator.
        •  
      • Fix bug preventing use of old format after using the new format.
        •  
      • Handle postfix (total_count:..) for destination port numbers.
        •  
      "},{"location":"changelog/#experts_20","title":"Experts","text":"
         
      • intelmq.bots.experts.cymru_whois.expert: Add optional parameter overwrite, current behavior was True, default if not given is True now, will change to False in 3.0.0 (#1452, #1455).
        •  
      • intelmq.bots.experts.modify.expert: Add optional parameter overwrite, current behavior was True, default if not given is True now, will change to False in 3.0.0 (#1452, #1455).
        •  
      • intelmq.bots.experts.reverse_dns.expert: Add optional parameter overwrite, current behavior was True, default if not given is True now, will change to False in 3.0.0 (#1452, #1455).
        •  
      "},{"location":"changelog/#outputs_16","title":"Outputs","text":"
         
      • intelmq.bots.outputs.amqptopic.output: use default SSL context for client purposes, fixes compatibility with python < 3.6 if TLS is used.
        •  
      "},{"location":"changelog/#packaging_10","title":"Packaging","text":"
         
      • Rules:
        •  
      • Exclude intelmqsetup tool in packages
        •  
      • Include update-rfiprisk-data in packages
        •  
      "},{"location":"changelog/#tests_18","title":"Tests","text":"
         
      • Tests for intelmq.lib.upgrades.v202_fixes.
        •  
      • Tests for intelmq.lib.upgrades.v110_deprecations.
        •  
      • Extended tests for intelmq.bots.parser.cymru.parser_cap_program.
        •  
      "},{"location":"changelog/#tools_16","title":"Tools","text":"
         
      • intelmqctl:
        •  
      • More and more precise logging messages for botnet starting and restarting, enable and disable.
        •  
      • No error message for disabled bots on botnet reload.
        •  
      • Fix upgrade-conf is state file is empty or not existing.
        •  
      • Use arpgarse's store_true action for flags instead of store_const.
        •  
      • If the loading of the defaults configuration failed, a variable definition was missing and causing an exception (#1456).
        •  
      "},{"location":"changelog/#contrib_9","title":"Contrib","text":"
         
      • Check MK Statistics Cronjob:
        •  
      • Use statistics_* parameters.
        •  
      • Make file executable
        •  
      • Handle None values in *.temporary.* keys and treat them as 0.
        •  
      • systemd:
        •  
      • Add PIDFile parameter to service file.
        •  
      "},{"location":"changelog/#known-issues_19","title":"Known issues","text":"
         
      • MongoDB authentication: compatibility on different MongoDB and pymongo versions (#1439)
        •  
      • ctl: shell colorizations are logged (#1436)
        •  
      • http stream collector: retry on regular connection problems? (#1435)
        •  
      • tests: capture logging with context manager (#1342)
        •  
      • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
        •  
      • n6 parser: mapping is modified within each run (#905)
        •  
      • reverse DNS: Only first record is used (#877)
        •  
      • Corrupt dump files when interrupted during writing (#870)
        •  
      "},{"location":"changelog/#201-2019-08-23","title":"2.0.1 (2019-08-23)","text":""},{"location":"changelog/#core_21","title":"Core","text":"
         
      • intelmq.lib.harmonization:
        •  
      • IPAddress: Remove Scope/Zone IDs for IPv6 addresses in sanitation.
        •  
      • All types: Handle None for validation and sanitation gracefully.
        •  
      • intelmq.lib.bot:
        •  
      • fix parameters of ParserBot and CollectorBot constructors, allowing intelmqctl run with these bots again (#1414).
        •  
      • Also run rate_limit after retry counter reset (#1431).
        •  
      • __version_info__:
        •  
      • is now available in the top level module.
        •  
      • uses integer values now instead of strings for numerical version parts
        •  
      • Also provide (empty) ROOT_DIR for non-pip installations.
        •  
      • intelmq.lib.upgrades: New library file upgrades with upgrade functions.
        •  
      • intelmq.lib.utils:
        •  
      • New function setup_list_logging for intelmqctl check an possibly others.
           
        • Fix return values (#1423).
          •  
         
        •  
      • New function version_smaller for version comparisons.
        •  
      • New function lazy_int for version conversions.
        •  
      • parse_logline: Handle thread IDs.
        •  
      • log takes a new argument logging_level_stream for the logging level of the console handler.
        •  
      • New constant LOG_FORMAT_SIMPLE, used by intelmqctl.
        •  
      • New function write_configuration to write dicts to files in the correct json formatting.
        •  
      • New function create_request_session_from_bot.
        •  
      • intelmq.lib.pipeline:
        •  
      • AMQP:
           
        • Actually use source/destination_pipeline_amqp_virtual_host parameter.
          •  
        • Support for SSL with source/destination_pipeline_ssl parameter.
          •  
         
        •  
      • pipeline base class: add missing dummy methods.
        •  
      • Add missing return types.
        •  
      • Redis: Evaluate return parameter of queue/key deletion.
        •  
      • Variable STATE_FILE_PATH added.
        •  
      "},{"location":"changelog/#development_7","title":"Development","text":"
         
      • intelmq.bin.intelmq_gen_docs: For yaml use safe_load instead of unsafe load.
        •  
      "},{"location":"changelog/#harmonization_3","title":"Harmonization","text":"
         
      • IPAddress type: Remove Scope/Zone IDs for IPv6 addresses in sanitation.
        •  
      • TLP: Sanitation handles now more cases: case-insensitive prefixes and arbitrary whitespace between the prefix and the value (#1420).
        •  
      "},{"location":"changelog/#bots_22","title":"Bots","text":""},{"location":"changelog/#collectors_17","title":"Collectors","text":"
         
      • intelmq.bots.collectors.http.collector_http: Use utils.create_request_session_from_bot.
        •  
      • intelmq.bots.collectors.http.collector_http_stream: Use utils.create_request_session_from_bot and thus fix some retries on connection timeouts.
        •  
      • intelmq.bots.collectors.mail.collector_mail_url: Use utils.create_request_session_from_bot.
        •  
      • intelmq.bots.collectors.microsoft.collector_interflow: Use utils.create_request_session_from_bot and thus fix retries on connection timeouts.
        •  
      • intelmq.bots.collectors.rt.collector_rt: Use utils.create_request_session_from_bot and thus fix retries on connection timeouts.
        •  
      • intelmq.bots.collectors.twitter.collector_twitter: Use utils.create_request_session_from_bot and thus fix retries on connection timeouts for non-twitter connections.
        •  
      "},{"location":"changelog/#parsers_20","title":"Parsers","text":"
         
      • intelmq.bots.parsers.n6.parser_n6stomp: use malware-generic instead of generic-n6-drone for unknown infected system events.
        •  
      • intelmq.bots.parsers.abusech.parser_ip: Support LastOnline column in feodo feed (#1400) and use it for time.source if available.
        •  
      • Use lower case malware names as default, should not make a difference in practice.
        •  
      • Fix handling of CSV header for feodotracker (#1417, #1418).
        •  
      • intelmq.bots.parsers.netlab_360.parser: Detect feeds with https:// too.
        •  
      "},{"location":"changelog/#experts_21","title":"Experts","text":"
         
      • intelmq.bots.experts.generic_db_lookup: Recommend psycopg2-binary package.
        •  
      • intelmq.bots.experts.modify.expert:
        •  
      • Compile regular expressions (all string rules) at initialization, improves the speed.
        •  
      • Warn about old configuration style deprecation.
        •  
      • intelmq.bots.experts.do_portal.expert:
        •  
      • Use utils.create_request_session_from_bot and thus fix retries on connection timeouts (#1432).
        •  
      • Treat \"502 Bad Gateway\" as timeout which can be retried.
        •  
      • intelmq.bots.experts.ripe.expert: Use utils.create_request_session_from_bot and thus fix retries on connection timeouts.
        •  
      • intelmq.bots.experts.url2fqdn.expert: Support for IP addresses in hostnames (#1416).
        •  
      • intelmq.bots.experts.national_cert_contact_certat.expert: Use utils.create_request_session_from_bot and thus fix retries on connection timeouts.
        •  
      "},{"location":"changelog/#outputs_17","title":"Outputs","text":"
         
      • intelmq.bots.outputs.postgresql: Recommend psycopg2-binary package.
        •  
      • intelmq.bots.outputs.amqptopic:
        •  
      • Shutdown: Close connection only if connection exists.
        •  
      • Add support for pika > 1. Pika changed the way it indicates (Non-)Acknowledgments of sent messages.
        •  
      • Gracefully handle unroutable messages and give advice.
        •  
      • Support for connections without authentication.
        •  
      • Replace deprecated parameter type with exchange_type for exchange_declare, supporting pika >= 0.11 (#1425).
        •  
      • New parameters message_hierarchical_output, message_with_type, message_jsondict_as_string.
        •  
      • New parameter use_ssl for SSL connections.
        •  
      • New parameter single_key for sending single fields instead of the full event.
        •  
      • intelmq.bots.outputs.mongodb.output: Support for pymongo >= 3.0.0 (#1063, PR#1421).
        •  
      • intelmq.bots.outputs.file: time.* field serialization: support for microseconds.
        •  
      • intelmq.bots.outputs.mongodb.output: Support for authentication in pymongo >= 3.5 (#1062).
        •  
      • intelmq.bots.outputs.restapi.output: Use utils.create_request_session_from_bot and thus fix retries on connection timeouts.
        •  
      "},{"location":"changelog/#documentation_20","title":"Documentation","text":"
         
      • Add certbund-contact to the ecosystem document.
        •  
      • Rename the IDEA expert to \"IDEA Converter\".
        •  
      • Add the new configuration upgrade function to the docs.
        •  
      • User Guide:
        •  
      • Clarify on Uninstallation
        •  
      "},{"location":"changelog/#packaging_11","title":"Packaging","text":"
         
      • Do not execute the tcp collector tests during Debian and Ubuntu builds as they fail there.
        •  
      "},{"location":"changelog/#tests_19","title":"Tests","text":"
         
      • intelmq.lib.test: Disable statistics for test runs of bots.
        •  
      • contrib.malware_name_mapping: Added tests.
        •  
      • Travis: Also run tests of contrib.
        •  
      "},{"location":"changelog/#tools_17","title":"Tools","text":"
         
      • intelmqsetup: Only change directory ownerships if necessary.
        •  
      • intelmqctl:/**---
        •  
      • Provide new command upgrade-conf to upgrade configuration to a newer version.
           
        • Makes backups of configurations files on its own.
          •  
        • Also checks for previously skipped or new functions of older versions and catches up.
          •  
         
        •  
      • Provides logging level on class layer.
        •  
      • Fix -q flag for intelmqctl list queues by renaming its alternative name to --non-zero to avoid a name collision with the global --quiet parameter.
        •  
      • For console output the string intelmqctl: at the beginning of each line is no longer present.
        •  
      • check: Support for the state file added. Checks if it exists and all upgrade functions have been executed successfully.
        •  
      • Waits for up to 2 seconds when stopping a bot (#1434).
        •  
      • Exits early on restart when stopping a bot did not work (#1434).
        •  
      • intelmqctl run process -m debugging: Mock acknowledge method if incoming message is mocked too, otherwise a different message is acknowledged.
        •  
      • Queue listing for AMQP: Support non-default monitoring URLs, see User-Guide.
        •  
      "},{"location":"changelog/#contrib_10","title":"Contrib","text":"
         
      • logcheck rules: Adapt ignore rule to cover the instance IDs of bot names.
        •  
      • malware name mapping:
        •  
      • Ignore lines in mapping starting with '#'.
        •  
      • Optionally include malpedia data.
        •  
      • Fix command line parsing for not arguments (#1427).
        •  
      • bash-completion: Support for intelmqctl upgrade-config added.
        •  
      "},{"location":"changelog/#known-issues_20","title":"Known issues","text":"
         
      • http stream collector: retry on regular connection problems? (#1435)
        •  
      • tests: capture logging with context manager (#1342)
        •  
      • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
        •  
      • n6 parser: mapping is modified within each run (#905)
        •  
      • reverse DNS: Only first record is used (#877)
        •  
      • Corrupt dump files when interrupted during writing (#870)
        •  
      "},{"location":"changelog/#200-2019-05-22","title":"2.0.0 (2019-05-22)","text":"
      See also the changelog for 2.0.0.beta1 below.
      "},{"location":"changelog/#configurations","title":"Configurations","text":"
         
      • Defaults: New parameters statistics_host, statistics_port, statistics_databasae, statistics_password for statistics redis database (#1402).
        •  
      "},{"location":"changelog/#core_22","title":"Core","text":"
         
      • Add more and fix some existing type annotations.
        •  
      • intelmq.lib.bot:
        •  
      • Use statistics_* parameters for bot's statistics (#1402).
        •  
      • Introduce collector_empty_process for collectors with an empty process() method, hardcoded 1s minimum sleep time, preventing endless loops, causing high load (#1364).
        •  
      • Allow to disable multithreading by initialization parameter, used by intelmqctl / the bot debugger (#1403).
        •  
      • intelmq.lib.pipeline: redis: OOM can also be low memory, add this to log message (#1405).
        •  
      • intelmq.lib.harmonization: ClassificationType: Update RSIT mapping (#1380):
        •  
      • replace botnet drone with infected-system
        •  
      • replace infected system with infected-system
        •  
      • replace ids alert with ids-alert
        •  
      • replace c&c with c2server
        •  
      • replace malware configuration with malware-configuration
        •  
      • sanitize replaces these values on the fly
        •  
      • Allow using non-opt/ (LSB) paths with environment variable INTELMQ_PATHS_NO_OPT.
        •  
      • Disable/disallow threading for all collectors and some other bots.
        •  
      "},{"location":"changelog/#development_8","title":"Development","text":"
         
      • Applied isort to all core files and core-related test files, sorting the imports there (every thing except bots and bots' tests).
        •  
      "},{"location":"changelog/#harmonization_4","title":"Harmonization","text":"
         
      • See the Core section for the changes in the allowed values for classification.type.
        •  
      "},{"location":"changelog/#bots_23","title":"Bots","text":"
         
      • Use the new RSIT types in several bots, see above
        •  
      "},{"location":"changelog/#parsers_21","title":"Parsers","text":"
         
      • intelmq.bots.parsers.spamhaus.parser_cert: Added support for extortion events.
        •  
      "},{"location":"changelog/#experts_22","title":"Experts","text":"
         
      • added intelmq.bots.experts.do_portal.expert.
        •  
      "},{"location":"changelog/#outputs_18","title":"Outputs","text":"
         
      • intelmq.bots.outputs.elasticsearch.output: Support for TLS added (#1406).
        •  
      • intelmq.bots.outputs.tcp.output: Support non-intelmq counterparts again. New parameter counterpart_is_intelmq, see NEWS.md for more information (#1385).
        •  
      "},{"location":"changelog/#packaging_12","title":"Packaging","text":"
         
      • Update IntelMQ path fix patch after INTELMQ_PATHS_NO_OPT introduction, provide INTELMQ_PATHS_OPT environment variable for packaged instances.
        •  
      "},{"location":"changelog/#tests_20","title":"Tests","text":"
         
      • test_conf: For yaml use safe_load instead of unsafe load.
        •  
      • Travis: Switch distribution from trusty to xenial, adapt scripts.
        •  
      • Add Python 3.7 to tests.
        •  
      • Don't use Cerberus 1.3 because of https://github.com/pyeve/cerberus/issues/489
        •  
      • Add tests for intelmqctl.lib.upgrades.
        •  
      "},{"location":"changelog/#tools_18","title":"Tools","text":"
         
      • intelmqdump: Fix creation of pipeline object by providing a logger.
        •  
      • intelmqctl: Disable multithreading for interactive runs / the bot debugger (#1403).
        •  
      "},{"location":"changelog/#known-issues_21","title":"Known issues","text":"
         
      • tests: capture logging with context manager (#1342)
        •  
      • pymongo 3.0 deprecates used insert method  (#1063)
        •  
      • pymongo >= 3.5: authentication changes  (#1062)
        •  
      • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
        •  
      • n6 parser: mapping is modified within each run (#905)
        •  
      • reverse DNS: Only first record is used (#877)
        •  
      • Corrupt dump files when interrupted during writing (#870)
        •  
      "},{"location":"changelog/#200beta1-2019-04-10","title":"2.0.0.beta1 (2019-04-10)","text":"
      There are some features considered as beta and marked as such in the documentation, do not use them in production yet.
      "},{"location":"changelog/#removals-of-deprecated-code","title":"Removals of deprecated code:","text":"
         
      • Removed compatibility shim intelmq.bots.collectors.n6.collector_stomp, use intelmq.bots.collectors.stomp.collector instead (see #1124).
        •  
      • Removed compatibility shim intelmq.bots.parsers.cymru_full_bogons.parser, use intelmq.bots.parsers.cymru.parser_full_bogons instead.
        •  
      • Removed compatibility shim handling deprecated parameter feed for collectors. Use name instead.
        •  
      • Removed deprecated and unused method intelmq.lib.pipeline.Pipeline.sleep.
        •  
      • Removed support for deprecated parameter query_ripe_stat in intelmq.bots.experts.ripe.expert, use query_ripe_stat_asn and query_ripe_stat_ip instead (#1291).
        •  
      • Removed deprecated and unused function intelmq.lib.utils.extract_tar.
        •  
      "},{"location":"changelog/#core_23","title":"Core","text":"
         
      • lib/pipeline:
        •  
      • Allow setting the broker of source and destination independently.
        •  
      • Support for a new AMQP broker. See User Guide for configuration. (#1179)
        •  
      • lib/bot:
        •  
      • Dump messages locks the dump file using Unix file locks (#574).
        •  
      • Print idle/rate limit time also in human readable format (#1332).
        •  
      • set_request_parameters: Use {} as default proxy value instead of None. Allows updating of existing proxy dictionaries.
        •  
      • Bots drop privileges if they run as root.
        •  
      • Save statistics on successfully and failed processed messages in the redis database 3.
        •  
      • lib/utils
        •  
      • Function unzip to extract files from gz-zipped and/or tar-archives.
        •  
      • New class ListHandler: new handler for logging purpose which saves the messages in a list.
        •  
      • Add function seconds_to_human.
        •  
      • Add function drop_privileges.
        •  
      • parse_relative: Strip string before parsing.
        •  
      • parse_logline: Do not convert the timestamps to UTC, leave them as is.
        •  
      • lib/cache:
        •  
      • Allow ttl to be None explicitly.
        •  
      • Overwrite existing cache keys in the database instead of discarding the new data.
        •  
      • lib/bot:
        •  
      • Basic, but easy-to-configure multi-threading using python's threading library. See the User-Guide for more information (#111, #186).
        •  
      • bin/intelmqctl:
        •  
      • Support for Supervisor as process manager (#693, #1360).
        •  
      "},{"location":"changelog/#development_9","title":"Development","text":"
         
      • upgraded all files to python3-only syntax, e.g. use super() instead of super(..., ...) in all files. Migration from old to new string formatting has not been applied if the resulting code would be longer.
        •  
      "},{"location":"changelog/#bots_24","title":"Bots","text":""},{"location":"changelog/#collectors_18","title":"Collectors","text":"
         
      • added intelmq.bots.parsers.opendxl.collector (#1265).
        •  
      • added intelmq.bots.collectors.api: collecting data using an HTTP API (#123, #1187).
        •  
      • added intelmq.bots.collectors.rsync (#1286).
        •  
      • intelmq.bots.collectors.http.collector_http:
        •  
      • Add support for uncompressing of gz-zipped-files (#1270).
        •  
      • Add time-delta support for time formatted URLs (#1366).
        •  
      • intelmq.collectors.blueliv.collector_crimeserver: Allow setting the API URL by parameter (#1336).
        •  
      • intelmq.collectors.mail:
        •  
      • Use internal lib for functionality.
        •  
      • Add intelmq.bots.collectors.mail.collector_mail_body.
        •  
      • Support for ssl_ca_certificate parameter (#1362).
        •  
      "},{"location":"changelog/#parsers_22","title":"Parsers","text":"
         
      • added intelmq.bots.parsers.mcafee.parser_atd (#1265).
        •  
      • intelmq.bots.parsers.generic.parser_csv:
        •  
      • New parameter columns_required to optionally ignore parse errors for columns.
        •  
      • added intelmq.bots.parsers.cert_eu.parser_csv (#1287).
        •  
      • Do not overwrite the local time.observation with the data from the feed. The feed's field 'observation time' is now saved in the field extra.cert_eu_time_observation.
        •  
      • Fix parsing of asn (renamed to source asn, source.asn internally) and handle existing feed.accuracy for parsing confidence.
        •  
      • Update columns and mapping to current (2019-04-02) data.
        •  
      • added intelmq.bots.parsers.surbl.surbl
        •  
      • added intelmq.bots.parsers.html_table (#1381).
        •  
      • intelmq.bots.parsers.netlab_360.parser: Handle empty lines containing blank characters (#1393).
        •  
      • intelmq.bots.parsers.n6.parser_n6stomp: Handle events without IP addresses.
        •  
      • intelmq.bots.parsers.cymru.parser_cap_program: Handle new feed format.
        •  
      • intelmq.bots.parsers.shadowserver:
        •  
      • Add support for the Accessible-FTP feed (#1391).
        •  
      • intelmq.bots.parsers.dataplane.parser:
        •  
      • Fix parse errors and log more context (#1396).
        •  
      • added intelmq.bots.parsers.fraunhofer.parser_ddosattack_cnc.py and intelmq.bots.parsers.fraunhofer.parser_ddosattack_target.py (#1373).
        •  
      "},{"location":"changelog/#experts_23","title":"Experts","text":"
         
      • added intelmq.bots.experts.recordedfuture_iprisk (#1267).
        •  
      • added intelmq.bots.experts.mcafee.expert_mar (1265).
        •  
      • renamed intelmq.bots.experts.ripencc_abuse_contact.expert to intelmq.bots.experts.ripe.expert, compatibility shim will be removed in version 3.0.
        •  
      • Added support for geolocation information in ripe expert with a new parameter query_ripe_stat_geolocation (#1317).
        •  
      • Restructurize the expert and code de-duplicataion (#1384).
        •  
      • Handle '?' in geolocation country data (#1384).
        •  
      • intelmq.bots.experts.ripe.expert:
        •  
      • Use a requests session (#1363).
        •  
      • Set the requests parameters once per session.
        •  
      • intelmq.bots.experts.maxmind_geoip.expert: New parameter use_registered to use the registered country (#1344).
        •  
      • intelmq.bots.experts.filter.expert: Support for paths (#1208).
        •  
      "},{"location":"changelog/#outputs_19","title":"Outputs","text":"
         
      • added intelmq.bots.experts.mcafee.output_esm (1265).
        •  
      • added intelmq.bots.outputs.blackhole (#1279).
        •  
      • intelmq.bots.outputs.restapi.expert:
        •  
      • Set the requests parameters once per session.
        •  
      • intelmq.bots.outputs.redis:
        •  
      • New parameter hierarchichal_output (#1388).
        •  
      • New parameter with_type.
        •  
      • intelmq.bots.outputs.amqptopic.output: Compatibility with pika 1.0.0 (#1084, #1394).
        •  
      "},{"location":"changelog/#documentation_21","title":"Documentation","text":"
         
      • added documentation for feeds
        •  
      • CyberCrime Tracker
        •  
      • Feodo Tracker Latest
        •  
      • Feeds: Document abuse.ch URLhaus feed (#1379).
        •  
      • Install and Upgrading: Use intelmqsetup tool.
        •  
      • Added an ecosystem overview document describing related software.
        •  
      "},{"location":"changelog/#tests_21","title":"Tests","text":"
         
      • Add tests of AMQP broker.
        •  
      • Travis: Change the ownership of /opt/intelmq to the current user.
        •  
      "},{"location":"changelog/#tools_19","title":"Tools","text":"
         
      • intelmqctl check: Now uses the new ListHandler from utils to handle the logging in JSON output mode.
        •  
      • intelmqctl run: The message that a running bot has been stopped, is not longer a warning, but an informational message. No need to inform sysadmins about this intended behavior.
        •  
      • intelmqdump: Inspecting dumps locks the dump file using unix file locks (#574).
        •  
      • intelmqctl:
        •  
      • After the check if the program runs as root, it tries to drop privileges. Only if this does not work, a warning is shown.
        •  
      • intelmqsetup: New tool for initializing an IntelMQ environment.
        •  
      "},{"location":"changelog/#contrib_11","title":"Contrib","text":"
         
      • malware_name_mapping:
        •  
      • Added the script apply_mapping_eventdb.py to apply the mapping to an EventDB.
        •  
      • Possibility to add local rules using the download tool.
        •  
      • check_mk:
        •  
      • Added scripts for monitoring queues and statistics.
        •  
      "},{"location":"changelog/#known-issues_22","title":"Known issues","text":"
         
      • Multi-threaded bots require multiple SIGTERMs (#1403)
        •  
      • Stats can't be saved with AMQP if redis is password-protected (#1402)
        •  
      • Update taxonomies to current RSIT and vice-versa (#1380)
        •  
      • stomp collector bot constantly uses 100% of CPU (#1364)
        •  
      • tests: capture logging with context manager (#1342)
        •  
      • Consistent message counter log messages for all kind of bots (#1278)
        •  
      • pymongo 3.0 deprecates used insert method  (#1063)
        •  
      • pymongo >= 3.5: authentication changes  (#1062)
        •  
      • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
        •  
      • n6 parser: mapping is modified within each run (#905)
        •  
      • reverse DNS: Only first record is used (#877)
        •  
      • Corrupt dump files when interrupted during writing (#870)
        •  
      "},{"location":"changelog/#112-2019-03-25","title":"1.1.2 (2019-03-25)","text":""},{"location":"changelog/#core_24","title":"Core","text":"
         
      • intelmq.lib.bot:
        •  
      • Bot.__handle_sighup: Handle exceptions in shutdown method of bots.
        •  
      "},{"location":"changelog/#harmonization_5","title":"Harmonization","text":"
         
      • FQDN: Disallow : in FQDN values to prevent values like '10.0.0.1:8080' (#1235).
        •  
      "},{"location":"changelog/#bots_25","title":"Bots","text":""},{"location":"changelog/#collectors_19","title":"Collectors","text":"
         
      • intelmq.bots.collectors.stomp.collector
        •  
      • Fix name of shutdown method, was ineffective in the past.
        •  
      • Ignore NotConnectedException errors on disconnect during shutdown.
        •  
      • intelmq.bots.collectors.mail.collector_mail_url: Decode body if it is bytes (#1367).
        •  
      • intelmq.bots.collectors.tcp.collector: Timeout added. More stable version.
        •  
      "},{"location":"changelog/#parsers_23","title":"Parsers","text":"
         
      • intelmq.bots.parsers.shadowserver:
        •  
      • Add support for the Amplification-DDoS-Victim, HTTP-Scanners, ICS-Scanners and Accessible-Ubiquiti-Discovery-Service feeds (#1368, #1383)
        •  
      • intelmq.bots.parsers.microsoft.parser_ctip:
        •  
      • Workaround for mis-formatted data in networkdestinationipv4 field (since 2019-03-14).
        •  
      • Ignore \"hostname\" (\"destination.fqdn\") if it contains invalid data.
        •  
      • intelmq.bots.parsers.shodan.parser:
        •  
      • In minimal_mode:
           
        • Fix the parsing, previously only source.geolocation.cc and extra.shodan was correctly filled with information.
          •  
        • Add a classification.type = 'other' to all events.
          •  
        • Added tests for this mode.
          •  
         
        •  
      • Normal mode:
           
        • Fix the parsing of timestamp to `time.source in the normal mode, previously no timezone information has been added and thus every event raised an exception.
          •  
        • ISAKMP: Ignore isakmp.aggressive, as the content is same as isakmp or less.
          •  
         
        •  
      • intelmq.bots.parsers.abusech.parser_ip: Re-structure the bot and support new format of the changed \"Feodo Tracker Domains\" feed.
        •  
      • intelmq.bots.parsers.n6.parser:
        •  
      • Add parsing for fields \"confidence\", \"expires\" and \"source\".
        •  
      • Add support for type \"bl-other\" (category \"other\").
        •  
      "},{"location":"changelog/#experts_24","title":"Experts","text":"
         
      • intelmq.bots.experts.sieve.expert: Fix key definition to allow field names with numbers (malware.hash.md5/sha1, #1371).
        •  
      "},{"location":"changelog/#outputs_20","title":"Outputs","text":"
         
      • intelmq.bots.outputs.tcp.output: Timeout added. When no separator used, awaits that every message is acknowledged by a simple \"Ok\" string to ensure more stability.
        •  
      "},{"location":"changelog/#documentation_22","title":"Documentation","text":"
         
      • Install: Update operating system versions
        •  
      • Sieve Expert: Fix elsif -> elif.
        •  
      • Rephrase the description of time.* fields.
        •  
      • Feeds: New URL and format of the \"Feodo Tracker IPs\" feed. \"Feodo Tracker Domains\" has been discontinued.
        •  
      "},{"location":"changelog/#packaging_13","title":"Packaging","text":""},{"location":"changelog/#tests_22","title":"Tests","text":"
         
      • Add missing __init__.py files in 4 bot's test directories. Previously these tests have never been executed.
        •  
      • intelmq.lib.test: Allow bot test class names with an arbitrary postfix separated by an underscore. E.g. TestShodanParserBot_minimal.
        •  
      "},{"location":"changelog/#tools_20","title":"Tools","text":"
         
      • intelmqctl:
        •  
      • status: Show commandline differences if a program with the expected PID could be found, but they do not match (previous output was None).
        •  
      • Use logging level from defaults configuration if possible, otherwise intelmq's internal default. Previously, DEBUG was used unconditionally.
        •  
      "},{"location":"changelog/#known-issues_23","title":"Known issues","text":"
         
      • Bots started with IntelMQ-Manager stop when the webserver is restarted (#952).
        •  
      • stomp collector bot constantly uses 100% of CPU (#1364).
        •  
      "},{"location":"changelog/#111-2019-01-15","title":"1.1.1 (2019-01-15)","text":""},{"location":"changelog/#core_25","title":"Core","text":"
         
      • lib/harmonization.py: Change parse_utc_isoformat of DateTime class from private to public (related to #1322).
        •  
      • lib/utils.py: Add new function object_pair_hook_bots.
        •  
      • lib.bot.py:
        •  
      • ParserBot's method recover_line_csv now also handles given tempdata.
        •  
      • Bot.acknowledge_message() deletes __current_message to free the memory, saves memory in idling parsers with big reports.
        •  
      • start(): Warn once per run if error_dump_message is set to false.
        •  
      • Bot.start(), ParserBot.process(): If errors happen on bots without destination pipeline, the on_error path has been queried and lead to an exception being raised.
        •  
      • start(): If error_procedure is pass and on pipeline errors, the bot retries forever (#1333).
        •  
      • lib/message.py:
        •  
      • Fix add('extra', ..., overwrite=True): old extra fields have not been deleted previously (#1335).
        •  
      • Do not ignore empty or ignored (as defined in _IGNORED_VALUES) values of extra.* fields for backwards compatibility (#1335).
        •  
      • lib/pipeline.py (Redis.receive): Wait in 1s steps if redis is busy loading its snapshot from disk (#1334).
        •  
      "},{"location":"changelog/#default-configuration","title":"Default configuration","text":"
         
      • Set error_dump_message to true by default in defaults.conf.
        •  
      • Fixed typo in defaults.conf: proccess_manager -> process_manager
        •  
      "},{"location":"changelog/#development_10","title":"Development","text":"
         
      • bin/rewrite_config_files.py: Fix ordering of BOTS file (#1327).
        •  
      "},{"location":"changelog/#harmonization_6","title":"Harmonization","text":"
      Update allowed classification fields to 2018-09-26 version (#802, #1350, #1380). New values for classification.type are per taxonomy: - Taxonomy 'intrusions':   - \"application-compromise\"   - \"burglary\"   - \"privileged-account-compromise\"   - \"unprivileged-account-compromise\" - Taxonomy 'fraud':   - \"copyright\"   - \"masquerade\"   - \"unauthorized-use-of-resources\" - Taxonomy 'information content security':   - \"data-loss\" - Taxonomy 'vulnerable':   - \"ddos-amplifier\"   - \"information-disclosure\"   - \"potentially-unwanted-accessible\"   - \"vulnerable-system\"   - \"weak-crypto\" - Taxonomy 'availability':   - \"dos\"   - \"outage\"   - \"sabotage\" - Taxonomy 'abusive-content':   - \"harmful-speech\"   - \"violence\" - Taxonomy 'malicious code':   - \"malware-distribution\" - Taxonomy 'information-gathering':   - \"social-engineering\"   - \"sniffing\" - Taxonomy 'information content security':   - \"Unauthorised-information-access\"   - \"Unauthorised-information-modification\"
      "},{"location":"changelog/#bots_26","title":"Bots","text":""},{"location":"changelog/#collectors_20","title":"Collectors","text":"
         
      • intelmq.bots.collectors.http.collector_http:
        •  
      • Fix parameter name extract_files in BOTS (#1331).
        •  
      • Fix handling of extract_files parameter if the value is an empty string.
        •  
      • Handle not installed dependency library requests gracefully.
        •  
      • Explain extract_files parameter in docs and use a sane default in BOTS file.
        •  
      • intelmq.bots.collectors.mail.collector_mail_url:
        •  
      • Handle HTTP status codes != 2xx the same as HTTP timeouts: No exception, but graceful handling.
        •  
      • Handle HTTP errors (bad status code and timeouts) with error_procedure == 'pass' but marking the mail as read and logging the error.
        •  
      • Handle not installed dependency library requests gracefully.
        •  
      • intelmq.bots.collectors.http.collector_http_stream:
        •  
      • Handle not installed dependency library requests gracefully.
        •  
      • intelmq.bots.collectors.microsoft.collector_interflow:
        •  
      • Handle not installed dependency library requests gracefully.
        •  
      • intelmq.bots.collectors.rt.collector_rt:
        •  
      • Handle not installed dependency library requests gracefully.
        •  
      • added intelmq.bots.collectors.shodan.collector_stream for collecting shodan stream data (#1096).
        •  
      • Correctly check the version of the shodan library, it resulted in wrong comparisons with two digit numbers.
        •  
      • intelmq.bots.collectors.microsoft.collector_interflow:
        •  
      • Add check if Cache's TTL is big enough compared to not_older_than and throw an error otherwise.
        •  
      "},{"location":"changelog/#parsers_24","title":"Parsers","text":"
         
      • intelmq.bots.parsers.misp: Fix Object attribute (#1318).
        •  
      • intelmq.bots.parsers.cymru.parser_cap_program:
        •  
      • Add support for new format (extra data about botnet of 'bots').
        •  
      • Handle AS number 0.
        •  
      • intelmq.bots.parsers.shadowserver:
        •  
      • Spam URL reports: remove src_naics, src_sic columns.
        •  
      • fix parsing of 'spam' events in ShadowServer's 'Botnet Drone Hadoop' Report (#1271).
        •  
      • Add support in parser to ignore some columns in config file by using False as intelmq key.
        •  
      • Add support for the Outdated-DNSSEC-Key and Outdated-DNSSEC-Key-IPv6 feeds.
        •  
      • Add support for the Accessible-Rsync feed.
        •  
      • Document support for the Open-LDAP-TCP feed.
        •  
      • Add support for Accessible-HTTP and Open-DB2-Discovery-Service (#1349).
        •  
      • Add support for Accessible-AFP (#1351).
        •  
      • Add support for Darknet (#1353).
        •  
      • intelmq.bots.parsers.generic.parser_csv: If the skip_header parameter was set to True, the header was not part of the raw field as returned by the recover_line method. The header is now saved and handled correctly by the fixed recovery method.
        •  
      • intelmq.bots.parsers.cleanmx.parser: Use field first instead of firsttime for time.source (#1329, #1348).
        •  
      • intelmq.bots.parsers.twitter.parser: Support for url-normalize >= 1.4.1 and recommend it. Added new optional parameter default_scheme, passed to url-normalize (#1356).
        •  
      "},{"location":"changelog/#experts_25","title":"Experts","text":"
         
      • intelmq.bots.experts.national_cert_contact_certat.expert:
        •  
      • Handle not installed dependency library requests gracefully.
        •  
      • intelmq.bots.experts.ripencc_abuse_contact.expert:
        •  
      • Handle not installed dependency library requests gracefully.
        •  
      • intelmq.bots.experts.sieve.expert:
        •  
      • check method: Load missing harmonization, caused an error for every check.
        •  
      • Add text and more context to error messages.
        •  
      • README: Fix 'modify' to 'update' (#1340).
        •  
      • Handle empty rules file (#1343).
        •  
      • intelmq.bots.experts.idea.expert: Add mappings for new harmonization classification.type values, see above.
        •  
      "},{"location":"changelog/#outputs_21","title":"Outputs","text":"
         
      • intelmq.bots.outputs.redis:
        •  
      • Fix sending password to redis server.
        •  
      • Fix for redis-py >= 3.0.0: Convert Event to string explicitly (#1354).
        •  
      • Use Redis class instead of deprecated StrictRedis for redis-py >= 3.0.0 (#1355).
        •  
      • intelmq.bots.outputs.mongodb:
        •  
      • New parameter replacement_char (default: '_') for non-hierarchical output as dots in key names are not allowed (#1324, #1322).
        •  
      • Save value of fields time.observation and time.source as native datetime object, not as string (#1322).
        •  
      • intelmq.bots.outputs.restapi.output:
        •  
      • Handle not installed dependency library requests gracefully.
        •  
      "},{"location":"changelog/#documentation_23","title":"Documentation","text":"
         
      • FAQ
        •  
      • Explanation and solution on orphaned queues.
        •  
      • Section on how and why to remove raw data.
        •  
      • Add or fix the tables of contents for all documentation files.
        •  
      • Feeds:
        •  
      • Fix Autoshun Feed URL (#1325).
        •  
      • Add parameters name and provider to intelmq/etc/feeds.yaml, docs/Feeds.md and intelmq/bots/BOTS (#1321).
        •  
      • Add SECURITY.md file.
        •  
      "},{"location":"changelog/#packaging_14","title":"Packaging","text":"
         
      • Change the maintainer from Sasche Wilde to Sebastian Wagner (#1320).
        •  
      "},{"location":"changelog/#tests_23","title":"Tests","text":"
         
      • intelmq.tests.lib.test_bot: Skip test_logging_level_other on python 3.7 because of unclear behavior related to copies of loggers (#1269).
        •  
      • intelmq.tests.bots.collectors.rt.test_collector: Remove test because the REST interface of the instance has been closed (see also https://github.com/CZ-NIC/python-rt/issues/28).
        •  
      "},{"location":"changelog/#tools_21","title":"Tools","text":"
         
      • intelmqctl check: Shows more detailed information on orphaned queues.
        •  
      • intelmqctl:
        •  
      • Correctly determine the status of bots started with intelmqctl run.
        •  
      • Fix output of errors during bot status determination, making it compatible to IntelMQ Manager.
        •  
      • check subcommand: Show bot ID for messages also in JSON output.
        •  
      • run [bot-id] process -m [message] works also with bots without a configured source pipeline (#1307).
        •  
      "},{"location":"changelog/#contrib_12","title":"Contrib","text":"
         
      • elasticsearch/elasticmapper: Add tlp field (#1308).
        •  
      • feeds-config-generator/intelmq_gen_feeds_conf:
        •  
      • Add parameters to write resulting configuration directly to files (#1321).
        •  
      • Handle collector's feed.name and feed.provider (#1314).
        •  
      "},{"location":"changelog/#known-issues_24","title":"Known issues","text":"
         
      • Bots started with IntelMQ-Manager stop when the webserver is restarted (#952).
        •  
      • Tests: capture logging with context manager (#1342).
        •  
      • stomp collector bot constantly uses 100% of CPU (#1364).
        •  
      "},{"location":"changelog/#110-2018-09-05","title":"1.1.0 (2018-09-05)","text":"
         
      • Support for Python 3.3 has been dropped in IntelMQ and some dependencies of it. Python 3.3 reached its end of life and Python 3.4 or newer is a hard requirement now.
        •  
      • The list of feeds docs/Feeds.md has now a machine-readable equivalent YAML file in intelmq/etc/feeds.yaml   A tool to convert from yaml to md has been added.
        •  
      "},{"location":"changelog/#tools_22","title":"Tools","text":"
         
      • intelmq_gen_feeds_docs added to bin directory, allows generating the Feeds.md documentation file from feeds.yaml
        •  
      • intelmq_gen_docs merges both intelmq_gen_feeds_docs and intelmq_gen_harm_docs in one file and automatically updates the documentation files.
        •  
      "},{"location":"changelog/#intelmqctl","title":"intelmqctl","text":"
         
      • intelmqctl start prints the bot's last error messages if the bot failed to start (#1021).
        •  
      • intelmqctl start message \"is running\" is printed every time. (Until now, it wasn't said when a bot was just starting.)
        •  
      • intelmqctl start/stop/restart/reload/status now has a \"--group\" flag which allows you to specify the group of the bots that should be influenced by the command.
        •  
      • intelmqctl check checks for defaults.conf completeness if the shipped file from the package can be found.
        •  
      • intelmqctl check shows errors for non-importable bots.
        •  
      • intelmqctl list bots -q only prints the IDs of enabled bots.
        •  
      • intelmqctl list queues-and-status prints both queues and bots statuses (so that it can be used in eg. intelmq-manager).
        •  
      • intelmqctl run parameter for showing a sent message.
        •  
      • intelmqctl run if message is sent to a non-default path, it is printed out.
        •  
      • intelmqctl restart bug fix; returned some half-nonsense, now returns return state of start and stop operation in a list (#1226).
        •  
      • intelmqctl check: New parameter --no-connections to prevent the command from making connections e.g. to the redis pipeline.s
        •  
      • intelmqctl list queues: don't display named paths among standard queues.
        •  
      • The process status test failed if the PATH did not include the bot executables and the which command failed. Then the proccess's command line could not be compared correctly. The fix warns of this and adds a new status 'unknown' (#1297).
        •  
      "},{"location":"changelog/#contrib_13","title":"Contrib","text":"
         
      • tool feeds-config-generator to automatically generate the collector and parser runtime and pipeline configurations.
        •  
      • malware_name_mapping: Download and convert tool for malware family name mapping has been added.
        •  
      • Added a systemd script which creates systemd units for bots (#953).
        •  
      • contrib/cron-jobs/update-asn-data, contrib/cron-jobs/update-geoip-data, contrib/cron-jobs/update-tor-nodes: Errors produce proper output.
        •  
      "},{"location":"changelog/#core_26","title":"Core","text":"
         
      • lib/bot
        •  
      • use SIGTERM instead of SIGINT to stop bots (#981).
        •  
      • Bots can specify a static method check(parameters) which can perform individual checks specific to the bot.     These functions will be called by intelmqctl check if the bot is configured with the given parameters
        •  
      • top level bot parameters (description, group, module, name) are exposed as members of the class.
        •  
      • The parameter feed for collectors is deprecated for 2.0 and has been replaced by the more consistent name (#1144).
        •  
      • bug: allow path parameter for CollectorBot class.
        •  
      • Handle errors better when the logger could not be initialized.
        •  
      • ParserBot:
           
        • For the csv parsing methods, ParserBot.csv_params is now used for all these methods.
          •  
        • ParserBot.parse_csv_dict now saves the field names in ParserBot.csv_fieldnames.
          •  
        • ParserBot.parse_csv_dict now saves the raw current line in ParserBot.current_line.
          •  
        • ParserBot.recover_line_csv_dict now uses the raw current line.
          •  
         
        •  
      • lib/message:
        •  
      • Subitems in fields of type JSONDict (see below) can be accessed directly. E.g. you can do:     event['extra.foo'] = 'bar'     event['extra.foo'] # gives 'bar'     It is still possible to set and get the field as whole, however this may be removed or changed in the future:     event['extra'] = '{\"foo\": \"bar\"}'     event['extra'] # gives '{\"foo\": \"bar\"}'     \"Old\" bots and configurations compatible with 1.0.x do still work.     Also, the extra field is now properly exploded when exporting events, analogous to all other fields.     The in operator works now for both - the old and the new - behavior.
        •  
      • Message.add: The parameter overwrite accepts now three different values: True, False and None (new).     True: An existing value will be overwritten     False: An existing value will not be overwritten (previously an exception has been raised when the value was given).     None (default): If the value exists an KeyExists exception is thrown (previously the same as False).     This allows shorter code in the bots, as an 'overwrite' configuration parameter can be directly passed to the function.
        •  
      • The message class has now the possibility to return a default value for non-existing fields, see Message.set_default_value.
        •  
      • Message.get behaves the same like Message.__getitem__ (#1305).
        •  
      • Add RewindableFileHandle to utils making handling of CSV files more easy (optionally)
        •  
      • lib/pipeline:
        •  
      • you may now define more than one destination queues path the bot should pass the message to, see Pipelines (#1088, #1190).
        •  
      • the special path \"_on_error\" can be used to pass messages to different queues in case of processing errors (#1133).
        •  
      • lib/harmonization: Accept AS prefix for ASN values (automatically stripped).
        •  
      • added intelmq.VAR_STATE_PATH for variable state data of bots.
        •  
      "},{"location":"changelog/#bots_27","title":"Bots","text":"
         
      • Removed print statements from various bots.
        •  
      • Replaced various occurrences of self.logger.error() + self.stop() with raise ValueError.
        •  
      "},{"location":"changelog/#collectors_21","title":"Collectors","text":"
         
      • bots.collectors.mail:
        •  
      • New parameters; sent_from: filter messages by sender, sent_to: filter messages by recipient
        •  
      • More debug logs
        •  
      • bots.collectors.n6.collector_stomp: renamed to bots.collectors.stomp.collector (#716)
        •  
      • bots.collectors.rt:
        •  
      • New parameter search_requestor to search for field Requestor.
        •  
      • Empty strings and null as value for search parameters are ignored.
        •  
      • Empty parameters attachment_regex and url_regex handled.
        •  
      • bots.collectors.http.collector_http: Ability to optionally use the current time in parameter http_url, added parameter http_url_formatting.
        •  
      • bots.collectors.stomp.collector: Heartbeat timeout is now logged with log level info instead of warning.
        •  
      • added intelmq.bots.collectors.twitter.collector_twitter
        •  
      • added intelmq.bots.collectors.tcp.collector that can be bound to another IntelMQ instance by a TCP output
        •  
      • bots.collectors.microsoft.collector_interflow: added for MS interflow API
        •  
      • Automatic ungzipping for .gz files.
        •  
      • added intelmq.bots.collectors.calidog.collector_certstream for collecting certstream data (#1120).
        •  
      • added intelmq.bots.collectors.shodan.collector_stream for collecting shodan stream data (#1096).
        •  
      • Add proxy support.
        •  
      • Fix handling of parameter countries.
        •  
      "},{"location":"changelog/#parsers_25","title":"Parsers","text":"
         
      • bots.parsers.shadowserver:
        •  
      • changed feednames. Please refer to it's README for the exact changes.
        •  
      • If the conversion function fails for a line, an error is raised and the offending line will be handled according to the error handling configuration.     Previously errors like these were only logged and ignored otherwise.
        •  
      • add support for the feeds
           
        • Accessible-Hadoop (#1231)
          •  
        • Accessible ADB (#1285)
          •  
         
        •  
      • Remove deprecated parameter override, use overwrite instead (#1071).
        •  
      • The raw values now are exactly the input with quotes unchanged, the ParserBot methods are now used directly (#1011).
        •  
      • The Generic CSV Parser bots.parsers.generic.parser_csv:
        •  
      • It is possible to filter the data before processing them using the new parameters filter_type and filter_text.
        •  
      • It is possible to specify multiple columns using | character in parameter columns.
        •  
      • The parameter time_format now supports 'epoch_millis' for seconds since the Epoch, milliseconds are supported but not used.
        •  
      • renamed bots.parsers.cymru_full_bogons.parser to bots.parsers.cymru.parser_full_bogons, compatibility shim will be removed in version 2.0
        •  
      • added bots.parsers.cymru.parser_cap_program
        •  
      • added intelmq.bots.parsers.zoneh.parser for ZoneH feeds
        •  
      • added intelmq.bots.parsers.sucuri.parser
        •  
      • added intelmq.bots.parsers.malwareurl.parser
        •  
      • added intelmq.bots.parsers.threatminer.parser
        •  
      • added intelmq.bots.parsers.webinspektor.parser
        •  
      • added intelmq.bots.parsers.twitter.parser
        •  
      • added intelmq.bots.parsers.microsoft.parser_ctip
        •  
      • ignore the invalid IP '0.0.0.0' for the destination
        •  
      • fix the raw/dumped messages, did not contain the paling list previously.
        •  
      • use the new harmonization field tlp instead of extra.tlp.
        •  
      • bots.parsers.alienvault.parser_otx: Save TLP data in the new harmonization field tlp.
        •  
      • added intelmq.bots.parsers.openphish.parser_commercial
        •  
      • added intelmq.bots.parsers.microsoft.parser_bingmurls
        •  
      • added intelmq.bots.parsers.calidog.parser_certstream for parsing certstream data (#1120).
        •  
      • added intelmq.bots.parsers.shodan.parser for parsing shodan data (#1096).
        •  
      • change the classification type from 'botnet drone' to 'infected system' in various parses.
        •  
      • intelmq.bots.parsers.spamhaus.parser_cert: Added support for all known bot types.
        •  
      "},{"location":"changelog/#experts_26","title":"Experts","text":"
         
      • Added sieve expert for filtering and modifying events (#1083)
        •  
      • capable of distributing the event to appropriate named queues
        •  
      • bots.experts.modify
        •  
      • default rulesets: all malware name mappings have been migrated to the Malware Name Mapping repository ruleset. See the new added contrib tool for download and conversion.
        •  
      • new parameter case_sensitive (default: True)
        •  
      • Added wait expert for sleeping
        •  
      • Added domain suffix expert to extract the TLD/Suffix from a domain name.
        •  
      • bots.experts.maxmind_geoip: New (optional) parameter overwrite, by default false. The current default was to overwrite!
        •  
      • intelmq.bots.experts.ripencc_abuse_contact:
        •  
      • Extend deprecated parameter compatibility query_ripe_stat until 2.0 because of a logic bug in the compatibility code, use query_ripe_stat_asn and query_ripe_stat_ip instead (#1071, #1291).
        •  
      • Handle HTTP status code 404 for DB AS queries.
        •  
      • Add caching capability.
        •  
      • intelmq/bots/experts/asn_lookup/update-asn-data: Errors produce proper output on stdout/stderr.
        •  
      • intelmq/bots/experts/maxmind_geoip/update-geoip-data: Errors produce proper output on stdout/stderr.
        •  
      • intelmq/bots/experts/tor_nodes/update-tor-nodes: Errors produce proper output on stdout/stderr.
        •  
      "},{"location":"changelog/#outputs_22","title":"Outputs","text":"
         
      • bots.outputs.file:
        •  
      • String formatting can be used for file names with new parameter format_filename.
        •  
      • New parameter single_key to only save one field.
        •  
      • New parameter encoding_errors_mode with default value 'strict' to handle encoding errors for the files written.
        •  
      "},{"location":"changelog/#harmonization_7","title":"Harmonization","text":"
         
      • Renamed JSON to JSONDict and added a new type JSON. JSONDict saves data internally as JSON, but acts like a dictionary. JSON accepts any valid JSON.
        •  
      • fixed regex for protocol.transport it previously allowed more values than it should have.
        •  
      • New ASN type. Like integer but checks the range.
        •  
      • added destination.urlpath and source.urlpath to harmonization.
        •  
      • New field tlp for tlp level specification.
        •  
      • New TLP type. Allows all four tlp levels, removes 'TLP:' prefix and converts to upper case.
        •  
      • Added new classification.type 'vulnerable client'
        •  
      • Added (destination|source).domain_suffix to hold the TLD/domain suffix.
        •  
      • New allowed value for classification.type: infected system for taxonomy malicious code (#1197).
        •  
      "},{"location":"changelog/#requirements_1","title":"Requirements","text":"
         
      • Requests is no longer listed as dependency of the core. For depending bots the requirement is noted in their REQUIREMENTS.txt file.
        •  
      "},{"location":"changelog/#documentation_24","title":"Documentation","text":"
         
      • Use Markdown for README again, as pypi now supports it.
        •  
      • Developers Guide: Add instructions for pre-release testing.
        •  
      "},{"location":"changelog/#packaging_15","title":"Packaging","text":"
         
      • Add logcheck configuration to the packages.
        •  
      • Fix packaging of bash completion script.
        •  
      "},{"location":"changelog/#tests_24","title":"Tests","text":"
         
      • Travis now correctly stops if a requirement could not be installed (#1257).
        •  
      • New tests for validating etc/feeds.yaml and bots/BOTS using cerberus and schemes are added (#1166).
        •  
      • New test for checking if docs/Feeds.md is up to date with etc/feeds.yaml.
        •  
      "},{"location":"changelog/#known-bugs","title":"Known bugs","text":"
         
      • contrib: feeds-config-generator does not add feed name as parameter (#1314).
        •  
      • bot debugger requires configured source pipeline (#1307).
        •  
      • shadowserver parser: drone feed has spam events (#1271).
        •  
      • debug log level on python 3.7 not applied (#1269).
        •  
      • bots.experts.sieve does not support textX (#1246).
        •  
      • Bots started with IntelMQ-Manager stop when the webserver is restarted (#952).
        •  
      "},{"location":"changelog/#106-bugfix-release-2018-08-31","title":"1.0.6 Bugfix release (2018-08-31)","text":""},{"location":"changelog/#bots_28","title":"Bots","text":""},{"location":"changelog/#collectors_22","title":"Collectors","text":"
         
      • bots.collectors.rt.collector_rt: Log ticket id for downloaded reports.
        •  
      "},{"location":"changelog/#parsers_26","title":"Parsers","text":"
         
      • bots.parsers.shadowserver:
        •  
      • if required fields do not exist in data, an exception is raised, so the line will be dumped and not further processed.
        •  
      • fix a bug in the parsing of column cipher_suite in ssl poodle reports (#1288).
        •  
      "},{"location":"changelog/#experts_27","title":"Experts","text":"
         
      • Reverse DNS Expert: ignore all invalid results and use first valid one (#1264).
        •  
      • intelmq/bots/experts/tor_nodes/update-tor-nodes: Use check.torproject.org as source as internet2.us is down (#1289).
        •  
      "},{"location":"changelog/#outputs_23","title":"Outputs","text":"
         
      • bots.output.amqptopic:
        •  
      • The default exchange must not be declared (#1295).
        •  
      • Unencodable characters are prepended by backslashes by default. Otherwise Unicode characters can't be encoded and sent (#1296).
        •  
      • Gracefully close AMQP connection on shutdown of bot.
        •  
      "},{"location":"changelog/#documentation_25","title":"Documentation","text":"
         
      • Bots: document redis cache parameters.
        •  
      • Installation documentation: Ubuntu needs universe repositories.
        •  
      "},{"location":"changelog/#packaging_16","title":"Packaging","text":"
         
      • Dropped support for Ubuntu 17.10, it reached its End of Life as of 2018-07-19.
        •  
      "},{"location":"changelog/#tests_25","title":"Tests","text":"
         
      • Drop tests for Python 3.3 for the mode with all requirements, as some optional dependencies do not support Python 3.3 anymore.
        •  
      • lib.test: Add parameter compare_raw (default: True) to assertMessageEqual, to optionally skip the comparison of the raw field.
        •  
      • Add tests for RT collector.
        •  
      • Add tests for Shadowserver Parser:
        •  
      • SSL Poodle Reports.
        •  
      • Helper functions.
        •  
      "},{"location":"changelog/#tools_23","title":"Tools","text":"
         
      • intelmqctl list now sorts the output of bots and queues (#1262).
        •  
      • intelmqctl: Correctly handle the corner cases with collectors and outputs for getting/sending messages in the bot debugger (#1263).
        •  
      • intelmqdump: fix ordering of dumps in a file in runtime. All operations are applied to a sorted list (#1280).
        •  
      "},{"location":"changelog/#contrib_14","title":"Contrib","text":"
         
      • cron-jobs/update-tor-nodes: Use check.torproject.org as source as internet2.us is down (#1289).
        •  
      "},{"location":"changelog/#known-issues_25","title":"Known issues","text":"
         
      • shadowserver parser: drone feed has spam events (#1271).
        •  
      "},{"location":"changelog/#105-bugfix-release-2018-06-21","title":"1.0.5 Bugfix release (2018-06-21)","text":""},{"location":"changelog/#core_27","title":"Core","text":"
         
      • lib/message: Report() can now create a Report instance from Event instances (#1225).
        •  
      • lib/bot:
        •  
      • The first word in the log line Processed ... messages since last logging. is now adaptable and set to Forwarded in the existing filtering bots (#1237).
        •  
      • Kills oneself again after proper shutdown if the bot is XMPP collector or output (#970). Previously these two bots needed two stop commands to get actually stopped.
        •  
      • lib/utils: log: set the name of the py.warnings logger to the bot name (#1184).
        •  
      "},{"location":"changelog/#harmonization_8","title":"Harmonization","text":"
         
      • Added new types unauthorized-command and unauthorized-login to intrusions taxonomy.
        •  
      "},{"location":"changelog/#bots_29","title":"Bots","text":""},{"location":"changelog/#collectors_23","title":"Collectors","text":"
         
      • bots.collectors.mail.collector_mail_url: handle empty downloaded reports (#988).
        •  
      • bots.collectors.file.collector_file: handle empty files (#1244).
        •  
      "},{"location":"changelog/#parsers_27","title":"Parsers","text":"
         
      • Shadowserver parser:
        •  
      • SSL FREAK: Remove optional column device_serial and add several new ones.
        •  
      • Fixed HTTP URL parsing for multiple feeds (#1243).
        •  
      • Spamhaus CERT parser:
        •  
      • add support for smtpauth, l_spamlink, pop, imap, rdp, smb, iotscan, proxyget, iotmicrosoftds, automatedtest, ioturl, iotmirai, iotcmd, iotlogin and iotuser (#1254).
        •  
      • fix extra.destination.local_port -> extra.source.local_port.
        •  
      "},{"location":"changelog/#experts_28","title":"Experts","text":"
         
      • bots.experts.filter: Pre-compile regex at bot initialization.
        •  
      "},{"location":"changelog/#tests_26","title":"Tests","text":"
         
      • Ensure that the bots did process all messages (#291).
        •  
      "},{"location":"changelog/#tools_24","title":"Tools","text":"
         
      • intelmqctl:
        •  
      • intelmqctl run has a new parameter -l --loglevel to overwrite the log level for the run (#1075).
        •  
      • intelmqctl run [bot-id] message send can now send report messages (#1077).
        •  
      • intelmqdump:
        •  
      • has now command completion for bot names, actions and queue names in interactive console.
        •  
      • automatically converts messages from events to reports if the queue the message is being restored to is the source queue of a parser (#1225).
        •  
      • is now capable to read messages in dumps that are dictionaries as opposed to serialized dicts as strings and does not convert them in the show command (#1256).
        •  
      • truncated messages are no longer used/saved to the file after being shown (#1255).
        •  
      • now again denies recovery of dumps if the corresponding bot is running. The check was broken (#1258).
        •  
      • now sorts the dump by the time of the dump. Previously, the list was in random order (#1020).
        •  
      "},{"location":"changelog/#known-issues_26","title":"Known issues","text":"
      no known issues
      "},{"location":"changelog/#104-bugfix-release-2018-04-20","title":"1.0.4 Bugfix release (2018-04-20)","text":"
         
      • make code style compatible to pycodestyle 2.4.0
        •  
      • fixed permissions of some files (they were executable but shouldn't be)
        •  
      "},{"location":"changelog/#core_28","title":"Core","text":"
         
      • lib/harmonization:
        •  
      • FQDN validation now handles None correctly (raised an Exception).
        •  
      • Fixed several sanitize() methods, the generic sanitation method were called by is_valid, not the sanitize methods (#1219).
        •  
      "},{"location":"changelog/#bots_30","title":"Bots","text":"
         
      • Use the new pypi website at https://pypi.org/ everywhere.
        •  
      "},{"location":"changelog/#parsers_28","title":"Parsers","text":"
         
      • Shadowserver parser:
        •  
      • The fields url and http_url now handle HTTP URL paths and HTTP requests for all feeds (#1204).
        •  
      • The conversion function validate_fqdn now handles empty strings correctly.
        •  
      • Feed 'drone (hadoop)':
           
        • Correct validation of field cc_dns, will now only be added as destination.fqdn if correct FQDN, otherwise ignored. Previously this field could be saved in extra containing an IP address.
          •  
        • Adding more mappings for added columns.
          •  
         
        •  
      • Added feeds:
           
        • Drone-Brute-Force
          •  
        • IPv6-Sinkhole-HTTP-Drone
          •  
         
        •  
      • A lot of newly added fields and fixed conversions.
        •  
      • Optional fields can now use one column multiple times.
        •  
      • Add newly added columns of Ssl-Scan feed to parser
        •  
      • Spamhaus CERT parser:
        •  
      • fix parsing and classification for bot names 'openrelay', 'iotrdp', 'sshauth', 'telnetauth', 'iotcmd', 'iotuser', 'wpscanner', 'w_wplogin', 'iotscan'     see the NEWS file - Postgresql section - for all changes.
        •  
      • CleanMX phishing parser: handle FQDNs in IP column (#1162).
        •  
      "},{"location":"changelog/#experts_29","title":"Experts","text":"
         
      • bots.experts.ripencc_abuse_contact: Add existing parameter mode to BOTS file.
        •  
      "},{"location":"changelog/#tools_25","title":"Tools","text":"
         
      • intelmqctl check: Fixed and extended message for 'run_mode' check.
        •  
      • intelmqctl start botnet. When using --type json, no non-JSON information about wrong bots are output because that would confuse eg. intelmq-manager
        •  
      "},{"location":"changelog/#tests_27","title":"Tests","text":"
         
      • lib/bot: No dumps will be written during tests (#934).
        •  
      • lib/test: Expand regular expression on python version to match pre-releases (debian testing).
        •  
      "},{"location":"changelog/#packaging_17","title":"Packaging","text":"
         
      • Static data is now included in source tarballs, development files are excluded
        •  
      "},{"location":"changelog/#known-issues_27","title":"Known issues","text":"
         
      • bots.collectors/outputs.xmpp must be killed two times (#970).
        •  
      • When running bots with intelmqctl run [bot-id] the log level is always INFO (#1075).
        •  
      • intelmqctl run [bot-id] message send [msg] does only support Events, not Reports (#1077).
        •  
      • A warning issued by the python warnings module is logged without the bot-id (#1184).
        •  
      "},{"location":"changelog/#103-bugfix-release-2018-02-05","title":"1.0.3 Bugfix release (2018-02-05)","text":""},{"location":"changelog/#contrib_15","title":"Contrib","text":"
         
      • logrotate: use sudo for postrotate script
        •  
      • cron-jobs: use the scripts in the bots' directories and link them (#1056, #1142)
        •  
      "},{"location":"changelog/#core_29","title":"Core","text":"
         
      • lib.harmonization: Handle idna encoding error in FQDN sanitation (#1175, #1176).
        •  
      • lib.bot:
        •  
      • Bots stop when redis gives the error \"OOM command not allowed when used memory > 'maxmemory'.\" (#1138).
        •  
      • warnings of bots are caught by the logger (#1074, #1113).
        •  
      • Fixed exitcodes 0 for graceful shutdowns .
        •  
      • better handling of problems with pipeline and especially it's initialization (#1178).
        •  
      • All parsers using ParserBot's methods now log the sum of successfully parsed and failed lines at the end of each run (#1161).
        •  
      "},{"location":"changelog/#harmonization_9","title":"Harmonization","text":"
         
      • Rule for harmonization keys is enforced (#1104, #1141).
        •  
      • New allowed values for classification.type: tor & leak (see n6 parser below ).
        •  
      "},{"location":"changelog/#bots_31","title":"Bots","text":""},{"location":"changelog/#collectors_24","title":"Collectors","text":"
         
      • bots.collectors.mail.collector_mail_attach: Support attachment file parsing for imbox versions newer than 0.9.5 (#1134).
        •  
      "},{"location":"changelog/#parsers_29","title":"Parsers","text":"
         
      • All CSV parsers ignore NULL-bytes now, because the csv-library cannot handle it (#967, #1114).
        •  
      • bots.parsers.shadowserver.parser: Add Accessible Cisco Smart Install (#1122).
        •  
      • bots.parsers.cleanmx.parser: Handle new columns first and last, rewritten for XML feed. See NEWS.md for upgrade instructions (#1131, #1136, #1163).
        •  
      • bots.parsers.n6.parser: Fix classification mappings. See NEWS file for changes values (#738, #1127).
        •  
      "},{"location":"changelog/#experts_30","title":"Experts","text":"
         
      • bots.experts.modify default ruleset: changed conficker rule to catch more spellings.
        •  
      "},{"location":"changelog/#outputs_24","title":"Outputs","text":"
         
      • bots.outputs.smtp.output: Fix STARTTLS, threw an exception (#1152, #1153).
        •  
      "},{"location":"changelog/#documentation_26","title":"Documentation","text":"
         
      • Release.md add release procedure documentation
        •  
      • Bots.md: fix example configuration for modify expert
        •  
      "},{"location":"changelog/#tools_26","title":"Tools","text":"
         
      • intelmqctl now exits with exit codes > 0 when errors happened or the operation was not successful. Also, the status operation exits with 1, if bots are stopped, but enabled. (#977, #1143)
        •  
      • intelmctl check checks for valid run_mode in runtime configuration (#1140).
        •  
      "},{"location":"changelog/#tests_28","title":"Tests","text":"
         
      • tests.lib.test_pipeline: Redis tests clear all queues before and after tests (#1086).
        •  
      • Repaired debian package build on travis (#1169).
        •  
      • Warnings are not allowed by default, an allowed count can be specified (#1129).
        •  
      • tests.bots.experts.cymru_whois/abusix: Skipped on travis because of ongoing problems.
        •  
      "},{"location":"changelog/#packaging_18","title":"Packaging","text":"
         
      • cron jobs: fix paths of executables
        •  
      "},{"location":"changelog/#known-issues_28","title":"Known issues","text":"
         
      • bots.collectors/outputs.xmpp must be killed two times (#970).
        •  
      • When running bots with intelmqctl run [bot-id] the log level is always INFO (#1075).
        •  
      • intelmqctl run [bot-id] message send [msg] does only support Events, not Reports (#1077).
        •  
      • python3 setup.py sdist does not include static files in the resulting tarballs (#1146).
        •  
      • bots.parsers.cleanmx.parser: The cleanMX feed may have FQDNs as IPs in rare cases, such lines are dumped (#1162).
        •  
      "},{"location":"changelog/#102-bugfix-release-2017-11-09","title":"1.0.2 Bugfix release (2017-11-09)","text":""},{"location":"changelog/#core_30","title":"Core","text":"
         
      • lib.message.add: parameter force has finally been removed, should have been gone in 1.0.0.rc1 already
        •  
      "},{"location":"changelog/#bots_32","title":"Bots","text":"
         
      • collectors.mail.collector_mail_url: Fix bug which prevented marking emails seen due to disconnects from server (#852).
        •  
      • parsers.spamhaus.parser_cert: Handle/ignore 'AS?' in feed (#1111)
        •  
      "},{"location":"changelog/#packaging_19","title":"Packaging","text":"
         
      • The following changes have been in effect for the built packages already since version 1.0.0
        •  
      • Support building for more distributions, now supported: CentOS 7, Debian 8 and 9, Fedora 25 and 26, RHEL 7, openSUSE Leap 42.2 and 42.3 and Tumbleweed, Ubuntu 14.04 and 16.04
        •  
      • Use LSB-paths for created packages (/etc/intelmq/, /var/lib/intelmq/, /run/intelmq/) (#470). Does does not affect installations with setuptools/pip.
        •  
      • Change the debian package format from native to quilt
        •  
      • Fix problems in postint and postrm scripts
        •  
      • Use systemd-tmpfile for creation of /run/intelmq/
        •  
      "},{"location":"changelog/#documentation_27","title":"Documentation","text":"
         
      • Add disclaimer on maxmind database in bot documentation and code and the cron-job (#1110)
        •  
      "},{"location":"changelog/#101-bugfix-release-2017-08-30","title":"1.0.1 Bugfix release (2017-08-30)","text":""},{"location":"changelog/#documentation_28","title":"Documentation","text":"
         
      • Feeds: use more https:// URLs
        •  
      • minor fixes
        •  
      "},{"location":"changelog/#bots_33","title":"Bots","text":"
         
      • bots/experts/ripencc_abuse_contact/expert.py: Use HTTPS URLs for rest.db.ripe.net
        •  
      • bots/outputs/file/output.py: properly close the file handle on shutdown
        •  
      • bots/parser/shadowserver: If conversion of a value via conversion function fails, only log the function name, not the representation string (#1157).
        •  
      "},{"location":"changelog/#core_31","title":"Core","text":"
         
      • lib/bot: Bots will now log the used intelmq version at startup
        •  
      "},{"location":"changelog/#tools_27","title":"Tools","text":"
         
      • intelmqctl: To check the status of a bot, the command line of the running process is compared to the actual executable of the bot. Otherwise unrelated programs with the same PID are detected as running bot.
        •  
      • intelmqctl: enable, disable, check, clear now support the JSON output
        •  
      "},{"location":"changelog/#100-stable-release-2017-08-04","title":"1.0.0 Stable release (2017-08-04)","text":""},{"location":"changelog/#core_32","title":"Core","text":"
         
      • Fixes a thrown FileNotFound exception when stopping bots started with intelmqctl run ...
        •  
      "},{"location":"changelog/#harmonization_10","title":"Harmonization","text":"
         
      • leading dots in FQDNs are rejected and removed in sanitation (#1022, #1030)
        •  
      "},{"location":"changelog/#bots_34","title":"Bots","text":"
         
      • shadowserver parser Accessible-SMB: smb_implant is converted to bool
        •  
      "},{"location":"changelog/#100rc1-release-candidate-2017-07-05","title":"1.0.0.rc1 Release candidate (2017-07-05)","text":""},{"location":"changelog/#core_33","title":"Core","text":"
         
      • Changing the value of an existing field to None deletes the field.
        •  
      • Message.update now behaves like dict.update. The old behavior is implemented in Message.change
        •  
      • Deprecated http_ssl_proxy has been dropped, use https_proxy instead
        •  
      • Deprecated http_timeout has been dropped, use http_timeout_sec instead
        •  
      • Deprecated parameters force and ignore of Message.add have been removed
        •  
      • Deprecated method Message.contains has been removed
        •  
      • Drop support for deprecated configuration files startup.conf and system.conf
        •  
      "},{"location":"changelog/#development_11","title":"Development","text":"
         
      • We are now testing with and without optional libraries/lowest recommended versions and most current versions of required libraries
        •  
      • Tests shadowserver with more data and checks for warnings and errors
        •  
      • Tests: if bots log warnings this counts as failure if not allowed explicitly
        •  
      • Tests: Bot preparation can be skipped
        •  
      "},{"location":"changelog/#documentation_29","title":"Documentation","text":"
         
      • The branching/releasing mechanism has been documented
        •  
      "},{"location":"changelog/#bots_35","title":"Bots","text":""},{"location":"changelog/#collectors_25","title":"Collectors","text":"
         
      • HTTP collectors: If http_username and http_password are both given and empty or null, 'None:None' has been used to authenticate. It is now checked that the username evaluates to non-false/null before adding the authentication. (fixes #1017)
        •  
      • Dropped unmaintained and undocumented FTP(S) collectors bots.collectors.ftp. Also, the FTPS collector had a license conflict (#842).
        •  
      • bots.collectors.http.collector_http_stream: drop deprecated parameter url in favor of http_url
        •  
      "},{"location":"changelog/#parsers_30","title":"Parsers","text":"
         
      • Removed bots.parsers.openbl as the source is offline since end of may (#1018, https://twitter.com/sshblorg/status/854669263671615489)
        •  
      • Removed bots.parsers.proxyspy as the source is offline (#1031)
        •  
      • Shadowserver: Added Accessible SMB
        •  
      • bots.experts.ripencc_abuse_contact now has the two additional parameters query_ripe_stat_asn and query_ripe_stat_ip.   Deprecated parameter query_ripe_stat. New parameter mode.
        •  
      • bots.experts.certat_contact has been renamed to bots.experts.national_cert_contact_certat (#995)
        •  
      • bots.experts.cymru_whois ignores registry other (#996)
        •  
      • bots.parsers.alienvault.parser_otx: handle timestamps without floating point seconds
        •  
      "},{"location":"changelog/#experts_31","title":"Experts","text":"
         
      • bots.experts.deduplicator: New parameter bypass to deactivate deduplication, default: False
        •  
      "},{"location":"changelog/#v100dev8-beta-release-2017-06-14","title":"v1.0.0.dev8 Beta release (2017-06-14)","text":""},{"location":"changelog/#general-changes","title":"General changes","text":"
         
      • It's now configurable how often the bots are logging how much events they have sent, based on both the amount and time. (fixes #743)
        •  
      • switch from pycodestyle to pep8
        •  
      "},{"location":"changelog/#configuration_7","title":"Configuration","text":"
         
      • Added log_processed_messages_count (500) and log_processed_messages_seconds (900) to defaults.conf.
        •  
      • http_timeout has been renamed to http_timeout_sec and http_timeout_max_tries has been added.    This setting is honored by bots.collectors.http.* and bots.collectors.mail.collector_mail_url, bots.collectors.rt (only http_timeout_sec), bots.outputs.restapi.output and bots.experts.ripencc_abuse_contact.
        •  
      "},{"location":"changelog/#documentation_30","title":"Documentation","text":"
         
      • Minor fixes
        •  
      • Dropped install scripts, see INSTALL.md for more detailed instructions and explanations
        •  
      • Better structure of INSTALL.md
        •  
      • Better documentation of packages
        •  
      "},{"location":"changelog/#tools_28","title":"Tools","text":"
         
      • added a bot debugger (#975)
        •  
      • missing bot executable is detected and handled by intelmqctl (#979)
        •  
      "},{"location":"changelog/#core_34","title":"Core","text":"
         
      • fix bug which prevented dumps to be written if the file did not exist (#986)
        •  
      • Fix reload of bots regarding logging
        •  
      • type annotations for all core libraries
        •  
      "},{"location":"changelog/#bots_36","title":"Bots","text":"
         
      • added bots.experts.idea, bots.outputs.files
        •  
      • possibility to split large csv Reports into Chunks, currently possible for mail url and file collector
        •  
      • elasticsearch output supports HTTP Basic Auth
        •  
      • bots.collectors.mail.collector_mail_url and bots collectors.file.collector can split large reports (#680)
        •  
      • bots.parsers.shadowserver support the VNC feed
        •  
      • handling of HTTP timeouts, see above #859
        •  
      • bots.parsers.bambenek saves the malware name
        •  
      • bots.parsers.fraunhofer.parser_dga saves the malware name
        •  
      • bots.parsers.shadowserver handles NULL bytes
        •  
      • bots.parsers.abusech.parser_ransomware handles the IP 0.0.0.0 specially
        •  
      "},{"location":"changelog/#harmonization_11","title":"Harmonization","text":"
         
      • New field named output to support export to foreign formats
        •  
      "},{"location":"changelog/#v100dev7-beta-release-2017-05-09","title":"v1.0.0.dev7 Beta release (2017-05-09)","text":""},{"location":"changelog/#documentation_31","title":"Documentation","text":"
         
      • more verbose installation and upgrade instructions
        •  
      "},{"location":"changelog/#bots_37","title":"Bots","text":""},{"location":"changelog/#collectors_26","title":"Collectors","text":"
         
      • bots.collectors.alienvault_otx: OTX library has been removed, install it as package instead
        •  
      "},{"location":"changelog/#parsers_31","title":"Parsers","text":"
         
      • API keys will be removed from feed.url if possible
        •  
      • intelmq.bots.parsers.shadowserver.config:
        •  
      • Added support for Compromised-Website, Open-Netis, NTP-Version, Sandbox-URL, Spam-URL, Vulnerable-ISAKMP, Botnet-CCIP, Accessible-RDP, Open-LDAP, Blacklisted-IP, Accessible-Telnet, Accessible-CWMP (#748).
        •  
      "},{"location":"changelog/#experts_32","title":"Experts","text":"
         
      • added bots.experts.field_reducer, bots.outputs.smtp.
        •  
      • bots.experts.deduplicator: ignore_keys has been renamed to filter_keys and filter_type has been removed.
        •  
      • bots.experts.modify: The configuration is now list-based for a consistent ordering.
        •  
      • bots.experts.tor_node as an optional parameter overwrite.
        •  
      "},{"location":"changelog/#harmonization_12","title":"Harmonization","text":"
         
      • New parameter and field named feed.documentation to link to documentation of the feed
        •  
      • classification.taxonomy is lower case only
        •  
      "},{"location":"changelog/#v100dev6-beta-release-2017-01-11","title":"v1.0.0.dev6 Beta release (2017-01-11)","text":"
      Changes between 0.9 and 1.0.0.dev6
      "},{"location":"changelog/#general-changes_1","title":"General changes","text":"
         
      • Dropped support for Python 2, Python >= 3.3 is needed
        •  
      • Dropped startup.conf and system.conf. Sections in BOTS can be copied directly to runtime.conf now.
        •  
      • Support two run modes: 'stream' which is the current implementation and a new one 'scheduled' which allows scheduling via cron or systemd.
        •  
      • Helper classes for parser bots
        •  
      • moved intelmq/conf to intelmq/etc
        •  
      • cleanup in code and repository
        •  
      • All bots capable of reloading on SIGHUP
        •  
      • packages
        •  
      • pip wheel format instead of eggs
        •  
      • unittests for library and bots
        •  
      • bots/BOTS now contains only generic and specific collectors. For a list of feeds, see docs/Feeds.md
        •  
      "},{"location":"changelog/#tools_29","title":"Tools","text":"
         
      • DEV: intelmq_gen_harm_docs: added to generate Harmonization documentation
        •  
      • intelmq_psql_initdb: creates a table for a postgresql database using the harmonization fields
        •  
      • intelmqctl: reworked argument parsing, many bugfixes
        •  
      • intelmqdump: added to inspect dumped messages and reinsert them into the queues
        •  
      • DEV: rewrite_config_files: added to rewrite configuration files with consistent style
        •  
      "},{"location":"changelog/#bots_38","title":"Bots","text":""},{"location":"changelog/#collectors_27","title":"Collectors","text":"
         
      • added alienvault, alienvault otx, bitsight, blueiv, file, ftp, misp, n6, rtir, xmpp collector
        •  
      • removed hpfeeds collector
        •  
      • removed microsoft DCU collector
        •  
      • renamed and reworked URL collector to HTTP
        •  
      • reworked Mail collectors
        •  
      "},{"location":"changelog/#parsers_32","title":"Parsers","text":"
         
      • source specific parsers added: abusech, alienvault, alienvault otx, anubisnetworks, autoshun, bambenek, bitcash, bitsight, blocklistde, blueliv, ci army, cleanmx, cymru_full_bogons, danger_rulez, dataplane, dshield (asn, block and domain), dyn, fraunhofer_dga, hphosts, malc0de, malwaredomains, misp, n6, netlab_360, nothink, openphish, proxyspy, spamhaus cert, taichung, turris, urlvir
        •  
      • generic parsers added: csv, json
        •  
      • specific parsers dropped: abusehelper (broken), arbor (source unavailable), bruteforceblocker, certeu, dragonresearchgroup parser (discontinued), hpfeeds, microsoft_dcu (broken), taichungcitynetflow, torexitnode parser
        •  
      • renamed intelmq.bots.parsers.spamhaus.parser to intelmq.bots.parsers.spamhaus.parser_drop.   renamed intelmq.bots.parsers.malwarepatrol.parser-dansguardian tointelmq.bots.parsers.malwarepatrol.parser_dansguardian`
        •  
      • renamed intelmq.bots.parsers.taichungcitynetflow.parser tointelmq.bots.parsers.taichung.parser`
        •  
      • major rework of shadowserver parsers
        •  
      • enhanced all parsers
        •  
      "},{"location":"changelog/#experts_33","title":"Experts","text":"
         
      • Added experts: asnlookup, cert.at contact lookup, filter, generic db lookup, gethostbyname, modify, reverse dns, rfc1918, tor_nodes, url2fqdn
        •  
      • removed experts: contactdb, countrycodefilter (obsolete), sanitizer (obsolete)
        •  
      • renamed intelmq.bots.experts.abusix.abusix to intelmq.bots.experts.abusix.expert intelmq.bots.experts.asnlookup.asnlookup to intelmq.bots.experts.asn_lookup.expert intelmq.bots.experts.cymru.expert to intelmq.bots.experts.cymru_whois.expert intelmq.bots.experts.deduplicator.deduplicator to intelmq.bots.experts.deduplicator.expert intelmq.bots.experts.geoip.geopip to intelmq.bots.experts.maxmind_geoip.expert intelmq.bots.experts.ripencc.ripencc to intelmq.bots.experts.ripencc_abuse_contact.expert intelmq.bots.experts.taxonomy.taxonomy to intelmq.bots.experts.taxonomy.expert
        •  
      • enhanced all experts
        •  
      • changed configuration syntax for intelmq.bots.experts.modify to a more simple variant
        •  
      "},{"location":"changelog/#outputs_25","title":"Outputs","text":"
         
      • added: amqp, elasticsearch, redis, restapi, smtp, stomp, tcp, udp, xmpp
        •  
      • removed: debug, intelmqmailer (broken), logcollector
        •  
      • enhanced all outputs
        •  
      "},{"location":"changelog/#bug-fixes","title":"Bug fixes","text":"
         
      • FIX: all bots handle message which are None
        •  
      • FIX: various encoding issues resolved in core and bots
        •  
      • FIX: time.observation is generated in collectors, not in parsers
        •  
      "},{"location":"changelog/#other-enhancements-and-changes","title":"Other enhancements and changes","text":"
         
      • TST: testing framework for core and tests. Newly introduced components should always come with proper unit tests.
        •  
      • ENH: intelmqctl has shortcut parameters and can clear queues
        •  
      • STY: code obeys PEP8, new code should always be properly formatted
        •  
      • DOC: Updated user and dev guide
        •  
      • Removed Message.contains, Message.update methods Message.add ignore parameter
        •  
      "},{"location":"changelog/#configuration_8","title":"Configuration","text":"
         
      • ENH: New parameter and field named accuracy to represent the accuracy of each feed
        •  
      • Consistent naming \"overwrite\" to switch overwriting capabilities of bots (as opposed to override)
        •  
      • Renamed http_ssl_proxy to https_proxy
        •  
      • parameter hierarchical_output for many output bots
        •  
      • deduplicator bot has a new required parameter to configure deduplication mode filter_type
        •  
      • deduplicator bot key ignore_keys was renamed to filter_keys
        •  
      • The tor_nodes expert has a new parameter overwrite, which is by default false.
        •  
      "},{"location":"changelog/#harmonization_13","title":"Harmonization","text":"
         
      • ENH: Additional data types: integer, float and Boolean
        •  
      • ENH: Added descriptions and matching types to all fields
        •  
      • DOC: harmonization documentation has same fields as configuration, docs are generated from configuration
        •  
      • BUG: FQDNs are only allowed in IDN representation
        •  
      • ENH: Removed UUID Type (duplicate of String)
        •  
      • ENH: New type LowercaseString and UppercaseString, doing automatic conversion
        •  
      • ENH: Removed UUID Type (duplicate of String)
        •  
      • ENH: FQDNs are converted to lowercase
        •  
      • ENH: regex, iregex and length checks when data is added to messages
        •  
      "},{"location":"changelog/#most-important-changes","title":"Most important changes:","text":"
         
      • (source|destination).bgp_prefix is now (source|destination).network
        •  
      • (source|destination).cc is now (source|destination).geolocation.cc
        •  
      • (source|destination).reverse_domain_name is (source|destination).reverse_dns
        •  
      • (source|destination).abuse_contact is lower case only
        •  
      • misp_id changed to misp.event_uuid
        •  
      • protocol.transport added, a fixed list of values is allowed
        •  
      • protocol.application is lower case only
        •  
      • webshot_url is now screenshot_url
        •  
      • additional_information renamed to extra, must be JSON
        •  
      • os.name, os.version, user_agent removed in favor of extra
        •  
      • all hashes are lower case only
        •  
      • added malware.hash.(md5|sha1|sha256), removed malware.hash
        •  
      • New parameter and field named feed.accuracy to represent the accuracy of each feed
        •  
      • New parameter and field named feed.provider to document the name of the source of each feed
        •  
      • New field classification.identifier -classification.taxonomy is now lower case only
        •  
      "},{"location":"changelog/#known-issues_29","title":"Known issues","text":"
         
      • Harmonization: hashes are not normalized and classified, see also issue #394 and pull #634
        •  
      "},{"location":"changelog/#contrib_16","title":"Contrib","text":"
         
      • ansible and vagrant scripts added
        •  
      • bash-completion for shells add
        •  
      • cron job scripts to update lookup data added
        •  
      • logcheck example rules added
        •  
      • logrotate configuration added
        •  
      "},{"location":"changelog/#20160618","title":"2016/06/18","text":"
         
      • improvements in pipeline:
        •  
      • PipelineFactory to give possibility to easily add a new broker (Redis, ZMQ, etc..)
        •  
      • Splitter feature: if this option is enable, will split the events in source queue to multiple destination queues
        •  
      • add different messages support:
        •  
      • the system is flexible to define a new type of message like 'tweet' without change anything in bot.py, pipeline.py. Just need to add a new class in message.py and harmonization.conf
        •  
      • add harmonization support
        •  
      • in harmonization.conf is possible to define the fields of a specific message in json format.
        •  
      • the harmonization.py has data types witch contains sanitize and validation methods that will make sure that the values are correct to be part of an event.
        •  
      • Error Handling
        •  
      • multiple parameters in configuration which gives possibility to define how bot will handle some errors. Example of parameters:
        •  
      • error_procedure - retry or pass in case of error
        •  
      • error_retry_delay - time in seconds to retry
        •  
      • error_max_retries - number of retries
        •  
      • error_log_message - log or not the message in error log
        •  
      • error_log_exception - log or not the exception in error log
        •  
      • error_dump_message - log or not the message in dump log to be fixed and re-insert in pipeline
        •  
      • Exceptions
        •  
      • custom exceptions for IntelMQ
        •  
      • Defaults configurations
        •  
      • new configuration file to specify the default parameters which will be applied to all bots. Bots can overwrite the configurations.
        •  
      • New bots/feeds
        •  
      "},{"location":"changelog/#20150603-aaron","title":"2015/06/03 (aaron)","text":"
         
      • fixed the license to AGPL in setup.py
        •  
      • moved back the documentation from the wiki repo to docs/. See #205.
        •  
      • added python-zmq as a setup requirement in UserGuide . See #206
        •  
      "},{"location":"community/","title":"Community","text":""},{"location":"community/#intelmq-organizational-structure","title":"IntelMQ Organizational Structure","text":"
      The central IntelMQ components are maintained by multiple people and organizations in the IntelMQ community. Please note that some components of the IntelMQ Universe can have a different project governance, but all are part of the IntelMQ universe and community.
      "},{"location":"community/#intelmq-enhancement-proposals-iep","title":"IntelMQ Enhancement Proposals (IEP)","text":"
      Major changes, including architecture, strategy and the internal data format, require so-called IEPs, IntelMQ Enhancement Proposals. Their name is based on the famous \"PEPs\" of Python.
       
      IEPs are collected in the separate IEP Repository.
      "},{"location":"community/#code-reviews-and-merging","title":"Code-Reviews and Merging","text":"
      Every line of code checked in for the IntelMQ Core, is checked by at least one trusted developer (excluding the author of the changes) of the IntelMQ community. Afterwards, the code can be merged. Currently, these three contributors, have the permission to push and merging code to IntelMQ Core, Manager and API:
       
         
      • Aaron Kaplan (aaronkaplan)
        •  
      • Sebastian Wagner (sebix)
        •  
      • Sebastian Waldbauer (waldbauer-certat)
        •  
       
      Additionally, these people significantly contributed to IntelMQ:
       
         
      • Bernhard Reiter
        •  
      • Birger Schacht
        •  
      • Edvard Rejthar
        •  
      • Filip Pokorn\u00fd
        •  
      • Karl-Johan Karlsson
        •  
      • Marius Karotkis
        •  
      • Marius Urkus
        •  
      • Mikk Margus M\u00f6ll
        •  
      • navtej
        •  
      • Pavel K\u00e1cha
        •  
      • Robert \u0160efr
        •  
      • Tomas Bellus
        •  
      • Zach Stone
        •  
      "},{"location":"community/#short-history","title":"Short history","text":"
      In 2013 and 2014 Aaron Kaplan (back then working at CERT.at) was researching ways to improve the automation of handling and distributing (IT security) incident reports across a whole country as part of the job of a national CERT. We would get many notifications of vulnerable systems, hacked systems, phishing domains, etc etc. The amount of reports we were getting required an automated solution. Back then, Aaron and a couple of other people looked at a tool called \"Abusehelper\". There was an open source version of Abusehelper, but it was deemed quite complex and complicated at that time.
       
      Frustration with this tool led to discussions amongst multiple CERTs.
       
      The idea and overall concept of an free, truly open source, simple (KISS principle! Keep it simple, stupid) community owned and maintained, extendible software for automated incident handling was born at an meeting of several European CSIRTs in Heraklion, Greece, in 2014. Following the event, Tom\u00e1s Lima \"SYNchroACK\" (working at CERT.pt back then) created IntelMQ from scratch. IntelMQ was born on June 24th, 2014. A major support came from CERT.pt at this early stage. Aaron Kaplan (CERT.at until 2020) engaged in the long-term advancement and from 2015 on, CERT.at took the burden of the maintenance and development (Sebastian Wagner 2015-2021 at CERT.at). From 2016 onward, CERT.at started projects, initiated and lead by Aaron Kaplan, receiving CEFF-funding from the European Union to support IntelMQ's development. IntelMQ became a software component of the EU-funded MeliCERTes framework for CSIRTs. In 2020, IntelMQ's organizational structure and architectural development gained new thrive by the newly founded Board and the start of the IEP process, creating more structure and more transparency in the IntelMQ community's decisions.
      "},{"location":"help/","title":"Help","text":""},{"location":"help/#getting-help","title":"Getting help","text":"
      In case you are lost, you need assistance or something is not discussed in this guide, you can ask the community for help. To be most efficient in seeking help, please describe your problem or question with all necessary information, for example:
       
         
      • Name and version of the operating system
        •  
      • Way of installation (deb/rpm packages, PyPI, docker, local git repository)
        •  
      • Used bots and configuration
        •  
      • Logs of bots or terminal output
        •  
      • Any other useful messages, screenshots
        •  
       
      Please report any errors and suggest improvements via issues. Thank you!
      "},{"location":"help/#github","title":"GitHub","text":"
      GitHub offers a discussion platform where you can ask questions and seek assistance.
       
      To report bugs, GitHub issues are the ideal place to do so. Every IntelMQ component has it's own repository on GitHub, with a separate Issue tracker.
       
      To participate on GitHub, you first need to create an account on the platform.
      "},{"location":"help/#mailing-list","title":"Mailing list","text":"
      The most traditional way is to ask your question, make a proposal or discuss a topic on the mailing IntelMQ Users mailing list. You need to subscribe to the mailing list before posting, but the archive is publicly available: IntelMQ Users Archive.
      "},{"location":"help/#assistance","title":"Assistance","text":"
      If your organisation is a member of the CSIRTs Network, you are eligible for support in the MeliCERTes project. You can also ask on for individual support, some members offer support, including, but not limited to:
       
         
      • Aaron Kaplan (founder of IntelMQ)
        •  
      • Institute for Common Good Technology (chairmen Sebastian Wager is an IntelMQ maintainer and developer)
        •  
      • Intevation GmbH (Develops and maintains several IntelMQ components)
        •  
      "},{"location":"overview/","title":"Overview","text":""},{"location":"overview/#overview","title":"Overview","text":"
      The complete IntelMQ universe consists of the following components:
       
         
      • IntelMQ
        •  
      • IntelMQ API
        •  
      • IntelMQ Manager
        •  
      • additional tools
        •  
      • useful scripts
        •  
      "},{"location":"overview/#intelmq","title":"IntelMQ","text":"
      This project contains the core functionality.
       
      The Core includes all the components required for processing data feeds. This includes the bots, configuration, pipeline, the internal data format, management tools etc.
       
      \u2192 Repository: IntelMQ
      "},{"location":"overview/#intelmq-api","title":"IntelMQ API","text":"
      This is an extension of IntelMQ providing hug based REST API for remote management.
       
      \u2192 Repository: IntelMQ API
      "},{"location":"overview/#intelmq-manager","title":"IntelMQ Manager","text":"
      The Manager is the most known software and can be seen as the face of IntelMQ. It's goal is to provide an intuitive web interface to allow non-programmers to specify the data flow in IntelMQ.
       
      \u2192 Repository: IntelMQ Manager
       
      "},{"location":"overview/#additional-tools","title":"Additional tools","text":"
      Here you can find a list of additional tools. If you think something is missing, please let us know!
       
      Unless stated otherwise, the tools are maintained by the IntelMQ community.
      "},{"location":"overview/#intelmq-webinput-csv","title":"IntelMQ Webinput CSV","text":"
      A web-based interface to ingest CSV data into IntelMQ with on-line validation and live feedback.
       
      This interface allows inserting \"one-shot\" data feeds into IntelMQ without the need to configure bots in IntelMQ.
       
      Developed and maintained by CERT.at.
       
      \u2192 Repository: intelmq-webinput-csv
       
      "},{"location":"overview/#intelmq-mailgen","title":"IntelMQ Mailgen","text":"
      A solution allowing an IntelMQ setup with a complex contact database, managed by a web interface and sending out aggregated email reports. In different words: To send grouped notifications to network owners using SMTP.
       
      Developed and maintained by Intevation, initially funded by BSI.
       
      It consists of the following three components, which can also be used on their own.
      "},{"location":"overview/#intelmq-certbund-contact","title":"IntelMQ CertBUND Contact","text":"
      The certbund-contact consists of two IntelMQ expert bots, which fetch and process the information from the contact database, and scripts to import RIPE data into the contact database. Based on user-defined rules, the experts determine to which contact the event is to be sent to, and which e-mail template and attachment format to use.
       
      \u2192 Repository: intelmq-certbund-contact
      "},{"location":"overview/#intelmq-fody","title":"IntelMQ Fody","text":"
      Fody is a web based interface for Mailgen. It allows to read and edit contacts, query sent mails (tickets) and call up data from the PostgreSQL database.
       
      It can also be used to just query the database without using Mailgen.
       
       
      \u2192 Repository: intelmq-fody
       
      \u2192 Repository: intelmq-fody-backend
      "},{"location":"overview/#intelmq-mailgen_1","title":"intelmq-mailgen","text":"
      Sends emails with grouped event data to the contacts determined by the certbund-contact. Mails can be encrypted with PGP.
       
      \u2192 Repository: intelmq-mailgen
      "},{"location":"overview/#constituency-portal-tuency","title":"\"Constituency Portal\" tuency","text":"
      A web application helping CERTs to enable members of their constituency to self-administrate how they get warnings related to their network objects (IP addresses, IP ranges, autonomous systems, domains). tuency is developed by Intevation for CERT.at.
       
      If features organizational hierarchies, contact roles, self-administration and network objects per organization (Autonomous systems, network ranges, (sub)domains, RIPE organization handles). A network object claiming and approval process prevents abuse. An hierarchical rule-system on the network objects allow fine-grained settings. The tagging system for contacts and organization complement the contact-management features of the portal. Authentication is based on keycloak, which enables the re-use of the user accounts in the portal. The integrated API enables IntelMQ to query the portal for the right abuse contact and notification settings with the intelmq.bots.experts.tuency.expert expert bot.
       
       
      \u2192 Repository: tuency
      "},{"location":"overview/#constituency-portal-do-portal-deprecated","title":"\"Constituency Portal\" do-portal (deprecated)","text":"
      Warning
       
      The do-portal is deprecated and succeeded by tuency.
       
      A contact portal with organizational hierarchies, role functionality and network objects based on RIPE, allows self-administration by the contacts. Can be queried from IntelMQ and integrates the stats-portal.
       
      Originally developed by CERT-EU, then adapted by CERT.at.
       
      \u2192 Repository: do-portal
      "},{"location":"overview/#stats-portal","title":"Stats Portal","text":"
      A Grafana-based statistics portal for the eventdb{.interpreted-text role=\"doc\"}. Can be integrated into do-portal. It uses aggregated data to serve statistical data quickly.
       
       
      \u2192 Repository: stats-portal
      "},{"location":"overview/#malware-name-mapping","title":"Malware Name Mapping","text":"
      A mapping for malware names of different feeds with different names to a common family name.
       
      \u2192 Repository: malware_name_mapping
      "},{"location":"overview/#intelmq-docker","title":"IntelMQ-Docker","text":"
      A repository with tools for IntelMQ docker instance.
       
      Developed and maintained by CERT.at.
       
      \u2192 Repository: intelmq-docker
      "},{"location":"overview/#useful-scripts","title":"Useful scripts","text":"
      The list of useful scripts contributed to the IntelMQ universe can be found in the main repository.
       
      \u2192 Repository: intelmq/contrib
      "},{"location":"security/","title":"Security","text":""},{"location":"security/#found-a-security-issue","title":"Found a security issue?","text":"
      In case you find security-relevant bugs in IntelMQ, please contact team@cert.at. More information including the PGP key can be found on CERT.at's website.
      "},{"location":"admin/beta-features/","title":"Beta Features","text":""},{"location":"admin/beta-features/#beta-features","title":"Beta Features","text":""},{"location":"admin/beta-features/#using-supervisor-as-a-process-manager","title":"Using Supervisor as a Process Manager","text":"
      Warning
       
      Do not use it in production environments yet! It has not been tested thoroughly yet.
       
      Supervisor is process manager written in Python. The main advantage is that it take care about processes, so if bot process exit with failure (exit code different than 0), supervisor try to run it again. Another advantage is that it not require writing PID files.
       
      This was tested on Ubuntu 18.04.
       
      Install supervisor. supervisor_twiddler is extension for supervisor, that makes possible to create process dynamically. (Ubuntu supervisor package is currently based on Python 2, so supervisor_twiddler must be installed with Python 2 pip.)
       
      apt install supervisor python-pip\npip install supervisor_twiddler\n
       
      Create default config /etc/supervisor/conf.d/intelmq.conf and restart supervisor service:
       
      [rpcinterface:twiddler]\nsupervisor.rpcinterface_factory=supervisor_twiddler.rpcinterface:make_twiddler_rpcinterface\n\n[group:intelmq]\n
       
      Change IntelMQ process manager in the global configuration:
       
      process_manager: supervisor\n
       
      After this it is possible to manage bots like before with intelmqctl command.
      "},{"location":"admin/beta-features/#using-amqp-message-broker","title":"Using AMQP Message Broker","text":"
      Starting with IntelMQ 1.2 the AMQP protocol is supported as message queue. To use it, install a broker, for example RabbitMQ. The configuration and the differences are outlined here. Keep in mind that it is slower, but has better monitoring capabilities and is more stable. The AMQP support is considered beta, so small problems might occur. So far, only RabbitMQ as broker has been tested.
       
      You can change the broker for single bots (set the parameters in the runtime configuration per bot) or for the whole botnet (using the global configuration).
       
      You need to set the parameter source_pipeline_broker/destination_pipeline_broker to amqp. There are more parameters available:
       
      Bug
       
      This section of the documentation is currently incomplete and will be updated later.
       
      destination_pipeline_broker
       
      (required, string) \"amqp\"
       
      destination_pipeline_host
       
      ()  (default: '127.0.0.1')
       
      destination_pipeline_port
       
      () (default: 5672)
       
      destination_pipeline_username
       
      ()
       
      destination_pipeline_password
       
      ()
       
      destination_pipeline_socket_timeout
       
      () (default: no timeout)
       
      destination_pipeline_amqp_exchange
       
      () Only change/set this if you know what you do. If set, the destination queues are not declared as queues, but used as routing key. (default: '').
       
      destination_pipeline_amqp_virtual_host
       
      () (default: '/')
       
      source_pipeline_host
       
      () (default: '127.0.0.1')
       
      source_pipeline_port
       
      () (default: 5672)
       
      source_pipeline_username
       
      ()
       
      source_pipeline_password
       
      ()
       
      source_pipeline_socket_timeout
       
      () (default: no timeout)
       
      source_pipeline_amqp_exchange
       
      () Only change/set this if you know what you do. If set, the destination queues are not declared as queues, but used as routing key. (default: ['']).
       
      source_pipeline_amqp_virtual_host
       
      () (default: '/')
       
      intelmqctl_rabbitmq_monitoring_url
       
      () string, see below (default: \"http://{host}:15672\")
       
      For getting the queue sizes, intelmqctl needs to connect to the monitoring interface of RabbitMQ. If the monitoring interface is not available under http://{host}:15672 you can manually set using the parameter intelmqctl_rabbitmq_monitoring_url. In a RabbitMQ's default configuration you might not provide a user account, as by default the administrator (guest:guest) allows full access from localhost. If you create a separate user account, make sure to add the tag \"monitoring\" to it, otherwise IntelMQ can't fetch the queue sizes.
       
       
      Setting the statistics (and cache) parameters is necessary when the local redis is running under a non-default host/port. If this is the case, you can set them explicitly:
       
      statistics_database
       
      () 3
       
      statistics_host
       
      () \"127.0.0.1\"
       
      statistics_password
       
      () null
       
      statistics_port
       
      () 6379
      "},{"location":"admin/beta-features/#multithreading","title":"Multithreading","text":"
      First of all: Do not use it in production environments yet! There are a few bugs, see below
       
      Since IntelMQ 2.0 it is possible to provide the following runtime parameter:
       
      instances_threads
       
      Set it to a non-zero integer, then this number of worker threads will be spawn. This is useful if bots often wait for system resources or if network-based lookups are a bottleneck.
       
      However, there are currently a few cavecats:
       
         
      • This is not possible for all bots, there are some exceptions (collectors and some outputs), see the FAQ for some reasons.
        •  
      • Only use it with the AMQP pipeline, as with Redis, messages may get duplicated because there's only one internal   queue
        •  
      • In the logs, you can see the main thread initializing first, then all of the threads which log with the   name [bot-id].[thread-id].
        •  
      "},{"location":"admin/common-problems/","title":"Common Problems","text":""},{"location":"admin/common-problems/#common-problems","title":"Common Problems","text":""},{"location":"admin/common-problems/#intelmq","title":"IntelMQ","text":""},{"location":"admin/common-problems/#permission-denied-when-using-redis-unix-socket","title":"Permission denied when using Redis Unix socket","text":"
      If you get an error like this:
       
      intelmq.lib.exceptions.PipelineError: pipeline failed - ConnectionError('Error 13 connecting to unix socket: /var/run/redis/redis.sock. Permission denied.',)\n
       
      Make sure the intelmq user as sufficient permissions for the socket.
       
      In /etc/redis/redis.conf (or wherever your configuration is), check the permissions and set it for example to group-writeable:
       
      unixsocketperm 770\n
       
      And add the user intelmq to the redis-group:
       
      usermod -aG redis intelmq\n
      "},{"location":"admin/common-problems/#my-bots-died-on-startup-with-no-errors-logged","title":"My bot(s) died on startup with no errors logged","text":"
      Rather than starting your bot(s) with intelmqctl start, try intelmqctl run [bot]. This will provide valuable debug output you might not otherwise see, pointing to issues like system configuration errors.
      "},{"location":"admin/common-problems/#orphaned-queues","title":"Orphaned Queues","text":"
      This section has been moved to the Management Guide.
      "},{"location":"admin/common-problems/#multithreading-is-not-available-for-this-bot","title":"Multithreading is not available for this bot","text":"
      Multithreading is not available for some bots and AMQP broker is necessary. Possible reasons why a certain bot or a setup does not support Multithreading include:
       
         
      • Multithreading is only available when using the AMQP broker.
        •  
      • For most collectors, Multithreading is disabled. Otherwise this      would lead to duplicated data, as the data retrieval is not      atomic.
        •  
      • Some bots use libraries which are not thread safe. Look a the      bot's documentation for more information.
        •  
      • Some bots' operations are not thread safe. Look a the bot's      documentation for more information.
        •  
       
      If you think this mapping is wrong, please report a bug.
      "},{"location":"admin/common-problems/#intelmq-api","title":"IntelMQ API","text":""},{"location":"admin/common-problems/#intelmqctlerror","title":"IntelMQCtlError","text":"
      If the command is not configured correctly, you will see exceptions on startup like this:
       
      intelmq_manager.runctl.IntelMQCtlError: <ERROR_MESSAGE>\n
       
      This means the intelmqctl command could not be executed as a subprocess. The <ERROR_MESSAGE> should indicate why.
      "},{"location":"admin/common-problems/#access-denied-authentication-required-please-provide-valid-token-verification-credentials","title":"Access Denied / Authentication Required \"Please provide valid Token verification credentials\"","text":"
      If you see the IntelMQ Manager interface and menu, but the API calls to the back-end querying configuration and status of IntelMQ fail with \"Access Denied\" or \"Authentication Required: Please provide valid Token verification credentials\" errors, you are maybe not logged in while the API requires authentication.
       
      By default, the API requires authentication. Create user accounts and login with them or - if you have other protection means in place - deactivate the authentication requirement by removing or renaming the session_store parameter in the configuration.
      "},{"location":"admin/common-problems/#internal-server-error","title":"Internal Server Error","text":"
      There can be various reasons for internal server errors. You need to look at the error log of your web server, for example /var/log/apache2/error.log or /var/log/httpd/error_log for Apache 2. It could be that the sudo-setup is not functional, the configuration file or session database file can not be read or written or other errors in regards to the execution of the API program.
      "},{"location":"admin/common-problems/#can-i-just-install-it-from-the-debrpm-packages-while-installing-intelmq-from-a-different-source","title":"Can I just install it from the deb/rpm packages while installing IntelMQ from a different source?","text":"
      Yes, you can install the API and the Manager from the deb/rpm repositories, and install your IntelMQ from a somewhere else, e.g. a local repository. However, knowledge about Python and system administration experience is recommended if you do so.
       
      The packages install IntelMQ to /usr/lib/python3*/site-packages/intelmq/. Installing with pip results in /usr/local/lib/python3*/site-packages/intelmq/ (and some other accompaning resources) which overrides the installation in /usr/lib/. You probably need to adapt the configuration parameter intelmq_ctl_cmd to the /usr/local/bin/intelmqctl executable and some other tweaks.
      "},{"location":"admin/common-problems/#sqlite3operationalerror-attempt-to-write-a-readonly-database","title":"sqlite3.OperationalError: attempt to write a readonly database","text":"
      SQLite does not only need write access to the database itself, but also the folder the database file is located in. Please check that the webserver has write permissions to the folder the session file is located in.
      "},{"location":"admin/faq/","title":"FAQ","text":""},{"location":"admin/faq/#frequently-asked-questions","title":"Frequently asked questions","text":""},{"location":"admin/faq/#how-can-i-improve-the-speed","title":"How can I improve the speed?","text":"
      In most cases the bottlenecks are look-up experts. In these cases you can easily use the integrated load balancing features.
      "},{"location":"admin/faq/#multithreading","title":"Multithreading","text":"
      When using the AMQP broker, you can make use of Multi-threading. See the multithreading section.
      "},{"location":"admin/faq/#classic-load-balancing-multiprocessing","title":"\"Classic\" load-balancing (Multiprocessing)","text":"
      Before Multithreading was available in IntelMQ, and in case you use Redis as broker, the only way to do load balancing involves more work. Create multiple instances of the same bot and connect them all to the same source and destination bots. Then set the parameter load_balance to true for the bot which sends the messages to the duplicated bot. Then, the bot sends messages to only one of the destination queues and not to all of them.
       
      True Multiprocessing is not available in IntelMQ. See also this discussion on a possible enhanced load balancing <186>.
      "},{"location":"admin/faq/#other-options","title":"Other options","text":"
      For any bottleneck based on (online) lookups, optimize the lookup itself and if possible use local databases.
       
      It is also possible to use multiple servers to spread the workload. To get the messages from one system to the other you can either directly connect to the other's pipeline or use a fast exchange mechanism such as the TCP Collector/Output (make sure to secure the network by other means).
      "},{"location":"admin/faq/#removing-raw-data-for-higher-performance-and-less-space-usage","title":"Removing raw data for higher performance and less space usage","text":"
      If you do not need the raw data, you can safely remove it. For events (after parsers), it keeps the original data, eg. a line of a CSV file. In reports it keeps the actual data to be parsed, so don't delete the raw field in Reports - between collectors and parsers.
       
      The raw data consumes about 50% - 30% of the messages' size. The size of course depends on how many additional data you add to it and how much data the report includes. Dropping it, will improve the speed as less data needs to be transferred and processed at each step.
       
      In a bot
       
      You can do this for example by using the Field Reducer Expert. The configuration could be:
       
         
      • type: blacklist
        •  
      • keys: raw
        •  
       
      Other solutions are the Modify bot and the Sieve bot. The last one is a good choice if you already use it and you only need to add the command:
       
      remove raw\n
       
      In the database
       
      In case you store data in the database and you want to keep its size small, you can (periodically) delete the raw data there.
       
      To remove the raw data for a events table of a PostgreSQL database, you can use something like:
       
      UPDATE events SET raw = NULL WHERE \"time.source\" < '2018-07-01';\n
       
      If the database is big, make sure only update small parts of the database by using an appropriate WHERE clause. If you do not see any negative performance impact, you can increase the size of the chunks, otherwise the events in the output bot may queue up. The id column can also be used instead of the source's time.
       
      Another way of reducing the raw-data from the database is described in the EventDB documentation: eventdb_raws_table.
      "},{"location":"admin/faq/#how-to-uninstall","title":"How to Uninstall","text":"
      If you installed intelmq with native packages: Use the package management tool to remove the package intelmq. These tools do not remove configuration by default.
       
      If you installed manually via pip (note that this also deletes all configuration and possibly data):
       
      pip3 uninstall intelmq\nrm -r /opt/intelmq\n
      "},{"location":"admin/hardware-requirements/","title":"Hardware Requirements","text":""},{"location":"admin/hardware-requirements/#hardware-requirements","title":"Hardware Requirements","text":"
      Do you ask yourself how much RAM do you need to give your new IntelMQ virtual machine?
       
      The honest answer is simple and pointless: It depends ;)
      "},{"location":"admin/hardware-requirements/#intelmq-and-the-messaging-queue-broker","title":"IntelMQ and the messaging queue (broker)","text":"
      IntelMQ uses a messaging queue to move the messages between the bots. All bot instances can only process one message at a time, therefore all other messages need to wait in the queue. As not all bots are equally fast, the messages will naturally \"queue up\" before the slower ones. Further, parsers produce many events with just one message (the report) as input.
       
      The following estimations assume Redis as messaging broker which is the default for IntelMQ. When RabbitMQ is used, the required resources will differ, and RabbitMQ can handle system overload and therefore a shortage of memory.
       
      As Redis stores all data in memory, the data which is processed at any point in time must fit there, including overheads. Please note that IntelMQ does neither store nor cache any input data. These estimates therefore only relate to the processing step, not the storage.
       
      For a minimal system, these requirements suffice:
       
         
      • 4 GB of RAM
        •  
      • 2 CPUs
        •  
      • 10 GB disk size
        •  
       
      Depending on your data input, you will need the twentiethfold of the input data size as memory for processing.
       
      When using Redis persistence, you will additionally need twice as much memory for Redis.
      "},{"location":"admin/hardware-requirements/#disk-space","title":"Disk space","text":"
      Disk space is only relevant if you save your data to a file, which is not recommended for production setups, and only useful for testing and evaluation.
       
      Do not forget to rotate your logs or use syslog, especially if you use the logging level \"DEBUG\". logrotate is in use by default for all installation with deb/rpm packages. When other means of installation are used (pip, manual), configure log rotation manually. See logging configuration.
      "},{"location":"admin/hardware-requirements/#background-on-memory","title":"Background on memory","text":"
      For experimentation, we used multiple Shadowserver Poodle reports for demonstration purpose, totaling in 120 MB of data. All numbers are estimates and are rounded. In memory, the report data requires 160 MB. After parsing, the memory usage increases to 850 MB in total, as every data line is stored as JSON, with additional information plus the original data encoded in Base 64. The further processing steps depend on the configuration, but you can estimate that caches (for lookups and deduplication) and other added information cause an additional size increase of about 2x. Once a dataset finished processing in IntelMQ, it is no longer stored in memory. Therefore, the memory is only needed to catch high load.
       
      The above numbers result in a factor of 14 for input data size vs. memory required by Redis. Assuming some overhead and memory for the bots' processes, a factor of 20 seems sensible.
       
      To reduce the amount of required memory and disk size, you can optionally remove the raw data field, see this section in the FAQ.
      "},{"location":"admin/hardware-requirements/#additional-components","title":"Additional components","text":"
      If some of the optional components are in use, they can add additional hardware requirements.
       
      Those components do not add relevant requirements:
       
         
      • IntelMQ API: It is just an API for intelmqctl.
        •  
      • IntelMQ Manager: Only contains static files served by the webserver.
        •  
      • IntelMQ Webinput CSV: Just a webinterface to insert data. Requires     the amount of processed data to fit in memory, see above.
        •  
      • Stats Portal: The aggregation step and Graphana require some     resources, but no exact numbers are known.
        •  
      • Malware Name Mapping
        •  
      • Docker: The docker layer adds only minimal hardware requirements.
        •  
      "},{"location":"admin/hardware-requirements/#database","title":"Database","text":"
      When storing data in databases (such as MongoDB, PostgreSQL, ElasticSearch), it is recommended to do this on separate machines for operational reasons. Using a different machine results in a separation of stream processing to data storage and allows for a specialized system optimization for both use-cases.
      "},{"location":"admin/hardware-requirements/#intelmq-cb-mailgen","title":"IntelMQ cb mailgen","text":"
      While the Fody backend and frontend do not have significant requirements, the RIPE import tool of the certbund-contact requires about 8 GB of memory as of March 2021.
      "},{"location":"admin/intro/","title":"Intro","text":""},{"location":"admin/intro/#intro","title":"Intro","text":"
      This guide provides instructions on how to install, configure and manage IntelMQ and it's components.
       
      IntelMQ uses a message broker such as Redis. This is required for IntelMQ to run.
       
      IntelMQ doesn't handle long term storage of processed Events beyond writing to a file. However it provides connectors (output bots) for writing events to various database systems and log collectors. It is recommended to configure such system to preserve processed events.
      "},{"location":"admin/intro/#base-requirements","title":"Base Requirements","text":"
      The following instructions assume the following requirements. Python versions >= 3.7 are supported.
       
      Supported and recommended operating systems are:
       
         
      • Debian
        •  
      • openSUSE Tumbleweed/Leap
        •  
      • Ubuntu
        •  
      • For the Docker-installation: Docker Engine: 18.x and higher
        •  
       
      Other distributions which are (most probably) supported include AlmaLinux, CentOS, Fedora, FreeBSD 12, RHEL and RockyLinux.
       
      A short guide on hardware requirements can be found on the page Hardware Requirements.
      "},{"location":"admin/upgrade/","title":"Upgrade","text":""},{"location":"admin/upgrade/#upgrade-instructions","title":"Upgrade instructions","text":"
      In order to upgrade your IntelMQ installation it is recommended to follow these five steps:
      "},{"location":"admin/upgrade/#1-read-newsmd","title":"1. Read NEWS.md","text":"
      Read the NEWS.md file to look for things you need to have a look at.
      "},{"location":"admin/upgrade/#2-stop-intelmq-and-create-a-backup","title":"2. Stop IntelMQ and create a backup","text":"
         
      • Make sure that your IntelMQ system is completely stopped: intelmqctl stop
        •  
      • Create a backup of IntelMQ Home directory, which includes all configurations. They are not overwritten, but backups are always nice to have!
        •  
       
      sudo cp -R /opt/intelmq /opt/intelmq-backup\n
      "},{"location":"admin/upgrade/#3-upgrade-intelmq","title":"3. Upgrade IntelMQ","text":"
      Before upgrading, check that your setup is clean and there are no events in the queues:
       
      intelmqctl check\nintelmqctl list queues -q\n
       
      The upgrade depends on how you installed IntelMQ.
      "},{"location":"admin/upgrade/#linux-packages","title":"Linux Packages","text":"
      Use your system's package manager.
      "},{"location":"admin/upgrade/#pypi","title":"PyPi","text":"
      pip install -U --no-deps intelmq\nsudo intelmqsetup\n
       
      Using --no-deps will not upgrade dependencies, which would probably overwrite the system's libraries. Remove this option to also upgrade dependencies.
      "},{"location":"admin/upgrade/#docker","title":"Docker","text":"
      You can check out all current versions on our DockerHub.
       
      docker pull certat/intelmq-full:latest\n\ndocker pull certat/intelmq-nginx:latest\n
       
      Alternatively you can use docker-compose:
       
      docker-compose pull\n
       
      You can check the current versions from intelmq & intelmq-manager & intelmq-api via git commit ref.
       
      The Version format for each included item is key=value and they are saparated via ,. I. e. IntelMQ=ab12cd34f,IntelMQ-API=xy65z23.
       
      docker inspect --format '{{ index .Config.Labels \"org.opencontainers.image.version\" }}' intelmq-full:latest\n
       
      Now restart your container, if you're using docker-compose you simply run:
       
      docker-compose down\n
       
      If you don't use docker-compose, you can restart a single container using:
       
      docker ps | grep certat\n\ndocker restart CONTAINER_ID\n
      "},{"location":"admin/upgrade/#source-repository","title":"Source repository","text":"
      If you have an editable installation, refer to the instructions in the /dev/guide.
       
      Update the repository depending on your setup (e.g. [git pull origin master]).
       
      And run the installation again:
       
      pip install .\nsudo intelmqsetup\n
       
      For editable installations (development only), run [pip install -e .] instead.
      "},{"location":"admin/upgrade/#4-upgrade-configuration-and-check-the-installation","title":"4. Upgrade configuration and check the installation","text":"
      Go through NEWS.md and apply necessary adaptions to your setup. If you have adapted IntelMQ's code, also read the CHANGELOG.md.
       
      Check your installation and configuration to detect any problems:
       
      intelmqctl upgrade-config\nintelmqctl check\n
       
      intelmqctl upgrade-config supports upgrades from one IntelMQ version to the succeeding. If you skip one or more IntelMQ versions, some automatic upgrades may not work and manual intervention may be necessary.
      "},{"location":"admin/upgrade/#5-start-intelmq","title":"5. Start IntelMQ","text":"
      intelmqctl start\n
      "},{"location":"admin/configuration/intelmq-api/","title":"IntelMQ API","text":""},{"location":"admin/configuration/intelmq-api/#configuring-intelmq-api","title":"Configuring IntelMQ API","text":"
      Depending on your setup you might have to install sudo to make it possible for the intelmq-api to run the intelmq command as the user-account usually used to run intelmq (which is also often called intelmq).
       
      intelmq-api is configured using a configuration file in json format. intelmq-api tries to load the configuration file from /etc/intelmq/api-config.json and ${PREFIX}/etc/intelmq/api-config.json, but you can override the path setting the environment variable INTELMQ_API_CONFIG. (When using Apache, you can do this by modifying the Apache configuration file shipped with intelmq-api, the file contains an example)
       
      When running the API using hug, you can set the environment variable like this:
       
      INTELMQ_API_CONFIG=/etc/intelmq/api-config.json hug -m intelmq_api.serve\n
       
      The default configuration which is shipped with the packages is also listed here for reference:
       
      {\n    \"intelmq_ctl_cmd\": [\"sudo\", \"-u\", \"intelmq\", \"intelmqctl\"],\n    \"allowed_path\": \"/opt/intelmq/var/lib/bots/\",\n    \"session_store\": \"/etc/intelmq/api-session.sqlite\",\n    \"session_duration\": 86400,\n    \"allow_origins\": [\"*\"]\n}\n
       
      On Debian based systems, the default path for the session_store is /var/lib/dbconfig-common/sqlite3/intelmq-api/intelmqapi, because the Debian package uses the Debian packaging tools to manage the database file.
       
      The following configuration options are available:
       
         
      • intelmq_ctl_cmd: Your intelmqctl command. If this is not set in     a configuration file the default is used, which is     [\"sudo\", \"-u\", \"intelmq\", \"/usr/local/bin/intelmqctl\"] The option     \"intelmq_ctl_cmd\" is a list of strings so that we can avoid     shell-injection vulnerabilities because no shell is involved when     running the command. This means that if the command you want to use     needs parameters, they have to be separate strings.
        •  
      • allowed_path: intelmq-api can grant read-only access to     specific files - this setting defines the path those files can     reside in.
        •  
      • session_store: this is an optional path to a sqlite database,     which is used for session storage and authentication. If it is not     set (which is the default), no authentication is used!
        •  
      • session_duration: the maximal duration of a session, its 86400     seconds by default
        •  
      • allow_origins: a list of origins the responses of the API can be     shared with. Allows every origin by default.
        •  
      "},{"location":"admin/configuration/intelmq-api/#permissions","title":"Permissions","text":"
      intelmq-api tries to write a couple of configuration files in the ${PREFIX}/etc/intelmq directory - this is only possible if you set the permissions accordingly, given that intelmq-api runs under a different user. The user the API run as also needs write access to the folder the session_store is located in, otherwise there will be an error accessing the session data. If you\\'re using the default Apache 2 setup, you might want to set the group of the files to www-data and give it write permissions (chmod -R g+w <directoryname>). In addition to that, the intelmq-manager tries to store the bot positions via the API into the file ${PREFIX}/etc/intelmq/manager/positions.conf. You should therefore create the folder ${PREFIX}/etc/intelmq/manager and the file positions.conf in it.
      "},{"location":"admin/configuration/intelmq-api/#adding-a-user","title":"Adding a user","text":"
      If you enable the session_store you will have to create user accounts to be able to access the API functionality. You can do this using intelmq-api-adduser:
       
      intelmq-api-adduser --user <username> --password <password>\n
      "},{"location":"admin/configuration/intelmq-api/#a-note-on-selinux","title":"A note on SELinux","text":"
      On systems with SELinux enabled, the API will fail to call intelmqctl. Therefore, SELinux needs to be disabled:
       
      setenforce 0\n
       
      We welcome contributions to provide SELinux policies.
      "},{"location":"admin/configuration/intelmq-manager/","title":"IntelMQ Manager","text":""},{"location":"admin/configuration/intelmq-manager/#configuring-intelmq-manager","title":"Configuring IntelMQ Manager","text":"
      In the file /usr/share/intelmq-manager/html/js/vars.js set ROOT to the URL of your intelmq-api installation - by default that's on the same host as intelmq-manager.
      "},{"location":"admin/configuration/intelmq-manager/#configuration-paths","title":"Configuration Paths","text":"
      The IntelMQ Manager queries the configuration file paths and directory names from intelmqctl and therefore any global environment variables (if set) are effective in the Manager too. The interface for this query is intelmqctl debug --get-paths, the result is also shown in the /about.html page of your IntelMQ Manager installation.
      "},{"location":"admin/configuration/intelmq-manager/#csp-headers","title":"CSP Headers","text":"
      It is recommended to set these two headers for all requests:
       
      Content-Security-Policy: script-src 'self'\nX-Content-Security-Policy: script-src 'self'\n
      "},{"location":"admin/configuration/intelmq-manager/#security-considerations","title":"Security considerations","text":"
      Never ever run intelmq-manager on a public webserver without SSL and proper authentication!
       
      The way the current version is written, anyone can send a POST request and change intelmq's configuration files via sending HTTP POST requests. Intelmq-manager will reject non JSON data but nevertheless, we don't want anyone to be able to reconfigure an intelmq installation.
       
      Therefore you will need authentication and SSL. Authentication can be handled by the intelmq-api. Please refer to its documentation on how to enable authentication and setup accounts.
       
      Never ever allow unencrypted, unauthenticated access to IntelMQ Manager!
      "},{"location":"admin/configuration/intelmq-manager/#docker-security-headers","title":"Docker: Security headers","text":"
      If you run our docker image in production, we recommend you to set security headers. You can do this by creating a new file called example_config/nginx/security.conf in the cloned intelmq-docker repository.
       
      Write the following inside the configuration file, and change the http(s)://<your-domain> to your domain name.
       
      server_tokens off; # turn off server_token, instead of nginx/13.2 now it will only show nginx\nadd_header X-Frame-Options SAMEORIGIN; # https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options\nadd_header X-Content-Type-Options nosniff; # https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Content-Type-Options\nadd_header X-XSS-Protection \"1; mode=block\"; # https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-XSS-Protection\nadd_header Content-Security-Policy \"script-src 'self' 'unsafe-inline' http(s)://<your-domain>; frame-src 'self' http(s)://<your-domain>; object-src 'self' http(s)://<your-domain>\"; # https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP\n
       
      After you created the file, edit the docker-compose.yml and mount it to the nginx with
       
      volumes:\n  - ./example_config/nginx/security.conf:/etc/nginx/conf.d/security.conf\n
       
      IMPORTANT Mount the exact name & not the directory, because otherwise you would overwrite the whole directory and the other files would be gone inside the container.
      "},{"location":"admin/configuration/intelmq/","title":"IntelMQ","text":""},{"location":"admin/configuration/intelmq/#configuring-intelmq","title":"Configuring IntelMQ","text":""},{"location":"admin/configuration/intelmq/#directories","title":"Directories","text":""},{"location":"admin/configuration/intelmq/#lsb","title":"LSB","text":"
      If you installed the packages, standard Linux paths (LSB paths) are used:
       
         
      • /etc/intelmq/ (configurations)
        •  
      • /var/log/intelmq/ (logs)
        •  
      • /var/lib/intelmq/ (local states)
        •  
      • /var/run/intelmq/ (PID files)
        •  
       
      Otherwise, the configuration directory is /opt/intelmq/etc/. Using the environment variable INTELMQ_ROOT_DIR allows setting any arbitrary root directory.
       
      You can switch this by setting the environment variables INTELMQ_PATHS_NO_OPT and INTELMQ_PATHS_OPT, respectively.
       
         
      • When installing the Python packages, you can set INTELMQ_PATHS_NO_OPT to something non-empty to use LSB-paths.
        •  
      • When installing the deb/rpm packages, you can set INTELMQ_PATHS_OPT to something non-empty to use /opt/intelmq/ paths, or a path set with INTELMQ_ROOT_DIR.
        •  
       
      The environment variable ROOT_DIR is meant to set an alternative root directory instead of /. This is primarily meant for package build environments an analogous to setuptool's --root parameter. Thus it is only used in LSB-mode.
      "},{"location":"admin/configuration/intelmq/#environment-variables","title":"Environment Variables","text":"Name Type Description INTELMQ_PATHS_OPT INTELMQ_PATHS_NO_OPT INTELMQ_ROOT_DIR ROOT_DIR"},{"location":"admin/configuration/intelmq/#configuration-files","title":"Configuration Files","text":""},{"location":"admin/configuration/intelmq/#runtimeyaml","title":"runtime.yaml","text":"
      This is the main configuration file. It uses YAML format since IntelMQ 3.0. It consists of two parts:
       
         
      • Global Configuration
        •  
      • Individual Bot Configuration
        •  
       
      Warning
       
      Comments in YAML are currently not preserved by IntelMQ (known bug #2003).
       
      Example runtime.yaml configuration file is installed by the tool intelmqsetup. If this is not the case, make sure the program was run. It is shipped preconfigured with 4 collectors and parsers, 6 common experts and one output bot. The default collector and the parser handle data from malware domain list, the file output bot writes all data to one of these files (based on your installation):
       
         
      •  
        /opt/intelmq/var/lib/bots/file-output/events.txt
         
        •  
      •  
        /var/lib/intelmq/bots/file-output/events.txt
         
        •  
       
      The runtime.yaml configuration is divided into two sections:
       
         
      • Global configuration which is applied to each bot.
        •  
      • Individual bot configuration which overloads the global configuration and contains bot specific options.
        •  
       
      Example configuration snippet:
       
      global: # global configuration section\n  # ...\n  http_timeout_max_tries: 3\n  http_timeout_sec: 30\n  http_user_agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36\n  http_verify_cert: true\n\nblocklistde-apache-collector: # individual bot configuration section\n  group: Collector\n  name: Blocklist.de Apache List\n  module: intelmq.bots.collectors.http.collector_http\n  description: Blocklist.de Apache Collector fetches all IP addresses which have been reported within the last 48 hours as having run attacks on the service Apache, Apache-DDOS, RFI-Attacks.\n  parameters:\n    http_url: https://lists.blocklist.de/lists/apache.txt\n    name: Blocklist.de Apache\n    rate_limit: 3600\n    http_verify_cert: false # overriding the global configuration for this particular bot\n
      "},{"location":"admin/configuration/intelmq/#global-configuration","title":"Global Configuration","text":"
      The global configuration parameters apply to all bots, however they can be overridden in the individual bot configuration.
      "},{"location":"admin/configuration/intelmq/#logging","title":"Logging","text":"
      The logging can be configured with the following parameters:
       
      logging_handler
       
      (required, string) Allowed values are file or syslog.
       
      logging_level
       
      (required, string) Allowed values are CRITICAL, ERROR, WARNING, INFO or DEBUG. Defines the system-wide log level that will be use by all bots and the intelmqctl tool. We recommend logging_level WARNING for production environments and INFO if you want more details. In any case, watch your free disk space!
       
      logging_path
       
      (required, string) When the logging_handler is file this parameter is used to set the logging directory for all the bots as well as the intelmqctl tool. Defaults to /opt/intelmq/var/log/ or /var/log/intelmq/ respectively.
       
      logging_syslog
       
      (required, string) When the logging_handler is syslog. Either a list with hostname and UDP port of syslog service, e.g. [\"localhost\", 514] or a device name/path. Defaults to /var/log.
      "},{"location":"admin/configuration/intelmq/#log-rotation","title":"Log Rotation","text":"
      To rotate the logs, you can use the standard Linux-tool logrotate. An example logrotate configuration is given in contrib/logrotate/ and delivered with all deb/rpm-packages. When not using logrotate, IntelMQ can rotate the logs itself, which is not enabled by default! You need to set both values.
       
      logging_max_size
       
      (optional, integer) Maximum number of bytes to be stored in one logfile before the file is rotated. Defaults to 0 (log rotation disabled).
       
      logging_max_copies
       
      (optional, integer) Maximum number of logfiles to keep. Compression is not supported. Default is unset.
       
      Some information can as well be found in Python's documentation on the used RotatingFileHandler.
      "},{"location":"admin/configuration/intelmq/#error-handling","title":"Error Handling","text":"
      error_log_message
       
      (required, boolean) Whether to write the message (Event/Report) to the log file in case of an error.
       
      error_log_exception
       
      (required, boolean) Whether to write an error exception to the log file in case of an error.
       
      error_procedure
       
      (required, string) Allowed values are stop or pass. In case of an error, this option defines the procedure that the bot will adopt. Use the following values:
       
         
      •  
        stop - stop bot after retrying X times (as defined in error_max_retries) with a delay between retries (as defined in error_retry_delay). If the bot reaches the error_max_retries value, it will remove the message from the pipeline and stop. If the option error_dump_message is also enable, the bot will dump the removed message to its dump file (to be found in var/log).
         
        •  
      •  
        pass - will skip this message and will process the next message after retrying X times, removing the current message from pipeline. If the option error_dump_message is also enable, then the bot will dump the removed message to its dump file. After max retries are reached, the rate limit is applied (e.g. a collector bot fetch an unavailable resource does not try forever).
         
        •  
       
      error_max_retries
       
      (required, integer) In case of an error, the bot will try to re-start processing the current message X times as   defined by this option.
       
      error_retry_delay
       
      (required, integer) Defines the number of seconds to wait between subsequent re-tries in case of an error.
       
      error_dump_message
       
      (required, boolean) Specifies if the bot will write queued up messages to its dump file (use intelmqdump to   re-insert the message).
       
      If the path _on_error exists for a bot, the message is also sent to this queue, instead of (only) dumping the file if configured to do so.
      "},{"location":"admin/configuration/intelmq/#miscellaneous","title":"Miscellaneous","text":"
      load_balance
       
      (required, boolean) this option allows you to choose the behavior of the queue. Use the following values:
       
         
      • true - splits the messages into several queues without duplication
        •  
      • false - duplicates the messages into each queue - When using AMQP as message broker, take a look at the multithreading{.interpreted-text role=\"ref\"} section and the instances_threads parameter.
        •  
       
      rate_limit
       
      (required, integer) time interval (in seconds) between messages processing. int value.
       
      ssl_ca_certificate
       
      (optional, string) trusted CA certificate for IMAP connections (supported by some bots).
       
      source_pipeline_broker
       
      (optional, string) Allowed values are redis and amqp. Selects the message broker IntelMQ should use. As this parameter can be overridden by each bot, this allows usage of different broker systems and hosts, as well as switching between them on the same IntelMQ instance. Defaults to redis.
       
         
      • redis - Please note that persistence has to be manually activated.
        •  
      • amqp - Using the AMQP broker is currently beta but there are no known issues. A popular AMQP broker is RabbitMQ.
        •  
       
      destination_pipeline_broker
       
      (required, string) See source_pipeline_broker.
       
      source_pipeline_host
       
      (required, string) Hostname or path to Unix socket that the bot will use to connect and receive messages.
       
      source_pipeline_port
       
      (optional, integer) Broker port that the bot will use to connect and receive messages. Can be empty for Unix socket.
       
      source_pipeline_password
       
      (optional, string) Broker password that the bot will use to connect and receive messages. Can be null for unprotected broker.
       
      source_pipeline_db
       
      (required, integer) broker database that the bot will use to connect and receive messages (requirement from   redis broker).
       
      destination_pipeline_host
       
      (optional, string) broker IP, FQDN or Unix socket that the bot will use to connect and send messages.
       
      destination_pipeline_port
       
      (optional, integer) broker port that the bot will use to connect and send messages. Can be empty for   Unix socket.
       
      destination_pipeline_password
       
      (optional, string) broker password that the bot will use to connect and send messages. Can be null   for unprotected broker.
       
      destination_pipeline_db
       
      (required, integer) broker database that the bot will use to connect and send messages (requirement from   redis broker).
       
      http_proxy
       
      (optional, string) Proxy to use for HTTP.
       
      https_proxy
       
      (optional, string) Proxy to use for HTTPS.
       
      http_user_agent
       
      (optional, string) User-Agent to be used for HTTP requests.
       
      http_verify_cert
       
      (optional, boolean) Verify the TLS certificate of the server. Defaults to true.
      "},{"location":"admin/configuration/intelmq/#individual-bot-configuration","title":"Individual Bot Configuration","text":"
      Info
       
      For the individual bot configuration please see the Bots document in the User Guide.
      "},{"location":"admin/configuration/intelmq/#run-mode","title":"Run Mode","text":"
      This sections provides more detailed explanation of the two run modes of the bots.
      "},{"location":"admin/configuration/intelmq/#continuous","title":"Continuous","text":"
      Most of the cases, bots will need to be configured as continuous run mode (the default) in order to have them always running and processing events. Usually, the types of bots that will require the continuous mode will be Parsers, Experts and Outputs. To do this, set run_mode to continuous in the runtime.yaml for the bot. Check the following example:
       
      blocklistde-apache-parser:\n  name: Blocklist.de Parser\n  group: Parser\n  module: intelmq.bots.parsers.blocklistde.parser\n  description: Blocklist.DE Parser is the bot responsible to parse the report and sanitize the information.\n  enabled: false\n  run_mode: continuous\n  parameters: ...\n
       
      You can now start the bot using the following command:
       
      intelmqctl start blocklistde-apache-parser\n
       
      Bots configured as continuous will never exit except if there is an error and the error handling configuration requires the bot to exit. See the Error Handling section for more details.
      "},{"location":"admin/configuration/intelmq/#scheduled","title":"Scheduled","text":"
      In many cases, it is useful to schedule a bot at a specific time (i.e. via cron(1)), for example to collect information from a website every day at midnight. To do this, set run_mode to scheduled in the runtime.yaml for the bot. Check out the following example:
       
      blocklistde-apache-collector:\n  name: Generic URL Fetcher\n  group: Collector\n  module: intelmq.bots.collectors.http.collector_http\n  description: All IP addresses which have been reported within the last 48 hours as having run attacks on the service Apache, Apache-DDOS, RFI-Attacks.\n  enabled: false\n  run_mode: scheduled\n  parameters:\n    feed: Blocklist.de Apache\n    provider: Blocklist.de\n    http_url: https://lists.blocklist.de/lists/apache.txt\n    ssl_client_certificate: null\n
       
      You can schedule the bot with a crontab-entry like this:
       
      0 0 * * * intelmqctl start blocklistde-apache-collector\n
       
      Bots configured as scheduled will exit after the first successful run. Setting enabled to false will cause the bot to not start with intelmqctl start, but only with an explicit start, in this example intelmqctl start blocklistde-apache-collector.
      "},{"location":"admin/configuration/intelmq/#additional-runtime-parameters","title":"Additional Runtime Parameters","text":"
      Some of the parameters are deliberately skipped from the User Guide because they are configured via graphical user interface provided by the IntelMQ Manager. These parameters have to do with configuring the pipeline: defining how the data is exchanged between the bots. Using the IntelMQ Manager for this have many benefits as it guarantees that the configuration is correct upon saving.
       
      However as an administrator you should be also familiar with the manual (and somewhat tedious) configuration. For each bot there are two parameters that need to be set:
       
      source_queue
       
      (optional, string) The name of the source queue from which the bot is going to processing data. Each bot has maximum one source queue (collector bots don't have any source queue as they fetch data from elsewhere). Defaults to the bot id appended with the string -queue.
       
      Example: a bot with id example-bot will have a default source queue named example-bot-queue.
       
      destination_queues
       
      (optional, object) Bots can have multiple destination queues. Destination queues can also be grouped into named paths. There are two special path names _default and _on_error.  The path _default is used if the path is not is specified by the bot itself (which is the most common case). In case of an error during the processing, the message will be sent to the _on_error path if specified (optional).
       
      Only few of the bots (mostly expert bots with filtering capabilities) can take advantage of arbitrarily named paths. Some expert bots are capable of sending messages to paths, this feature is explained in their documentation, e.g. the Filter expert and the Sieve expert.
       
      Example:
       
      blocklistde-apache-collector:\n  # ...\n  parameters:\n    # ...\n    destination_queues:\n      _default:\n        - <first destination pipeline name>\n        - <second destination pipeline name>\n      _on_error:\n        - <optional first destination pipeline name in case of errors>\n        - <optional second destination pipeline name in case of errors>\n      other-path:\n        - <second destination pipeline name>\n        - <third destination pipeline name>\n
      "},{"location":"admin/configuration/intelmq/#harmonizationconf","title":"harmonization.conf","text":"
      This configuration is used to specify the fields for all message types. The harmonization library will load this configuration to check, during the message processing, if the values are compliant to the configured harmonization format. Usually, this configuration doesn't need any change. It is mostly maintained by the IntelMQ maintainers.
       
      Template:
       
      {\n  \"<message type>\": {\n    \"<field 1>\": {\n      \"description\": \"<field 1 description>\",\n      \"type\": \"<field value type>\"\n    },\n    \"<field 2>\": {\n      \"description\": \"<field 2 description>\",\n      \"type\": \"<field value type>\"\n    }\n  }\n}\n
       
      Example:
       
      {\n  \"event\": {\n    \"destination.asn\": {\n      \"description\": \"The autonomous system number from which originated the connection.\",\n      \"type\": \"Integer\"\n    },\n    \"destination.geolocation.cc\": {\n      \"description\": \"Country-Code according to ISO3166-1 alpha-2 for the destination IP.\",\n      \"regex\": \"^[a-zA-Z0-9]{2}$\",\n      \"type\": \"String\"\n    }\n  }\n}\n
      "},{"location":"admin/database/elasticsearch/","title":"Elasticsearch","text":""},{"location":"admin/database/elasticsearch/#using-elasticsearch-as-a-database-for-intelmq","title":"Using Elasticsearch as a database for IntelMQ","text":"
      If you wish to run IntelMQ with Elasticsearch or full ELK stack (Elasticsearch, Logstash, Kibana) it is entirely possible. This guide assumes the reader is familiar with basic configuration of ELK and does not aim to cover using ELK in general. It is based on the version 6.8.0 (ELK is a fast moving train therefore things might change). Assuming you have IntelMQ (and Redis) installation in place, lets dive in.
      "},{"location":"admin/database/elasticsearch/#configuration-without-logstash","title":"Configuration without Logstash","text":"
      This case involves two steps:
       
         
      1.  
        Configure IntelMQ to output data directly into Elasticsearch.
         
        2.  
      2.  
        Configure Elasticsearch for ingesting the inserted data.
         
        3.  
       
      Bug
       
      This section of the documentation is currently incomplete and will be updated later.
      "},{"location":"admin/database/elasticsearch/#configuration-with-logstash","title":"Configuration with Logstash","text":"
      This case involves three steps:
       
         
      1.  
        Configuring IntelMQ to output data to Redis.
         
        2.  
      2.  
        Configure Logstash to collect data from Redis and insert them into Elasticsearch.
         
        3.  
      3.  
        Configure Elasticsearch for ingesting the inserted data.
         
        4.  
       
      Each step is described in detail in the following sections.
      "},{"location":"admin/database/elasticsearch/#configuring-intelmq","title":"Configuring IntelMQ","text":"
      In order to pass IntelMQ events to Logstash we will utilize already installed Redis. Add a new Redis Output Bot to your pipeline. As the minimum fill in the following parameters: bot-id, redis_server_ip (can be hostname) , redis_server_port, redis_password (if required, else set to empty!), redis_queue (name for the queue). It is recommended to use a different redis_db parameter than used by the IntelMQ (specified as source_pipeline_db , destination_pipeline_db and statistics_database).
       
      Example values:
       
      bot-id: redis-output\nredis_server_ip: 10.10.10.10\nredis_server_port: 6379\nredis_db: 4\nredis_queue: logstash-queue\n
       
      Warning
       
      You will not be able to monitor this redis queue via IntelMQ Manager.
      "},{"location":"admin/database/elasticsearch/#configuring-logstash","title":"Configuring Logstash","text":"
      Logstash defines pipelines as well. In the pipeline configuration of Logstash you need to specify where it should look for IntelMQ events, what to do with them and where to pass them.
      "},{"location":"admin/database/elasticsearch/#input","title":"Input","text":"
      This part describes how to receive data from Redis queue. See the example configuration and comments below:
       
      input {\n  redis {\n    host => \"10.10.10.10\"\n    port => 6379\n    db => 4\n    data_type => \"list\"\n    key => \"logstash-queue\"\n  }\n}\n
       
         
      • host - same as redis_server_ip from the Redis Output Bot
        •  
      • port - the redis_server_port from the Redis Output Bot
        •  
      • db - the redis_db parameter from the Redis Output Bot
        •  
      • data_type - set to list
        •  
      • key - same as redis_queue from the Redis Output Bot
        •  
       
      Tip
       
      You can use environment variables for the Logstash configuration, for example host => \"${REDIS_HOST:10.10.10.10}\". The value will be taken from the environment variable $REDIS_HOST. If the environment variable is not set then the default value of 10.10.10.10 will be used instead.
      "},{"location":"admin/database/elasticsearch/#filter-optional","title":"Filter (optional)","text":"
      Before passing the data to the database you can apply certain changes. This is done with filters. See an example:
       
      filter {\n  mutate {\n    lowercase => [\"source.geolocation.city\", \"classification.identifier\"]\n    remove_field => [\"__type\", \"@version\"]\n  }\n  date {\n    match => [\"time.observation\", \"ISO8601\"]\n  }\n}\n
       
      Tip
       
      It is recommended to use the date filter: generally we have two timestamp fields - time.source (provided by the feed source this can be understood as when the event happened; however it is not always present) and time.observation (when IntelMQ collected this event). Logstash also adds another field @timestamp with time of processing by Logstash. While it can be useful for debugging, I recommend to set the @timestamp to the same value as time.observation.
       
      Warning
       
      It is not recommended to apply any modifications to the data (within the mutate key) outside of the IntelMQ. All necessary modifications should be done only by appropriate IntelMQ bots. This example only demonstrates the possibility.
      "},{"location":"admin/database/elasticsearch/#output","title":"Output","text":"
      The pipeline also needs output, where we define our database (Elasticsearch). The simplest way of doing so is defining an output like this:
       
      output {\n  elasticsearch {\n    hosts => [\"http://10.10.10.11:9200\", \"http://10.10.10.12:9200\"]\n    index => \"intelmq-%{+YYYY.MM}\"\n  }\n}\n
       
         
      • hosts - Elasticsearch host (or more) with the correct port (9200 by default)
        •  
      • index - name of the index where to insert data
        •  
       
      Tip
       
      Authors experience, hardware equipment and the amount of events collected led to having a separate index for each month. This might not necessarily suit your needs, but it is a suggested option.
       
      Warning
       
      By default the ELK stack uses insecure HTTP. It is possible to setup Security for secure connections and basic user management. This is possible with the Basic (free) licence since versions 6.8.0 and 7.1.0.
      "},{"location":"admin/database/elasticsearch/#configuring-elasticsearch","title":"Configuring Elasticsearch","text":"
      Configuring Elasticsearch is entirely up to you and should be consulted with the official documentation. What you will most likely need is something called index template mappings. IntelMQ provides a tool for generating such mappings. See ElasticMapper Tool.
       
      Danger
       
      Default installation of Elasticsearch database allows anyone with cURL and connection capability to have administrative access to the database. Make sure you secure your toys!
      "},{"location":"admin/database/mssql/","title":"MSSQL","text":""},{"location":"admin/database/mssql/#mssql","title":"MSSQL","text":"
      For MSSQL support, the library pymssql>=2.2 is required.
       
      To output data to MSSQL use SQL Output Bot with parameter engine set to mssql.
       
      For more information see SQL Output Bot documentation page.
      "},{"location":"admin/database/postgresql/","title":"PostgreSQL","text":""},{"location":"admin/database/postgresql/#using-postgresql-as-a-database-for-intelmq","title":"Using PostgreSQL as a database for IntelMQ","text":"
      The EventDB is a database (usually PostgreSQL) that gets filled with with data from IntelMQ using the SQL Output Bot.
      "},{"location":"admin/database/postgresql/#intelmq_psql_initdb","title":"intelmq_psql_initdb","text":"
      IntelMQ comes with the intelmq_psql_initdb command line tool designed to help with creating the EventDB. It creates in the first line:
       
         
      • A CREATE TABLE events statement with all valid IntelMQ fields as columns and correct types
        •  
      • Several indexes as examples for a good read & search performance
        •  
       
      Having an events table as outlined in the SQL file, IntelMQ's SQL Output Bot can write all received events into this database table.
       
      In addition, the script supports some additional features supporting use cases described later in this document:
       
         
      • --partition-key - for generating schema aligned with TimescaleDB or partitioned tables,
        •  
      • --separate-raws - for generating views and triggers needed to eventdb_raws_table (works also together with adjustments for partitioning).
        •  
       
      For a full list of supported parameters, call the script help using -h parameter.
       
      All elements of the generated SQL file can be adapted and extended before running the SQL file against a database, especially the indexes. Please review the generated script before applying.
       
      Be aware that if you create tables using another DB user that is used later by the output bot, you may need to adjust ownership or privileges in the database. If you have problems with database permissions, refer to PostgreSQL documentation <https://www.postgresql.org/docs/current/ddl-priv.html>.
      "},{"location":"admin/database/postgresql/#eventdb-utilities","title":"EventDB Utilities","text":"
      Some scripts related to the EventDB are located in the contrib/eventdb folder in the IntelMQ git repository.
      "},{"location":"admin/database/postgresql/#apply-malware-name-mapping","title":"Apply Malware Name Mapping","text":"
      The apply_mapping_eventdb.py script applies the malware name mapping to the EventDB. Source and destination columns can be given, also a local file. If no local file is present, the mapping can be downloaded on demand. It queries the database for all distinct malware names with the taxonomy \"malicious-code\" and sets another column to the malware family name.
      "},{"location":"admin/database/postgresql/#apply-domain-suffix","title":"Apply Domain Suffix","text":"
      The apply_domain_suffix.py script writes the public domain suffix to the source.domain_suffix / destination.domain_suffix columns, extracted from source.fqdn / destination.fqdn.
      "},{"location":"admin/database/postgresql/#usage","title":"Usage","text":"
      The Python scripts can connect to a PostgreSQL server with an eventdb database and an events table. The command line arguments interface for both scripts are the same. See --help for more information:
       
      apply_mapping_eventdb.py -h\napply_domain_suffix.py -h\n
      "},{"location":"admin/database/postgresql/#postgresql-trigger","title":"PostgreSQL trigger","text":"
      PostgreSQL trigger is a trigger keeping track of the oldest inserted/updated \"time.source\" data. This can be useful to (re-)generate statistics or aggregation data.
       
      The SQL script can be executed in the database directly.
      "},{"location":"admin/database/postgresql/#eventdb-statistics","title":"EventDB Statistics","text":"
      The EventDB provides a great base for statistical analysis of the data.
       
      The eventdb-stats repository contains a Python script that generates an HTML file and includes the Plotly JavaScript Open Source Graphing Library. By modifying the configuration file it is possible to configure various queries that are then displayed using graphs:
       
      "},{"location":"admin/database/postgresql/#using-eventdb-with-timescale-db","title":"Using EventDB with Timescale DB","text":"
      Timescale DB is a PostgreSQL extension to add time-series support, which is quite handy as you don't have to learn other syntaxes as you already know. You can use the SQL Queries as before, the extension will handle the rest. To see all limitations, please check the Timescale DB Documentation.
      "},{"location":"admin/database/postgresql/#what-is-time-series","title":"What is time-series?","text":"
      Time-series has been invented as traditional database design like relational or nosql are not made for time-based data. A big benefit of time-series instead of other database designs over a time-based search pattern is the performance. As IntelMQ uses data based upon time, this design is awesome & will give you a performance boost.
      "},{"location":"admin/database/postgresql/#how-to-choose-the-time-column","title":"How to choose the time column?","text":"
      To utilize the time-series, choose a column containing the right time. This is then used by you for manual queries and graphs, and also by the database itself for organizing the data.
       
      An Event has two fields that can be used for this: time.source or time.observation. Depending on your needs (tracking when the event occurred or when it was detected, if different), choose one of them.
       
      You can use the :ref:intelmq_psql_initdb tool to generate SQL schema valid for TimescaleDB by passing the partitioning key:
       
      intelmq_psql_initdb --partition-key \"time.source\"\n
      "},{"location":"admin/database/postgresql/#how-to-setup","title":"How to setup","text":"
      Thanks to TimescaleDB its very easy to setup.
       
         
      1. Choose your preferred Timescale DB environment & follow the installation instructions. 2. Now lets create a hypertable, which is the timescale DB time-series structure. SELECT create_hypertable('', 'time.source');. 3. Now our hypertable is setup & timescaleDB takes care of the rest. You can perform queries as usual, for further information please check Timescale DB Documentation.
        2.  
      "},{"location":"admin/database/postgresql/#how-to-upgrade-from-my-existing-database","title":"How to upgrade from my existing database?","text":"
      To update your existing database to use this awesome time-series feature, just follow the How to setup instruction. You can perform the hypertable command even on already existing databases. BUT there are some limitations from timescaleDB.
      "},{"location":"admin/database/postgresql/#separating-raw-values-in-postgresql-using-view-and-trigger","title":"Separating raw values in PostgreSQL using view and trigger","text":"
      In order to reduce the row size in the events table, the raw column's data can be separated from the other columns. While the raw-data is about 30-50% of the data row's size, it is not used in most database queries, as it serves only a backup functionality. Other possibilities to reduce or getting rid of this field are described in the FAQ, section faq-remove-raw-data.
       
      The steps described here are best performed before the events table is filled with data, but can as well be done with existing data.
       
      The approach requires four steps:
       
         
      1. An existing events table, see the first section of     this document.
        2.  
      2. Deleting or renaming the raw column of the     events table.
        3.  
      3. Creating a table raws which holds only the     raw field of the events and linking both tables using     the event_id.
        4.  
      4. Creating the view v_events which joins the tables     events and raws.
        5.  
      5. Creating the function process_v_events_insert and     INSERT trigger tr_events.
        6.  
       
      The last steps brings us several advantages:
       
         
      • All INSERT statements can contain all data, including     the raw field.
        •  
      • No code changes are needed in the IntelMQ output bot or your own     scripts. A migration is seamless.
        •  
      • PostgreSQL itself ensures that the data of both tables is consistent     and linked correctly.
        •  
       
      The complete SQL script can be generated using the intelmq_psql_initdb. It does not cover step 2 to avoid accidental data loss - you need to do this step manually.
      "},{"location":"admin/database/postgresql/#other-docs","title":"Other docs","text":"
      You have two basic choices to run PostgreSQL:
       
         
      1. on the same machine as intelmq, then you could use Unix sockets if available on your platform
        2.  
      2. on a different machine. In which case you would need to use a TCP connection and make sure you give the right    connection parameters to each psql or client call.
        3.  
       
      Make sure to consult your PostgreSQL documentation about how to allow network connections and authentication in case 2.
       
      PostgreSQL Version
       
      Any supported version of PostgreSQL should work (v>=9.2 as of Oct 2016) [1].
       
      If you use PostgreSQL server v >= 9.4, it gives you the possibility to use the time-zone formatting string \"OF\" for date-times and the GiST index for the CIDR type. This may be useful depending on how you plan to use the events that this bot writes into the database.
       
      How to install
       
      Use intelmq_psql_initdb to create initial SQL statements from harmonization.conf. The script will create the required table layout and save it as /tmp/initdb.sql
       
      You need a PostgreSQL database-user to own the result database. The recommendation is to use the name intelmq . There may already be such a user for the PostgreSQL database-cluster to be used by other bots. (For example from setting up the expert/certbund_contact bot.)
       
      Therefore if still necessary: create the database-user as postgresql superuser, which usually is done via the system user postgres:
       
      createuser --no-superuser --no-createrole --no-createdb --encrypted --pwprompt intelmq\n
       
      Create the new database:
       
      createdb --encoding='utf-8' --owner=intelmq intelmq-events\n
       
      (The encoding parameter should ensure the right encoding on platform where this is not the default.)
       
      Now initialize it as database-user intelmq (in this example a network connection to localhost is used, so you would get to test if the user intelmq can authenticate):
       
      psql -h localhost intelmq-events intelmq </tmp/initdb.sql\n
       
      PostgreSQL and null characters
       
      While null characters (0, not SQL \"NULL\") in TEXT and JSON/JSONB fields are valid, data containing null characters can cause troubles in some combinations of clients, servers and each settings. To prevent unhandled errors and data which can't be inserted into the database, all null characters are escaped (u0000) before insertion.
      "},{"location":"admin/database/splunk/","title":"Splunk","text":""},{"location":"admin/database/splunk/#sending-intelmq-events-to-splunk","title":"Sending IntelMQ events to Splunk","text":"
         
      1. Go to Splunk and configure in order to be able to receive     logs (intelmq events) to a TCP port
        2.  
      2. Use TCP output bot and configure accordingly to the Splunk     configuration that you applied.
        3.  
      "},{"location":"admin/database/sqlite/","title":"SQLite","text":""},{"location":"admin/database/sqlite/#sqlite","title":"SQLite","text":"
      Similarly to PostgreSQL, you can use intelmq_psql_initdb to create initial SQL statements from harmonization.conf. The script will create the required table layout and save it as /tmp/initdb.sql.
       
      Create the new database (you can ignore all errors since SQLite doesn't know all SQL features generated for PostgreSQL):
       
      sqlite3 your-db.db\nsqlite> .read /tmp/initdb.sql\n
       
      Then, set the database parameter to the your-db.db file path.
       
      To output data to SQLite use SQL Output Bot with parameter engine set to sqlite. For more information see SQL Output Bot documentation page.
      "},{"location":"admin/installation/dockerhub/","title":"DockerHub","text":""},{"location":"admin/installation/dockerhub/#installation-from-dockerhub","title":"Installation from DockerHub","text":"
      This guide provides instruction on how to install IntelMQ and it's components using Docker.
       
      Warning
       
      Docker installation is currently in Beta state and things might break. Consider this if you plan to use IntelMQ as a production level system.
       
      Warning
       
      Currently you can't manage your botnet via intelmqctl command line tool. You need to use IntelMQ-Manager currently!
       
      The latest IntelMQ image is hosted on Docker Hub and the image build instructions are in our intelmq-docker repository.
       
      Follow Docker Install and Docker-Compose Install instructions.
       
      Before you start using docker-compose or any docker related tools, make sure docker is running:
       
      # To start the docker daemon\nsystemctl start docker.service\n# To enable the docker daemon for the future\nsystemctl enable docker.service\n
      "},{"location":"admin/installation/dockerhub/#docker-with-docker-compose","title":"Docker with docker-compose","text":"
      Now we can download IntelMQ and start the containers. Navigate to your preferred installation directory and run the following commands:
       
      git clone https://github.com/certat/intelmq-docker.git --recursive\ncd intelmq-docker\nsudo docker-compose pull\nsudo docker-compose up\n
       
      Your installation should be successful now. You're now able to visit http://127.0.0.1:1337/ to access the intelmq-manager. You have to login with the username intelmq and the password intelmq, if you want to change the username or password, you can do this by adding the environment variables INTELMQ_API_USER for the username and INTELMQ_API_PASS for the password.
       
      Note
       
      If you get an Permission denied error, you should run chown -R $USER:$USER example_config
      "},{"location":"admin/installation/dockerhub/#docker-without-docker-compose","title":"Docker without docker-compose","text":"
      If not already installed, please install Docker.
       
      Navigate to your preferred installation directory and run git clone https://github.com/certat/intelmq-docker.git --recursive.
       
      You need to prepare some volumes & configs. Edit the left-side after -v, to change paths.
       
      Change redis_host to a running redis-instance. Docker will resolve it automatically. All containers are connected using Docker Networks.
       
      In order to work with your current infrastructure, you need to specify some environment variables
       
      sudo docker pull redis:latest\n\nsudo docker pull certat/intelmq-full:latest\n\nsudo docker pull certat/intelmq-nginx:latest\n\nsudo docker network create intelmq-internal\n\nsudo docker run -v ~/intelmq/example_config/redis/redis.conf:/redis.conf \\\n                --network intelmq-internal \\\n                --name redis \\\n                redis:latest\n\nsudo docker run --network intelmq-internal \\\n                --name nginx \\\n                certat/intelmq-nginx:latest\n\nsudo docker run -e INTELMQ_IS_DOCKER=\"true\" \\\n                -e INTELMQ_SOURCE_PIPELINE_BROKER: \"redis\" \\\n                -e INTELMQ_PIPELINE_BROKER: \"redis\" \\\n                -e INTELMQ_DESTIONATION_PIPELINE_BROKER: \"redis\" \\\n                -e INTELMQ_PIPELINE_HOST: redis \\\n                -e INTELMQ_SOURCE_PIPELINE_HOST: redis \\\n                -e INTELMQ_DESTINATION_PIPELINE_HOST: redis \\\n                -e INTELMQ_REDIS_CACHE_HOST: redis \\\n                -v $(pwd)/example_config/intelmq/etc/:/etc/intelmq/etc/ \\\n                -v $(pwd)/example_config/intelmq-api/config.json:/etc/intelmq/api-config.json \\\n                -v $(pwd)/intelmq_logs:/etc/intelmq/var/log \\\n                -v $(pwd)/intelmq_output:/etc/intelmq/var/lib/bots \\\n                -v ~/intelmq/lib:/etc/intelmq/var/lib \\\n                --network intelmq-internal \\\n                --name intelmq \\\n                certat/intelmq-full:latest\n
       
      If you want to use another username and password for the intelmq-manager / api login, additionally add two new environment variables.
       
      -e INTELMQ_API_USER: \"your username\"\n-e INTELMQ_API_PASS: \"your password\"\n
      "},{"location":"admin/installation/linux-packages/","title":"Linux Package","text":""},{"location":"admin/installation/linux-packages/#installation-as-linux-package","title":"Installation as Linux package","text":"
      This guide provides instructions on how to install IntelMQ and it's components from Linux distribution's package repository.
       
      Note
       
      Some bots may have additional dependencies which are mentioned in their own documentation.
      "},{"location":"admin/installation/linux-packages/#supported-os","title":"Supported OS","text":"
      Native packages are currently provided for the following Linux distributions:
       
         
      • Debian 11 (bullseye)
        •  
      • Debian 12 (bookworm)
        •  
      • openSUSE Tumbleweed
        •  
      • Ubuntu 20.04 (focal fossa)
        •  
      • Ubuntu 22.04 (jammy jellyfish)
        •  
      "},{"location":"admin/installation/linux-packages/#debian-11-and-12","title":"Debian 11 and 12","text":"
      Add the repository to the package manager and install IntelMQ (packages intelmq-api and intelmq-manager are optional):
       
      echo \"deb http://download.opensuse.org/repositories/home:/sebix:/intelmq/Debian_$(lsb_release -rs)/ /\" | sudo tee /etc/apt/sources.list.d/intelmq.list\ncurl -fsSL \"https://download.opensuse.org/repositories/home:sebix:intelmq/Debian_$(lsb_release -rs)/Release.key\" | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/intelmq.gpg > /dev/null\nsudo apt update\nsudo apt install intelmq intelmq-api intelmq-manager\n
      "},{"location":"admin/installation/linux-packages/#opensuse-tumbleweed","title":"openSUSE Tumbleweed","text":"
      Add the repository to the package manager and install IntelMQ (packages intelmq-api and intelmq-manager are optional):
       
      zypper addrepo https://download.opensuse.org/repositories/home:sebix:intelmq/openSUSE_Tumbleweed/home:sebix:intelmq.repo\nzypper refresh\nzypper install intelmq intelmq-api intelmq-manager\n
      "},{"location":"admin/installation/linux-packages/#ubuntu-2004-and-2204","title":"Ubuntu 20.04 and 22.04","text":"
      For Ubuntu you must enable the Universe repository which provides community-maintained free and open-source software.
       
      Add the repository to the package manager and install IntelMQ (packages intelmq-api and intelmq-manager are optional):
       
         
      1.  
        Open the file /etc/apt/sources.list in an editor of your choice. Use sudo or the root user.
         
        2.  
      2.  
        Append universe to this line: 
        deb http://[...].archive.ubuntu.com/ubuntu/ focal main universe\n
         
        3.  
      3.  
        Next, add the IntelMQ APT Repository for Ubuntu: 
        echo \"deb http://download.opensuse.org/repositories/home:/sebix:/intelmq/xUbuntu_$(lsb_release -rs)/ /\" | sudo tee /etc/apt/sources.list.d/intelmq.list\ncurl -fsSL \"https://download.opensuse.org/repositories/home:sebix:intelmq/xUbuntu_$(lsb_release -rs)/Release.key\" | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/intelmq.gpg > /dev/null\n
         
        4.  
      4.  
        Now update the list of available packages and install the IntelMQ packages: 
        sudo apt update\nsudo apt install intelmq intelmq-api intelmq-manager\n
         
        5.  
      "},{"location":"admin/installation/pypi/","title":"PyPI","text":""},{"location":"admin/installation/pypi/#installation-from-pypi","title":"Installation from PyPI","text":"
      This guide provides instruction on how to install IntelMQ and it's components using the Python Package Index (PyPI) repository.
       
      Note
       
      Some bots may have additional dependencies which are mentioned in their own documentation.
      "},{"location":"admin/installation/pypi/#installing-intelmq","title":"Installing IntelMQ","text":""},{"location":"admin/installation/pypi/#requirements","title":"Requirements","text":""},{"location":"admin/installation/pypi/#ubuntu-debian","title":"Ubuntu / Debian","text":"
      apt install python3-pip python3-dnspython python3-psutil python3-redis python3-requests python3-termstyle python3-tz python3-dateutil redis-server bash-completion jq\n# optional dependencies\napt install python3-pymongo python3-psycopg2\n
      "},{"location":"admin/installation/pypi/#opensuse","title":"openSUSE:","text":"
      zypper install python3-dateutil python3-dnspython python3-psutil python3-redis python3-requests python3-python-termstyle redis bash-completion jq\n# optional dependencies\nzypper in python3-psycopg2 python3-pymongo\n
      "},{"location":"admin/installation/pypi/#centos-8","title":"CentOS 8:","text":"
      dnf install epel-release\ndnf install python3-dateutil python3-dns python3-pip python3-psutil python3-redis python3-requests redis bash-completion jq\n# optional dependencies\ndnf install python3-psycopg2 python3-pymongo\n
      "},{"location":"admin/installation/pypi/#centos-7-rhel-7","title":"CentOS 7 / RHEL 7:","text":"
      Warning
       
      We no longer support already end-of-life Python 3.6, which is the last Python version officially packaged for CentOS 7. You can either use alternative Python source, or stay on the IntelMQ 3.0.2.
       
      yum install epel-release\nyum install python36 python36-dns python36-requests python3-setuptools redis bash-completion jq\nyum install gcc gcc-c++ python36-devel\n# optional dependencies\nyum install python3-psycopg2\n
      "},{"location":"admin/installation/pypi/#installation","title":"Installation","text":"
      The default installation directory is /opt/intelmq/.
       
      If you prefer to use Linux Standard Base (LSB) paths, set the following environment variable:
       
      export INTELMQ_PATHS_NO_OPT=1\n
       
      If you want to use custom installation directory, set the following environment variable:
       
      export INTELMQ_ROOT_DIR=/my-installation-directory-path\n
       
      Run the following commands to install IntelMQ. The provided tool intelmqsetup will create all the necessary directories and installs a default configuration for new setups. If you are using the LSB paths installation, change the --home-dir parameter to /var/lib/intelmq
       
      sudo --preserve-env=INTELMQ_PATHS_NO_OPT,INTELMQ_ROOT_DIR -i\npip3 install intelmq\n[[ ! -z \"$INTELMQ_PATHS_NO_OPT\" ]] && export HOME_DIR=/var/lib/intelmq || export HOME_DIR=${INTELMQ_ROOT_DIR:-/opt/intelmq}\nuseradd --system --user-group --home-dir $HOME_DIR --shell /bin/bash intelmq\nintelmqsetup\n
      "},{"location":"admin/installation/pypi/#installation-to-python-virtual-environment","title":"Installation to Python virtual environment","text":"
      sudo mkdir -m 755 /opt/intelmq\nsudo useradd --system --user-group --home-dir /opt/intelmq --shell /bin/bash intelmq\nsudo chown intelmq:intelmq /opt/intelmq/\nsudo -u intelmq python3 -m venv /opt/intelmq/venv\nsudo -u intelmq /opt/intelmq/venv/bin/pip install intelmq intelmq-api intelmq-manager\nsudo /opt/intelmq/venv/bin/intelmqsetup\n
      "},{"location":"admin/installation/pypi/#installing-intelmq-api-optional","title":"Installing IntelMQ API (optional)","text":"
      The intelmq-api packages ships:
       
         
      • api configuration file in ${PREFIX}/etc/intelmq/api-config.json
        •  
      • positions configuration for the intelmq-manager in {PREFIX}/etc/intelmq/manager/positions.conf
        •  
      • virtualhost configuration file for Apache 2 in ${PREFIX}/etc/intelmq/api-apache.conf
        •  
      • sudoers configuration file in ${PREFIX}/etc/intelmq/api-sudoers.conf
        •  
       
      The value of ${PREFIX} depends on your environment and is something like /usr/local/lib/pythonX.Y/dist-packages/ (where X.Y is your Python version).
       
      The virtualhost configuration file needs to be placed in the correct directory for your Apache 2 installation.
       
         
      • On Debian or Ubuntu, move the file to /etc/apache2/conf-available.d/ directory and then execute a2enconf api-apache.
        •  
      • On CentOS, RHEL or Fedora, move the file to /etc/httpd/conf.d/ directory.
        •  
      • On openSUSE, move the file to /etc/apache2/conf.d/ directory.
        •  
       
      Don't forget to reload your webserver afterwards.
       
      The api configuration file and the positions configuration file need to be placed in one of the following directories (based on your IntelMQ installation directory):
       
         
      • /etc/intelmq/
        •  
      • /opt/intelmq/etc/
        •  
      • [my-installation-directory-path]/etc/
        •  
       
      The sudoers configuration file should be placed in the /etc/sudoers.d/ directory and adapt the webserver username in this file. Set the file permissions to 0o440.
       
      Afterwards continue with the section Permissions below.
       
      IntelMQ 2.3.1 comes with a tool intelmqsetup which performs these set-up steps automatically. Please note that the tool is very new and may not detect all situations correctly. Please report us any bugs you are observing. The tools is idempotent, you can execute it multiple times.
      "},{"location":"admin/installation/pypi/#installing-intelmq-manager-optional","title":"Installing IntelMQ Manager (optional)","text":"
      To use the IntelMQ Manager web interface, it is required to have a working IntelMQ and IntelMQ API installation.
       
      For installation via pip, the situation is more complex. The intelmq-manager package does not contain ready-to-use files, they need to be built locally. First, lets install the Manager itself:
       
      pip3 install intelmq-manager\n
       
      If your system uses wheel-packages, not the source distribution, you can use the intelmqsetup tool. intelmqsetup which performs these set-up steps automatically but it may not detect all situations correctly. If it finds intelmq-manager installed, calls its build routine is called. The files are placed in /usr/share/intelmq_manager/html, where the default Apache configuration expect it.
       
      If your system used the dist-package or if you are using a local source, the tool may not do all required steps. To call the build routine manually, use intelmq-manager-build --output-dir your/preferred/output/directory/.
       
      intelmq-manager ships with a default configuration for the Apache webserver (manager-apache.conf):
       
      Alias /intelmq-manager /usr/share/intelmq_manager/html/\n\n<Directory /usr/share/intelmq_manager/html>\n    <IfModule mod_headers.c>\n    Header set Content-Security-Policy \"script-src 'self'\"\n    Header set X-Content-Security-Policy \"script-src 'self'\"\n    </IfModule>\n</Directory>\n
       
      This file needs to be placed in the correct place for your Apache 2 installation.
       
         
      • On Debian and Ubuntu, the file needs to be placed at /etc/apache2/conf-available.d/manager-apache.conf and then execute   a2enconf manager-apache.
        •  
      • On CentOS, RHEL and Fedora, the file needs to be placed at /etc/httpd/conf.d/ and reload the webserver.
        •  
      • On openSUSE, the file needs to be placed at /etc/apache2/conf.d/ and reload the webserver.
        •  
      "},{"location":"admin/integrations/cifv3/","title":"CIFv3","text":""},{"location":"admin/integrations/cifv3/#cifv3-integrations-in-intelmq","title":"CIFv3 integrations in IntelMQ","text":"
      CIF creates an accessible indicator store. A REST API is exposed to interact with the store and quickly process/share indicators. CIFv3 can correlate indicators via the UUID attribute.
      "},{"location":"admin/integrations/cifv3/#cif3-api-output","title":"CIF3 API Output","text":"
      Can be used to submit indicators to a CIFv3 instance by using the CIFv3 API.
       
      Look at the CIFv3 API Output Bot for more information.
      "},{"location":"admin/integrations/misp/","title":"MISP","text":""},{"location":"admin/integrations/misp/#misp-integrations-in-intelmq","title":"MISP integrations in IntelMQ","text":"
      While MISP and IntelMQ seem to solve similar problems in the first hindsight, their intentions and strengths differ significantly.
       
      In a nutshell, MISP stores manually curated indicators (called attributes) grouped in events. An event can have an arbitrary number of attributes. MISP correlates these indicators with each other and can synchronize the data between multiple MISP instances.
       
      On the other side, IntelMQ in it's essence (not considering the EventDB <eventdb>) has no state or database, but is stream-oriented. IntelMQ acts as a toolbox which can be configured as needed to automate processes of mass data with little or no human interaction At the end of the processing the data may land in some database or be sent to other systems.
       
      Both systems do not intend to replace each other or do compete. They integrate seamless and combine each other enabling more use-cases and
      "},{"location":"admin/integrations/misp/#misp-api-collector","title":"MISP API Collector","text":"
      The MISP API Collector fetches data from MISP via the MISP API .
       
      Look at the Bots documentation page for more information.
      "},{"location":"admin/integrations/misp/#misp-expert","title":"MISP Expert","text":"
      The MISP Expert searches MISP by using the MISP API for attributes/events matching the source.ip of the event. The MISP Attribute UUID and MISP Event ID of the newest attribute are added to the event.
       
      Look at the Bots documentation page for more information.
      "},{"location":"admin/integrations/misp/#misp-feed-output","title":"MISP Feed Output","text":"
      This bot creates a complete MISP feed ready to be configured in MISP as incoming data source.
       
      Look at the Bots documentation page for more information.
      "},{"location":"admin/integrations/misp/#misp-api-output","title":"MISP API Output","text":"
      Can be used to directly create MISP events in a MISP instance by using the MISP API.
       
      Look at the Bots documentation page for more information.
      "},{"location":"admin/integrations/n6/","title":"N6","text":""},{"location":"admin/integrations/n6/#intelmq-n6-integration","title":"IntelMQ - n6 Integration","text":"
      n6 is an Open Source Tool with very similar aims as IntelMQ: processing and distributing IoC data. The use-cases, architecture and features differ and both tools have non-overlapping strengths. n6 is maintained and developed by CERT.pl.
       
      Information about n6 can be found here:
       
         
      • Website: cert.pl/en/n6
        •  
      • Source Code:   github.com/CERT-Polska/n6
        •  
      • n6 documentation: n6.readthedocs.io
        •  
       
       
      "},{"location":"admin/integrations/n6/#data-format","title":"Data format","text":"
      The internal data representation differs between IntelMQ and n6, so any data exchange between the systems requires a format conversion. For example, in n6 one message can contain multiple IP addresses, but IntelMQ is intentionally restricted to one IP address per message. Therefore, one n6 event results in one or more IntelMQ events. Because of this, and some other naming differences and ambiguities, the format conversion is not bidirectional.
      "},{"location":"admin/integrations/n6/#data-exchange-interface","title":"Data exchange interface","text":"
      n6 offers a STOMP interface via the RabbitMQ broker, which can be used for both sending and receiving data. IntelMQ offers both a STOMP collector bot for receiving data from n6, as well as a STOMP output bot for sending data to n6 instances.
       
         
      • Stomp Collector Bot
        •  
      • N6 Parser Bot
        •  
      • Stomp Output Bot
        •  
      "},{"location":"admin/integrations/n6/#data-conversion","title":"Data conversion","text":"
      IntelMQ can parse n6 data using the n6 parser and n6 can parse IntelMQ data using the Intelmq n6 parser.
       
         
      • N6 Parser Bot
        •  
      "},{"location":"admin/integrations/n6/#complete-example","title":"Complete example","text":""},{"location":"admin/integrations/n6/#data-flow-n6-to-intelmq","title":"Data flow n6 to IntelMQ","text":""},{"location":"admin/integrations/n6/#data-flow-intelmq-to-n6","title":"Data flow IntelMQ to n6","text":""},{"location":"admin/integrations/n6/#certpl-data-feed","title":"CERT.pl Data feed","text":"
      CERT.pl offers data feed available to their partners through the STOMP interface. Our feeds documentation contains details how it can be enabled in IntelMQ: CERT.pl n6 STOMP stream
      "},{"location":"admin/integrations/n6/#webinput-csv","title":"Webinput CSV","text":"
      The IntelMQ Webinput CSV software can also be used together with n6. The documentation on this component can be found in the software's repository: https://github.com/certat/intelmq-webinput-csv/blob/master/docs/webinput-n6.md
      "},{"location":"admin/management/intelmq-api/","title":"IntelMQ API","text":""},{"location":"admin/management/intelmq-api/#managing-intelmq-api","title":"Managing IntelMQ API","text":""},{"location":"admin/management/intelmq-api/#running","title":"Running","text":"
      For development purposes and testing you can run directly using hug:
       
      hug -m intelmq_api.serve\n
      "},{"location":"admin/management/intelmq/","title":"IntelMQ","text":""},{"location":"admin/management/intelmq/#managing-intelmq","title":"Managing IntelMQ","text":""},{"location":"admin/management/intelmq/#required-services","title":"Required services","text":"
      You need to enable and start Redis if not already done. Using systemd it can be done with:
       
      systemctl enable redis.service\nsystemctl start redis.service\n
      "},{"location":"admin/management/intelmq/#introduction","title":"Introduction","text":"
      intelmqctl is the main tool to handle a intelmq installation. It handles the bots themselves and has some tools to handle the installation.
       
      Should you get lost any time, just use the --help after any argument for further explanation.
       
      > intelmqctl run file-output --help\n
      "},{"location":"admin/management/intelmq/#manage-the-botnet","title":"Manage the botnet","text":"
      In IntelMQ, the botnet is the set of all currently configured and enabled bots. All configured bots have their configuration in runtime.yaml. By default, all bots are enabled.
       
      If no bot id is given, the command applies to all bots / the botnet. All commands except the start action are applied to all bots. But only enabled bots are started.
       
      In the examples below, a very minimal botnet is used.
      "},{"location":"admin/management/intelmq/#start","title":"start","text":"
      The start action applies to all bots which are enabled.
       
      > intelmqctl start\nStarting abusech-domain-parser...\nabusech-domain-parser is running.\nStarting abusech-feodo-domains-collector...\nabusech-feodo-domains-collector is running.\nStarting deduplicator-expert...\ndeduplicator-expert is running.\nfile-output is disabled.\nBotnet is running.\n
       
      As we can file-output is disabled and thus has not been started. You can always explicitly start disabled bots.
      "},{"location":"admin/management/intelmq/#stop","title":"stop","text":"
      The stop action applies to all bots. Assume that all bots have been running:
       
      > intelmqctl stop\nStopping Botnet...\nStopping abusech-domain-parser...\nabusech-domain-parser is stopped.\nStopping abusech-feodo-domains-collector...\nabusech-feodo-domains-collector is stopped.\nStopping deduplicator-expert...\ndeduplicator-expert is stopped.\nStopping file-output...\nfile-output is stopped.\nBotnet is stopped.\n
      "},{"location":"admin/management/intelmq/#status","title":"status","text":"
      With this command we can see the status of all configured bots. Here, the botnet was started beforehand:
       
      > intelmqctl status\nabusech-domain-parser is running.\nabusech-feodo-domains-collector is running.\ndeduplicator-expert is running.\nfile-output is disabled.\n
       
      And if the disabled bot has also been started:
       
      > intelmqctl status\nabusech-domain-parser is running.\nabusech-feodo-domains-collector is running.\ndeduplicator-expert is running.\nfile-output is running.\n
       
      If the botnet is stopped, the output looks like this:
       
      > intelmqctl status\nabusech-domain-parser is stopped.\nabusech-feodo-domains-collector is stopped.\ndeduplicator-expert is stopped.\nfile-output is disabled.\n
      "},{"location":"admin/management/intelmq/#restart","title":"restart","text":"
      The same as start and stop consecutively.
      "},{"location":"admin/management/intelmq/#reload","title":"reload","text":"
      The same as reload of every bot.
      "},{"location":"admin/management/intelmq/#enable-disable","title":"enable / disable","text":"
      The sub commands enable and disable set the corresponding flags in runtime.yaml.
       
      > intelmqctl status\nfile-output is stopped.\nmalware-domain-list-collector is stopped.\nmalware-domain-list-parser is stopped.\n> intelmqctl disable file-output\n> intelmqctl status\nfile-output is disabled.\nmalware-domain-list-collector is stopped.\nmalware-domain-list-parser is stopped.\n> intelmqctl enable file-output\n> intelmqctl status\nfile-output is stopped.\nmalware-domain-list-collector is stopped.\nmalware-domain-list-parser is stopped.\n
      "},{"location":"admin/management/intelmq/#manage-individual-bots","title":"Manage individual bots","text":"
      As all init systems, intelmqctl has the methods start, stop, restart, reload and status.
      "},{"location":"admin/management/intelmq/#start_1","title":"start","text":"
      This will start the bot with the ID file-output. A file with it's PID will be created in /opt/intelmq/var/run/[bot-id].pid.
       
      > intelmqctl start file-output\nStarting file-output...\nfile-output is running.\n
       
      If the bot is already running, it won't be started again:
       
      > intelmqctl start file-output\nfile-output is running.\n
      "},{"location":"admin/management/intelmq/#stop_1","title":"stop","text":"
      If the PID file does exist, a SIGINT will be sent to the process. After 0.25s we check if the process is running. If not, the PID file will be removed.
       
      > intelmqctl stop file-output\nStopping file-output...\nfile-output is stopped.\n
       
      If there's no running bot, there's nothing to do.
       
      > intelmqctl stop file-output\nfile-output was NOT RUNNING.\n
       
      If the bot did not stop in 0.25s, intelmqctl will say it's still running:
       
      > intelmqctl stop file-output\nfile-output is still running\n
      "},{"location":"admin/management/intelmq/#status_1","title":"status","text":"
      Checks for the PID file and if the process with the given PID is alive. If the PID file exists, but the process does not exist, it will be removed.
       
      > intelmqctl status file-output\nfile-output is stopped.\n> intelmqctl start file-output\nStarting file-output...\nfile-output is running.\n> intelmqctl status file-output\nfile-output is running.\n
      "},{"location":"admin/management/intelmq/#restart_1","title":"restart","text":"
      The same as stop and start consecutively.
       
      > intelmqctl restart file-output\nStopping file-output...\nfile-output is stopped.\nStarting file-output...\nfile-output is running.\n
      "},{"location":"admin/management/intelmq/#reload_1","title":"reload","text":"
      Sends a SIGHUP to the bot, which will then reload the configuration.
       
      > intelmqctl reload file-output\nReloading file-output ...\nfile-output is running.\n
       
      If the bot is not running, we can't reload it:
       
      > intelmqctl reload file-output\nfile-output was NOT RUNNING.\n
      "},{"location":"admin/management/intelmq/#run","title":"run","text":"
      This command is used for debugging purposes.
       
      If launched with no arguments, the bot will call its init method and start processing messages as usual -- but you see everything happens.
       
      > intelmqctl run file-output\nfile-output: RestAPIOutputBot initialized with id file-output and version 3.5.2 as process 12345.\nfile-output: Bot is starting.\nfile-output: Loading source pipeline and queue 'file-output-queue'.\nfile-output: Connected to source queue.\nfile-output: No destination queues to load.\nfile-output: Bot initialization completed.\nfile-output: Waiting for incoming message.\n
       
      Note that if another instance of the bot is running, only warning will be displayed.
       
      > intelmqctl run file-output\nMain instance of the bot is running in the background. You may want to launch: intelmqctl stop file-output\n
       
      You can set the log level with the -l flag, e.g. -l DEBUG. For the 'console' subcommand, 'DEBUG' is the default.
      "},{"location":"admin/management/intelmq/#console","title":"console","text":"
      This command is used for debugging purposes.
       
      If launched with console argument, you get a pdb live console; or ipdb or pudb consoles if they were previously installed (I.E. pip3 install ipdb --user).
       
      > intelmqctl run file-output console\n*** Using console ipdb. Please use 'self' to access to the bot instance properties. ***\nipdb> self. ...\n
       
      You may specify the desired console in the next argument.
       
      > intelmqctl run file-output console pudb\n
      "},{"location":"admin/management/intelmq/#message","title":"message","text":"
      Operate directly with the input / output pipelines.
       
      If get is the parameter, you see the message that waits in the input (source or internal) queue. If the argument is pop, the message gets popped as well.
       
      > intelmqctl run file-output message get\nfile-output: Waiting for a message to get...\n{\n    \"classification.type\": \"c&c\",\n    \"feed.url\": \"https://example.com\",\n    \"raw\": \"1233\",\n    \"source.ip\": \"1.2.3.4\",\n    \"time.observation\": \"2017-05-17T22:00:33+00:00\",\n    \"time.source\": \"2017-05-17T22:00:32+00:00\"\n}\n
       
      To send directly to the bot's output queue, just as it was sent by self.send_message() in bot's process() method, use the send argument. In our case of file-output, it has no destination queue so that nothing happens.
       
      > intelmqctl run file-output message send '{\"time.observation\": \"2017-05-17T22:00:33+00:00\", \"time.source\": \"2017-05-17T22:00:32+00:00\"}'\nfile-output: Bot has no destination queues.\n
       
      Note, if you would like to know possible parameters of the message, put a wrong one -- you will be prompted if you want to list all the current bot harmonization.
      "},{"location":"admin/management/intelmq/#process","title":"process","text":"
      With no other arguments, bot's process() method will be run one time.
       
      > intelmqctl run file-output process\nfile-output: Bot is starting.\nfile-output: Bot initialization completed.\nfile-output: Processing...\nfile-output: Waiting for incoming message.\nfile-output: Received message {'raw': '1234'}.\n
       
      If run with --dryrun|-d flag, the message gets never really popped out from the source or internal pipeline, nor sent to the output pipeline. Plus, you receive a note about the exact moment the message would get sent, or acknowledged. If the message would be sent to a non-default path, the name of this path is printed on the console.
       
      > intelmqctl run file-output process -d\nfile-output:  * Dryrun only, no message will be really sent through.\n...\nfile-output: DRYRUN: Message would be acknowledged now!\n
       
      You may trick the bot to process a JSON instead of the Message in its pipeline with --msg|-m flag.
       
      > intelmqctl run file-output process -m '{\"source.ip\":\"1.2.3.4\"}'\nfile-output:  * Message from cli will be used when processing.\n...\n
       
      If you wish to display the processed message as well, you the --show-sent|-s flag. Then, if sent through (either with --dryrun or without), the message gets displayed as well.
      "},{"location":"admin/management/intelmq/#disable","title":"disable","text":"
      Sets the enabled flag in the runtime configuration of the bot to false. By default, all bots are enabled.
       
      Example output:
       
      > intelmqctl status file-output\nfile-output is stopped.\n> intelmqctl disable file-output\n> intelmqctl status file-output\nfile-output is disabled.\n
      "},{"location":"admin/management/intelmq/#enable","title":"enable","text":"
      Sets the enabled flag in the runtime configuration of the bot to true.
       
      Example output:
       
      > intelmqctl status file-output\nfile-output is disabled.\n> intelmqctl enable file-output\n> intelmqctl status file-output\nfile-output is stopped.\n
      "},{"location":"admin/management/intelmq/#list-bots","title":"List bots","text":"
      intelmqctl list bots does list all configured bots and their description.
      "},{"location":"admin/management/intelmq/#list-queues","title":"List queues","text":"
      intelmqctl list queues shows all queues which are currently in use according to the configuration and how much events are in it:
       
      > intelmqctl list queues\nabusech-domain-parser-queue - 0\nabusech-domain-parser-queue-internal - 0\ndeduplicator-expert-queue - 0\ndeduplicator-expert-queue-internal - 0\nfile-output-queue - 234\nfile-output-queue-internal - 0\n
       
      Use the -q or --quiet flag to only show non-empty queues:
       
      > intelmqctl list queues -q\nfile-output-queue - 234\n
       
      The --sum or --count flag will show the sum of events on all queues:
       
      > intelmqctl list queues --sum\n42\n
      "},{"location":"admin/management/intelmq/#logging","title":"Logging","text":"
      intelmqctl can show the last log lines for a bot, filtered by the log level.
       
      Logs are stored in /opt/intelmq/var/log/ or /var/log/intelmq/ directory. In case of failures, messages are dumped to the same directory with the file extension .dump.
       
      See the help page for more information.
      "},{"location":"admin/management/intelmq/#check","title":"Check","text":"
      This command will do various sanity checks on the installation and especially the configuration.
      "},{"location":"admin/management/intelmq/#orphaned-queues","title":"Orphaned Queues","text":"
      The intelmqctl check tool can search for orphaned queues. \"Orphaned queues\" are queues that have been used in the past and are no longer in use. For example you had a bot which you removed or renamed afterwards, but there were still messages in it's source queue. The source queue won't be renamed automatically and is now disconnected. As this queue is no longer configured, it won't show up in the list of IntelMQ's queues too. In case you are using redis as message broker, you can use the redis-cli tool to examine or remove these queues:
       
      redis-cli -n 2\nkeys * # lists all existing non-empty queues\nllen [queue-name] # shows the length of the queue [queue-name]\nlindex [queue-name] [index] # show the [index]'s message of the queue [queue-name]\ndel [queue-name] # remove the queue [queue-name]\n
       
      To ignore certain queues in this check, you can set the parameter intelmqctl_check_orphaned_queues_ignore in the defaults configuration file. For example:
       
      \"intelmqctl_check_orphaned_queues_ignore\": [\"Taichung-Parser\"]\n
      "},{"location":"admin/management/intelmq/#configuration-upgrade","title":"Configuration upgrade","text":"
      The intelmqctl upgrade-config function upgrade, upgrade the configuration from previous versions to the current one. It keeps track of previously installed versions and the result of all \"upgrade functions\" in the \"state file\", locate in the $var_state_path/state.json /opt/intelmq/var/lib/state.json or /var/lib/intelmq/state.json).
       
      This function has been introduced in version 2.0.1.
       
      It makes backups itself for all changed files before every run. Backups are overridden if they already exists. So make sure to always have a backup of your configuration just in case.
      "},{"location":"admin/management/intelmq/#output-type","title":"Output type","text":"
      intelmqctl can be used as command line tool, as library and as tool by other programs. If called directly, it will print all output to the console (stderr). If used as python library, the python types themselves are returned. The third option is to use machine-readable JSON as output (used by other managing tools).
      "},{"location":"admin/management/intelmq/#exit-code","title":"Exit code","text":"
      In case of errors, unsuccessful operations, the exit code is higher than 0. For example, when running intelmqctl start and one enabled bot is not running, the exit code is 1. The same is valid for e.g. intelmqctl status, which can be used for monitoring, and all other operations.
      "},{"location":"admin/management/intelmq/#error-handling","title":"Error Handling","text":"
      When bots are failing due to bad input data or programming errors, they can dump the problematic message to a file along with a traceback, if configured accordingly. These dumps are saved at in the logging directory as [botid].dump as JSON files. IntelMQ comes with an inspection and reinjection tool, called intelmqdump. It is an interactive tool to show all dumped files and the number of dumps per file. Choose a file by bot-id or listed numeric id. You can then choose to delete single entries from the file with e 1,3,4, show a message in more readable format with s 1 (prints the raw-message, can be long!), recover some messages and put them back in the pipeline for the bot by a or r 0,4,5. Or delete the file with all dumped messages using d.
       
      intelmqdump -h\nusage:\n    intelmqdump [botid]\n    intelmqdump [-h|--help]\n\nintelmqdump can inspect dumped messages, show, delete or reinject them into\nthe pipeline. It's an interactive tool, directly start it to get a list of\navailable dumps or call it with a known bot id as parameter.\n\npositional arguments:\n  botid       botid to inspect dumps of\n\noptional arguments:\n  -h, --help  show this help message and exit\n  --truncate TRUNCATE, -t TRUNCATE\n                        Truncate raw-data with more characters than given. 0 for no truncating. Default: 1000.\n\nInteractive actions after a file has been selected:\n- r, Recover by IDs\n  > r id{,id} [queue name]\n  > r 3,4,6\n  > r 3,7,90 modify-expert-queue\n  The messages identified by a consecutive numbering will be stored in the\n  original queue or the given one and removed from the file.\n- a, Recover all\n  > a [queue name]\n  > a\n  > a modify-expert-queue\n  All messages in the opened file will be recovered to the stored or given\n  queue and removed from the file.\n- d, Delete entries by IDs\n  > d id{,id}\n  > d 3,5\n  The entries will be deleted from the dump file.\n- d, Delete file\n  > d\n  Delete the opened file as a whole.\n- s, Show by IDs\n  > s id{,id}\n  > s 0,4,5\n  Show the selected IP in a readable format. It's still a raw format from\n  repr, but with newlines for message and traceback.\n- e, Edit by ID\n  > e id\n  > e 0\n  > e 1,2\n  Opens an editor (by calling `sensible-editor`) on the message. The modified message is then saved in the dump.\n- q, Quit\n  > q\n\n$ intelmqdump\n id: name (bot id)                    content\n  0: alienvault-otx-parser            1 dumps\n  1: cymru-whois-expert               8 dumps\n  2: deduplicator-expert              2 dumps\n  3: dragon-research-group-ssh-parser 2 dumps\n  4: file-output2                     1 dumps\n  5: fraunhofer-dga-parser            1 dumps\n  6: spamhaus-cert-parser             4 dumps\n  7: test-bot                         2 dumps\nWhich dump file to process (id or name)? 3\nProcessing dragon-research-group-ssh-parser: 2 dumps\n  0: 2015-09-03T13:13:22.159014 InvalidValue: invalid value u'NA' (<type 'unicode'>) for key u'source.asn'\n  1: 2015-09-01T14:40:20.973743 InvalidValue: invalid value u'NA' (<type 'unicode'>) for key u'source.asn'\n(r)ecover by ids, recover (a)ll, delete (e)ntries, (d)elete file, (s)how by ids, (q)uit, edit id (v)? d\nDeleted file /opt/intelmq/var/log/dragon-research-group-ssh-parser.dump\n
       
      Bots and the intelmqdump tool use file locks to prevent writing to already opened files. Bots are trying to lock the file for up to 60 seconds if the dump file is locked already by another process (intelmqdump) and then give up. Intelmqdump does not wait and instead only shows an error message.
       
      By default, the show command truncates the raw field of messages at 1000 characters to change this limit or disable truncating at all (value 0), use the --truncate parameter.
      "},{"location":"admin/management/intelmq/#known-issues","title":"Known issues","text":"
      The currently implemented process managing using PID files is very erroneous.
      "},{"location":"admin/utilities/bash-completion/","title":"Bash Completion","text":""},{"location":"admin/utilities/bash-completion/#bash-completion","title":"Bash Completion","text":"
      To enable bash completion on intelmqctl and intelmqdump in order to help you run the commands in an easy manner, follow the installation process here.
       
      Bug
       
      This section of the documentation is currently incomplete and will be added later.
      "},{"location":"dev/adding-feeds/","title":"Adding Feeds","text":""},{"location":"dev/adding-feeds/#adding-feeds","title":"Adding Feeds","text":"
      Adding a feed doesn't necessarily require any programming experience. There are several collector and parser bots intended for general use. Depending on the data source you are trying to add as a feed, it might be only a matter of creating a working combination of collector bot (such as URL Fetcher) configuration and a parser bot (such as CSV parser) configuration. When you are satisfied with the configurations, add it to the intelmq/etc/feeds.yaml file using the following template and open a pull request!
       
      <NAME OF THE FEED PROVIDER>:\n    <NAME OF THE FEED>:\n      description: <DESCRIPTION OF WHAT KIND OF DATA THE FEED PROVIDES>\n      additional_information: <ANY ADDITIONAL INFORMATION>\n      documentation: <FEED HOMEPAGE/DOCUMENTATION URL>\n      revision: <DATE WHEN YOU ADDED THIS FEED>\n      public: <TRUE/FALSE IF THE DATA SOURCE IS PUBLICLY AVAILABLE>\n      bots:\n        collector:\n          module: <MODULE USED FOR THE COLLECTOR BOT>\n          parameters:\n            name: __FEED__ # KEEP AS IT IS\n            provider: __PROVIDER__  # KEEP AS IT IS\n            <ADDITIONAL COLLECTOR BOT PARAMETERS>\n        parser:\n          module: <MODULE USED FOR THE PARSER BOT>\n          parameters:\n            <ADDITIONAL PARSER BOT PARAMETERS>\n
       
      If the data source utilizes some unusual way of distribution or uses a custom format for the data it might be necessary to develop specialized bot(s) for this particular data source. Always try to use existing bots before you start developing your own. Please also consider extending an existing bot if your use-case is close enough to it's features. If you are unsure which way to take, start an issue and you will receive guidance.
      "},{"location":"dev/adding-feeds/#feeds-wishlist","title":"Feeds Wishlist","text":"
      This is a list with potentially interesting data sources, which are either currently not supported or the usage is not clearly documented in IntelMQ. If you want to contribute new feeds to IntelMQ, this is a great place to start!
       
      Note
       
      Some of the following data sources might better serve as an expert bot for enriching processed events.
       
         
      • Lists of feeds:
           
        • threatfeeds.io
          •  
        • TheCyberThreat
          •  
        • sbilly: Awesome Security
          •  
        • pannoniait:Backlists
          •  
        • hslatman:awesome-threat-intelligence
          •  
        • Zeek Intelligence Feeds
          •  
        • imuledx OSING feeds
          •  
         
        •  
      • Some third party intelmq bots: NRDCS IntelMQ fork
        •  
      • List of potentially interesting data sources:
           
        • Abuse.ch SSL Blacklists
          •  
        • AbuseIPDB
          •  
        • Adblock Plus
          •  
        • apivoid IP Reputation API
          •  
        • Anomali Limo Free Intel Feed
          •  
        • APWG's ecrimex
          •  
        • Avast Threat Intel IoCs of dark matter repository
          •  
        • Berkeley
          •  
        • Binary Defense
          •  
        • Bot Invaders Realtime tracker
          •  
        • Botherder Targetedthreats
          •  
        • Botscout Last Caught
          •  
        • botvrij
          •  
        • Carbon Black Feeds
          •  
        • CERT.pl Phishing Warning List
          •  
        • Chaos Reigns
          •  
        • Critical Stack
          •  
        • Cruzit
          •  
        • Cyber Crime Tracker
          •  
        • drb-ra C2IntelFeeds
          •  
        • DNS DB API
          •  
        • ESET Malware Indicators of Compromise
          •  
        • Facebook Threat Exchange
          •  
        • FilterLists
          •  
        • Firehol IPLists
          •  
        • Google Webmaster Alerts
          •  
        • GPF Comics DNS Blacklist
          •  
        • Greensnow
          •  
        • Greynoise
          •  
        • HP Feeds
          •  
        • IBM X-Force Exchange
          •  
        • ImproWare AntiSpam
          •  
        • ISightPartners
          •  
        • James Brine
          •  
        • Joewein
          •  
        • Maltrail:
             
          • Malware
            •  
          • Suspicious
            •  
          • Mass Scanners   (for whitelisting)
            •  
           
          •  
        • Malshare
          •  
        • MalSilo Malware URLs
          •  
        • Malware Config
          •  
        • Malware DB (cert.pl)
          •  
        • MalwareInt
          •  
        • Malware Must Die
          •  
        • Manity Spam IP addresses
          •  
        • Marc Blanchard DGA Domains
          •  
        • MaxMind Proxies
          •  
        • mIRC Servers
          •  
        • MISP Warning Lists
          •  
        • Monzymerza
          •  
        • Multiproxy
          •  
        • Neo23x0 signature-base
          •  
        • OpenBugBounty
          •  
        • Phishing Army
          •  
        • Phishstats (offers JSON API and CSV download)
          •  
        • Project Honeypot (#284)
          •  
        • RST Threat Feed (offers a free and a commercial feed)
          •  
        • SANS ISC
          •  
        • ShadowServer Sandbox API
          •  
        • Shodan search API
          •  
        • Snort
          •  
        • stopforumspam Toxic IP addresses and domains
          •  
        • Spamhaus Botnet Controller List
          •  
        • SteveBlack Hosts File
          •  
        • The Haleys
          •  
        • Threat Crowd
          •  
        • Threat Grid
          •  
        • Threatstream
          •  
        • TotalHash
          •  
        • UCE Protect
          •  
        • Unit 42 Public Report IOCs
          •  
        • URI BL
          •  
        • urlscan.io
          •  
        • Virustotal
          •  
        • virustream
          •  
        • VoIP Blacklist
          •  
        • YourCMC
          •  
         
        •  
      "},{"location":"dev/bot-development/","title":"Bot Development","text":""},{"location":"dev/bot-development/#bot-development","title":"Bot Development","text":"
      Here you should find everything you need to develop a new bot.
      "},{"location":"dev/bot-development/#steps","title":"Steps","text":"
         
      1. Create appropriately placed and named python file.
        2.  
      2. Use correct parent class.
        3.  
      3. Code the functionality you want (with mixins, inheritance, etc).
        4.  
      4. Create appropriately placed test file.
        5.  
      5. Prepare code for testing your bot.
        6.  
      6. Add documentation for your bot.
        7.  
      7. Add changelog and news info.
        8.  
      "},{"location":"dev/bot-development/#layout-rules","title":"Layout Rules","text":"
      intelmq/\n  lib/\n    bot.py\n    cache.py\n    message.py\n    pipeline.py\n    utils.py\n  bots/\n    collector/\n      <bot name>/\n            collector.py\n    parser/\n      <bot name>/\n            parser.py\n    expert/\n      <bot name>/\n            expert.py\n    output/\n      <bot name>/\n            output.py\n  etc/\n    runtime.yaml\n
       
      Assuming you want to create a bot for a new 'Abuse.ch' feed. It turns out that here it is necessary to create different parsers for the respective kind of events (e.g. malicious URLs). Therefore, the usual hierarchy intelmq/bots/parser/<FEED>/parser.py would not be suitable because it is necessary to have more parsers for each Abuse.ch Feed. The solution is to use the same hierarchy with an additional \"description\" in the file name, separated by underscore. Also see the section Directories and Files naming.
       
      Example (including the current ones):
       
      /intelmq/bots/parser/abusech/parser_domain.py\n/intelmq/bots/parser/abusech/parser_ip.py\n/intelmq/bots/parser/abusech/parser_ransomware.py\n/intelmq/bots/parser/abusech/parser_malicious_url.py\n
      "},{"location":"dev/bot-development/#directories-hierarchy-on-default-installation","title":"Directories Hierarchy on Default Installation","text":"
         
      • Configuration Files Path: /opt/intelmq/etc/
        •  
      • PID Files Path: /opt/intelmq/var/run/
        •  
      • Logs Files and dumps Path: /opt/intelmq/var/log/
        •  
      • Additional Bot Files Path, e.g. templates or databases:   /opt/intelmq/var/lib/bots/[bot-name]/
        •  
      "},{"location":"dev/bot-development/#directories-and-files-naming","title":"Directories and Files naming","text":"
      Any directory and file of IntelMQ has to follow the Directories and Files naming. Any file name or folder name has to:
       
         
      • be represented with lowercase and in case of the name has multiple words, the spaces between them must be removed or replaced by underscores
        •  
      • be self-explaining what the content contains.
        •  
       
      In the bot directories name, the name must correspond to the feed provider. If necessary and applicable the feed name can and should be used as postfix for the filename.
       
      Examples:
       
      intelmq/bots/parser/taichung/parser.py\nintelmq/bots/parser/cymru/parser_full_bogons.py\nintelmq/bots/parser/abusech/parser_ransomware.py\n
      "},{"location":"dev/bot-development/#guide","title":"Guide","text":""},{"location":"dev/bot-development/#naming-your-bot-class","title":"Naming your bot class","text":"
      Class name of the bot (ex: PhishTank Parser) must correspond to the type of the bot (ex: Parser) e.g. PhishTankParserBot
      "},{"location":"dev/bot-development/#choosing-the-parent-class","title":"Choosing the parent class","text":"
      Please use the correct bot type as parent class for your bot. The intelmq.lib.bot module contains the following classes:
       
         
      • CollectorBot
        •  
      • ParserBot
        •  
      • ExpertBot
        •  
      • OutputBot
        •  
      "},{"location":"dev/bot-development/#template","title":"Template","text":"
      Please adjust the doc strings accordingly and remove the in-line comments (#).
       
      \"\"\"\nSPDX-FileCopyrightText: 2021 Your Name\nSPDX-License-Identifier: AGPL-3.0-or-later\n\nParse data from example.com, be a nice ExampleParserBot.\n\nDocument possible necessary configurations.\n\"\"\"\nimport sys\n\n# imports for additional libraries and intelmq\nfrom intelmq.lib.bot import ParserBot\n\n\nclass ExampleParserBot(ParserBot):\n    option1: str = \"defaultvalue\"\n    option2: bool = False\n\n    def process(self):\n        report = self.receive_message()\n\n        event = self.new_event(report)  # copies feed.name, time.observation\n        ...  # implement the logic here\n        event.add('source.ip', '127.0.0.1')\n        event.add('extra', {\"os.name\": \"Linux\"})\n        if self.option2:\n            event.add('extra', {\"customvalue\": self.option1})\n\n        self.send_message(event)\n        self.acknowledge_message()\n\n\nBOT = ExampleParserBot\n
       
      Any attributes of the bot that are not private can be set by the user using the IntelMQ configuration settings.
       
      There are some names with special meaning. These can be used i.e. called:
       
         
      • stop: Shuts the bot down.
        •  
      • receive_message
        •  
      • send_message
        •  
      • acknowledge_message: see next section
        •  
      • start: internal method to run the bot
        •  
       
      These can be defined:
       
         
      • init: called at startup, use it to set up the bot (initializing classes, loading files etc)
        •  
      • process: processes the messages
        •  
      • shutdown: To Gracefully stop the bot, e.g. terminate connections
        •  
       
      All other names can be used freely.
      "},{"location":"dev/bot-development/#mixins","title":"Mixins","text":"
      For common settings and methods you can use mixins from intelmq.lib.mixins. To use the mixins, just let your bot inherit from the Mixin class (in addition to the inheritance from the Bot class). For example:
       
      class HTTPCollectorBot(CollectorBot, HttpMixin):\n
       
      The following mixins are available:
       
         
      • HttpMixin
        •  
      • SqlMixin
        •  
      • CacheMixin
        •  
       
      The HttpMixin provides the HTTP attributes described in common-parameters and the following methods:
       
         
      • http_get takes an URL as argument. Any other arguments get passed to the request.Session.get method. http_get   returns a   requests.Response.
        •  
      • http_session can be used if you ever want to work with the session object directly. It takes no arguments and   returns the bots request.Session.
        •  
       
      The SqlMixin provides methods to connect to SQL servers. Inherit this Mixin so that it handles DB connection for you. You do not have to bother:
       
         
      • connecting database in the self.init() method, self.cur will be set in the __init__()
        •  
      • catching exceptions, just call self.execute() instead of   self.cur.execute()
        •  
      • self.format_char will be set to '%s' in PostgreSQL and to '?' in SQLite
        •  
       
      The CacheMixin provides methods to cache values for bots in a Redis database. It uses the following attributes:
       
         
      • redis_cache_host: str = \"127.0.0.1\"
        •  
      • redis_cache_port: int = 6379
        •  
      • redis_cache_db: int = 9
        •  
      • redis_cache_ttl: int = 15
        •  
      • redis_cache_password: Optional[str] = None
        •  
       
      and provides the methods:
       
         
      • cache_exists
        •  
      • cache_get
        •  
      • cache_set
        •  
      • cache_flush
        •  
      • cache_get_redis_instance
        •  
      "},{"location":"dev/bot-development/#pipeline-interactions","title":"Pipeline Interactions","text":"
      We can call three methods related to the pipeline:
       
         
      • self.receive_message(): The pipeline handler pops one message     from the internal queue if possible. Otherwise one message from     the sources list is popped, and added it to an internal queue. In     case of errors in process handling, the message can still be found     in the internal queue and is not lost. The bot class unravels the     message a creates an instance of the Event or Report class.
        •  
      • self.send_message(event, path=\"_default\"): Processed     message is sent to destination queues. It is possible to change     the destination queues by optional path parameter.
        •  
      • self.acknowledge_message(): Message formerly received by     receive_message is removed from the internal     queue. This should always be done after processing and after the     sending of the new message. In case of errors, this function is     not called and the message will stay in the internal queue waiting     to be processed again.
        •  
      "},{"location":"dev/bot-development/#logging","title":"Logging","text":""},{"location":"dev/bot-development/#log-messages-format","title":"Log Messages Format","text":"
      Log messages have to be clear and well formatted. The format is the following:
       
      Format:
       
      <timestamp> - <bot id> - <log level> - <log message>\n
       
      Rules:
       
         
      • the Log message MUST follow the common rules of a sentence, beginning with uppercase and ending with period.
        •  
      • the sentence MUST describe the problem or has useful information to give to an inexperienced user a context. Pure stack traces without any further explanation are not helpful.
        •  
       
      When the logger instance is created, the bot id must be given as parameter anyway. The function call defines the log level, see below.
      "},{"location":"dev/bot-development/#log-levels","title":"Log Levels","text":"
         
      • debug: Debugging information includes retrieved and sent messages, detailed status information. Can include   sensitive information like passwords and amount can be huge.
        •  
      • info: Logs include loaded databases, fetched reports or waiting messages.
        •  
      • warning: Unexpected, but handled behavior.
        •  
      • error: Errors and Exceptions.
        •  
      • critical Program is failing.
        •  
      "},{"location":"dev/bot-development/#what-to-log","title":"What to Log","text":"
         
      • Try to keep a balance between obscuring the source code file with hundreds of log messages and having too little log   messages.
        •  
      • In general, a bot MUST report error conditions.
        •  
      "},{"location":"dev/bot-development/#how-to-log","title":"How to Log","text":"
      The Bot class creates a logger with that should be used by bots. Other components won't log anyway currently. Examples:
       
      self.logger.info('Bot start processing.')\nself.logger.error('Pipeline failed.')\nself.logger.exception('Pipeline failed.')\n
       
      The exception method automatically appends an exception traceback. The logger instance writes by default to the file /opt/intelmq/var/log/[bot-id].log and to stderr.
      "},{"location":"dev/bot-development/#string-formatting-in-logs","title":"String formatting in Logs","text":"
      Parameters for string formatting are better passed as argument to the log function, see https://docs.python.org/3/library/logging.html#logging.Logger.debug In case of formatting problems, the error messages will be better. For example:
       
      self.logger.debug('Connecting to %r.', host)\n
      "},{"location":"dev/bot-development/#error-handling","title":"Error handling","text":"
      The bot class itself has error handling implemented. The bot itself is allowed to throw exceptions and intended to fail! The bot should fail in case of malicious messages, and in case of unavailable but necessary resources. The bot class handles the exception and will restart until the maximum number of tries is reached and fail then. Additionally, the message in question is dumped to the file /opt/intelmq/var/log/[bot-id].dump and removed from the queue.
      "},{"location":"dev/bot-development/#initialization","title":"Initialization","text":"
      Maybe it is necessary so setup a Cache instance or load a file into memory. Use the init function for this purpose:
       
      class ExampleParserBot(Bot):\n    def init(self):\n        try:\n            self.database = pyasn.pyasn(self.database)\n        except IOError:\n            self.logger.error(\"pyasn data file does not exist or could not be \"\n                              \"accessed in '%s'.\" % self.database)\n            self.logger.error(\"Read 'bots/experts/asn_lookup/README.md' and \"\n                              \"follow the procedure.\")\n            self.stop()\n
      "},{"location":"dev/bot-development/#custom-configuration-checks","title":"Custom configuration checks","text":"
      Every bot can define a static method check(parameters) which will be called by intelmqctl check. For example the check function of the ASNLookupExpert:
       
      @staticmethod\ndef check(parameters):\n    if not os.path.exists(parameters.get('database', '')):\n        return [[\"error\", \"File given as parameter 'database' does not exist.\"]]\n    try:\n        pyasn.pyasn(parameters['database'])\n    except Exception as exc:\n        return [[\"error\", \"Error reading database: %r.\" % exc]]\n
      "},{"location":"dev/bot-development/#running","title":"Running","text":"
      You can always start any bot directly from command line by calling the executable. The executable will be created during installation a directory for binaries. After adding new bots to the code, install IntelMQ to get the files created. Don't forget to give an bot id as first argument. Also, running bots with other users than intelmq will raise permission errors.
       
      $ sudo -i intelmq\n$ intelmqctl run file-output  # if configured\n$ intelmq.bots.outputs.file.output file-output\n
       
      You will get all logging outputs directly on stderr as well as in the log file.
      "},{"location":"dev/bot-development/#examples","title":"Examples","text":"
         
      • Check Expert Bots
        •  
      • Check Parser Bots
        •  
      "},{"location":"dev/bot-development/#parsers","title":"Parsers","text":"
      Parsers can use a different, specialized Bot-class. It allows to work on individual elements of a report, splitting the functionality of the parser into multiple functions:
       
         
      • process: getting and sending data, handling of failures etc.
        •  
      • parse: Parses the report and splits it into single elements (e.g. lines). Can be overridden.
        •  
      • parse_line: Parses elements, returns an Event. Can be overridden.
        •  
      • recover_line: In case of failures and for the field raw, this function recovers a fully functional report containing only one element. Can be overridden.
        •  
       
      For common cases, like CSV, existing function can be used, reducing the amount of code to implement. In the best case, only parse_line needs to be coded, as only this part interprets the data.
       
      You can have a look at the implementation intelmq/lib/bot.py or at examples, e.g. the DummyBot in intelmq/tests/lib/test_parser_bot.py. This is a stub for creating a new Parser, showing the parameters and possible code:
       
      class MyParserBot(ParserBot):\n\n    def parse(self, report):\n        \"\"\"A generator yielding the single elements of the data.\n\n        Comments, headers etc. can be processed here. Data needed by\n        `self.parse_line` can be saved in `self.tempdata` (list).\n\n        Default parser yields stripped lines.\n        Override for your use or use an existing parser, e.g.:\n            parse = ParserBot.parse_csv\n        \"\"\"\n        for line in utils.base64_decode(report.get(\"raw\")).splitlines():\n            yield line.strip()\n\n    def parse_line(self, line, report):\n        \"\"\"A generator which can yield one or more messages contained in line.\n\n        Report has the full message, thus you can access some metadata.\n        Override for your use.\n        \"\"\"\n        raise NotImplementedError\n\n    def process(self):\n        self.tempdata = []  # temporary data for parse, parse_line and recover_line\n        self.__failed = []\n        report = self.receive_message()\n\n        for line in self.parse(report):\n            if not line:\n                continue\n            try:\n                # filter out None\n                events = list(filter(bool, self.parse_line(line, report)))\n            except Exception as exc:\n                self.logger.exception('Failed to parse line.')\n                self.__failed.append((exc, line))\n            else:\n                self.send_message(*events)\n\n        for exc, line in self.__failed:\n            self._dump_message(exc, self.recover_line(line))\n\n        self.acknowledge_message()\n\n    def recover_line(self, line):\n        \"\"\"Reverse of parse for single lines.\n\n        Recovers a fully functional report with only the problematic line.\n        \"\"\"\n        return 'n'.join(self.tempdata + [line])\n\n\nBOT = MyParserBot\n
      "},{"location":"dev/bot-development/#parse_line","title":"parse_line","text":"
      One line can lead to multiple events, thus parse_line can't just return one Event. Thus, this function is a generator, which allows to easily return multiple values. Use yield event for valid Events and return in case of a void result (not parsable line, invalid data etc.).
      "},{"location":"dev/bot-development/#tests","title":"Tests","text":"
      In order to do automated tests on the bot, it is necessary to write tests including sample data. Have a look at some existing tests:
       
         
      • The DummyParserBot in intelmq/tests/lib/test_parser_bot.py. This test has the example data (report and event) inside the file, defined as dictionary.
        •  
      • The parser for malwaregroup at intelmq/tests/bots/parsers/malwaregroup/test_parser_*.py. The latter loads a sample HTML file from the same directory, which is the raw report.
        •  
      • The test for ASNLookupExpertBot has two event tests, one is an expected fail (IPv6).
        •  
       
      Ideally an example contains not only the ideal case which should succeed, but also a case where should fail instead. (TODO: Implement assertEventNotEqual or assertEventNotcontainsSubset or similar) Most existing bots are only tested with one message. For newly written test it is appreciable to have tests including more then one message, e.g. a parser fed with an report consisting of multiple events.
       
      import unittest\n\nimport intelmq.lib.test as test\nfrom intelmq.bots.parsers.exampleparser.parser import ExampleParserBot  # adjust bot class name and module\n\n\nclass TestExampleParserBot(test.BotTestCase, unittest.TestCase):  # adjust test class name\n    \"\"\"A TestCase for ExampleParserBot.\"\"\"\n\n    @classmethod\n    def set_bot(cls):\n        cls.bot_reference = ExampleParserBot  # adjust bot class name\n        cls.default_input_message = EXAMPLE_EVENT  # adjust source of the example event (dict), by default an empty event or report (depending on bot type)\n\n    # This is an example how to test the log output\n    def test_log_test_line(self):\n        \"\"\"Test if bot does log example message.\"\"\"\n        self.run_bot()\n        self.assertRegexpMatches(self.loglines_buffer,\n                                 \"INFO - Lorem ipsum dolor sit amet\")\n\n    def test_event(self):\n        \"\"\"Test if correct Event has been produced.\"\"\"\n        self.run_bot()\n        self.assertMessageEqual(0, EXAMPLE_REPORT)\n\n\nif __name__ == '__main__':  # pragma: no cover\n    unittest.main()\n
       
      When calling the file directly, only the tests in this file for the bot will be expected. Some default tests are always executed (via the test.BotTestCase class), such as pipeline and message checks, logging, bot naming or empty message handling.
       
      See the testing section about how to run the tests.
      "},{"location":"dev/bot-development/#cache","title":"Cache","text":"
      Bots can use a Redis database as cache instance. Use the intelmq.lib.utils.Cache class to set this up and/or look at existing bots, like the cymru_whois expert how the cache can be used. Bots must set a TTL for all keys that are cached to avoid caches growing endless over time. Bots must use the Redis databases >= 10, but not those already used by other bots. Look at find intelmq -type f -name '*.py' -exec grep -r 'redis_cache_db' {} + to see which databases are already used.
       
      The databases < 10 are reserved for the IntelMQ core:
       
         
      • 2: pipeline
        •  
      • 3: statistics
        •  
      • 4: tests
        •  
      "},{"location":"dev/bot-development/#documentation","title":"Documentation","text":"
      Please document your added/modified code.
       
      For doc strings, we are using the sphinx-napoleon-google-type-annotation.
       
      Additionally, Python's type hints/annotations are used, see PEP484.
      "},{"location":"dev/bot-development/#testing-pre-releases","title":"Testing Pre-releases","text":""},{"location":"dev/bot-development/#installation","title":"Installation","text":"
      The installation procedures need to be adapted only a little bit.
       
      For native packages, you can find the unstable packages of the next version here: Installation Unstable Native Packages . The unstable only has a limited set of packages, so enabling the stable repository can be activated in parallel. For CentOS 8 unstable, the stable repository is required.
       
      For the installation with pip, use the --pre parameter as shown here following command:
       
      pip3 install --pre intelmq\n
       
      All other steps are not different. Please report any issues you find in our Issue Tracker.
      "},{"location":"dev/data-format/","title":"Data Format","text":""},{"location":"dev/data-format/#data-format","title":"Data Format","text":"
      Data passed between bots is called a Message. There are two types of Messages: Report and Event. Report is produced by collector bots and consists of collected raw data (CSV, JSON, HTML, etc) and feed metadata. It is passed to a parser bot which parses Report into a single or multiple Events. Expert bots and output bots handle only Events.
       
      All Messages (Reports and Events) are Python dictionaries (or JSONs). The key names and according types are defined by the IntelMQ Data Format.
       
      The source code for the Data Format can be found in the Python module intelmq.lib.harmonization and the configuration is present inside the harmonization.conf file. (The term Harmonization is used for historical reasons.)
      "},{"location":"dev/data-format/#rules-for-keys","title":"Rules for keys","text":"
      The keys are grouped together in sub-fields, e.g. source.ip or source.geolocation.latitude.
       
      Only the lower-case alphabet, numbers and the underscore are allowed. Further, the field name must not begin with a number. Thus, keys must match ^[a-z_][a-z_0-9]+(\\.[a-z_0-9]+)*$. These rules also apply for the otherwise unregulated extra. namespace.
      "},{"location":"dev/data-format/#data-types","title":"Data Types","text":"
      This document describes the IntelMQ data types used for individual events with a description of each allowed field.
      "},{"location":"dev/data-format/#asn","title":"ASN","text":"
      ASN type. Derived from Integer with forbidden values.
       
      Only valid are: 0 < ASN <= 4294967295
       
      See https://en.wikipedia.org/wiki/Autonomous_system_(Internet)
       
      The first and last ASNs of the original 16-bit integers, namely 0 and 65,535, and the last ASN of the 32-bit numbers, namely 4,294,967,295 are reserved and should not be used by operators.
      "},{"location":"dev/data-format/#accuracy","title":"Accuracy","text":"
      Accuracy type. A Float between 0 and 100.
      "},{"location":"dev/data-format/#base64","title":"Base64","text":"
      Base64 type. Always gives unicode strings.
       
      Sanitation encodes to base64 and accepts binary and unicode strings.
      "},{"location":"dev/data-format/#boolean","title":"Boolean","text":"
      Boolean type. Without sanitation only python bool is accepted.
       
      Sanitation accepts string 'true' and 'false' and integers 0 and 1.
      "},{"location":"dev/data-format/#classificationtaxonomy","title":"ClassificationTaxonomy","text":"
      classification.taxonomy type.
       
      The mapping follows Reference Security Incident Taxonomy Working Group \u2013 RSIT WG: https://github.com/enisaeu/Reference-Security-Incident-Taxonomy-Task-Force/
       
      These old values are automatically mapped to the new ones:
       
         
      • 'abusive content' -> 'abusive-content'
        •  
      • 'information gathering' -> 'information-gathering'
        •  
      • 'intrusion attempts' -> 'intrusion-attempts'
        •  
      • 'malicious code' -> 'malicious-code'
        •  
       
      Allowed values are:
       
         
      • abusive-content
        •  
      • availability
        •  
      • fraud
        •  
      • information-content-security
        •  
      • information-gathering
        •  
      • intrusion-attempts
        •  
      • intrusions
        •  
      • malicious-code
        •  
      • other
        •  
      • test
        •  
      • vulnerable
        •  
      "},{"location":"dev/data-format/#classificationtype","title":"ClassificationType","text":"
      classification.type type.
       
      The mapping extends Reference Security Incident Taxonomy Working Group \u2013 RSIT WG:
       
      https://github.com/enisaeu/Reference-Security-Incident-Taxonomy-Task-Force/
       
      These old values are automatically mapped to the new ones:
       
         
      • 'botnet drone' -> 'infected-system'
        •  
      • 'ids alert' -> 'ids-alert'
        •  
      • 'c&c' -> 'c2-server'
        •  
      • 'c2server' -> 'c2-server'
        •  
      • 'infected system' -> 'infected-system'
        •  
      • 'malware configuration' -> 'malware-configuration'
        •  
      • 'Unauthorised-information-access' -> 'unauthorised-information-access'
        •  
      • 'leak' -> 'data-leak'
        •  
      • 'vulnerable client' -> 'vulnerable-system'
        •  
      • 'vulnerable service' -> 'vulnerable-system'
        •  
      • 'ransomware' -> 'infected-system'
        •  
      • 'unknown' -> 'undetermined'
        •  
       
      These values changed their taxonomy: 'malware': In terms of the taxonomy 'malicious-code' they can be either 'infected-system' or 'malware-distribution' but in terms of malware actually, it is now taxonomy 'other'
       
      Allowed values are:
       
         
      • application-compromise
        •  
      • blacklist
        •  
      • brute-force
        •  
      • burglary
        •  
      • c2-server
        •  
      • copyright
        •  
      • data-leak
        •  
      • data-loss
        •  
      • ddos
        •  
      • ddos-amplifier
        •  
      • dga-domain
        •  
      • dos
        •  
      • exploit
        •  
      • harmful-speech
        •  
      • ids-alert
        •  
      • infected-system
        •  
      • information-disclosure
        •  
      • malware
        •  
      • malware-configuration
        •  
      • malware-distribution
        •  
      • masquerade
        •  
      • misconfiguration
        •  
      • other
        •  
      • outage
        •  
      • phishing
        •  
      • potentially-unwanted-accessible
        •  
      • privileged-account-compromise
        •  
      • proxy
        •  
      • sabotage
        •  
      • scanner
        •  
      • sniffing
        •  
      • social-engineering
        •  
      • spam
        •  
      • system-compromise
        •  
      • test
        •  
      • tor
        •  
      • unauthorised-information-access
        •  
      • unauthorised-information-modification
        •  
      • unauthorized-use-of-resources
        •  
      • undetermined
        •  
      • unprivileged-account-compromise
        •  
      • violence
        •  
      • vulnerable-system
        •  
      • weak-crypto
        •  
      "},{"location":"dev/data-format/#datetime","title":"DateTime","text":"
      Date and time type for timestamps.
       
      Valid values are timestamps with time zone and in the format '%Y-%m-%dT%H:%M:%S+00:00'. Invalid are missing times and missing timezone information (UTC). Microseconds are also allowed.
       
      Sanitation normalizes the timezone to UTC, which is the only allowed timezone.
       
      The following additional conversions are available with the convert function:
       
         
      • timestamp
        •  
      • windows_nt: From Windows NT / AD / LDAP
        •  
      • epoch_millis: From Milliseconds since Epoch
        •  
      • from_format: From a given format, eg. 'from_format|%H %M %S %m %d %Y %Z'
        •  
      • from_format_midnight: Date from a given format and assume midnight, e.g. 'from_format_midnight|%d-%m-%Y'
        •  
      • utc_isoformat: Parse date generated by datetime.isoformat()
        •  
      • fuzzy (or None): Use dateutils' fuzzy parser, default if no specific parser is given
        •  
      "},{"location":"dev/data-format/#fqdn","title":"FQDN","text":"
      Fully qualified domain name type.
       
      All valid lowercase domains are accepted, no IP addresses or URLs. Trailing dot is not allowed.
       
      To prevent values like '10.0.0.1:8080' (#1235), we check for the non-existence of ':'.
      "},{"location":"dev/data-format/#float","title":"Float","text":"
      Float type. Without sanitation only python float/integer/long is accepted. Boolean is explicitly denied.
       
      Sanitation accepts strings and everything float() accepts.
      "},{"location":"dev/data-format/#ipaddress","title":"IPAddress","text":"
      Type for IP addresses, all families. Uses the ipaddress module.
       
      Sanitation accepts integers, strings and objects of ipaddress.IPv4Address and ipaddress.IPv6Address.
       
      Valid values are only strings. 0.0.0.0 is explicitly not allowed.
      "},{"location":"dev/data-format/#ipnetwork","title":"IPNetwork","text":"
      Type for IP networks, all families. Uses the ipaddress module.
       
      Sanitation accepts strings and objects of ipaddress.IPv4Network and ipaddress.IPv6Network. If host bits in strings are set, they will be ignored (e.g 127.0.0.1/32).
       
      Valid values are only strings.
      "},{"location":"dev/data-format/#integer","title":"Integer","text":"
      Integer type. Without sanitation only python integer/long is accepted. Bool is explicitly denied.
       
      Sanitation accepts strings and everything int() accepts.
      "},{"location":"dev/data-format/#json","title":"JSON","text":"
      JSON type.
       
      Sanitation accepts any valid JSON objects.
       
      Valid values are only unicode strings with JSON objects.
      "},{"location":"dev/data-format/#jsondict","title":"JSONDict","text":"
      JSONDict type.
       
      Sanitation accepts pythons dictionaries and JSON strings.
       
      Valid values are only unicode strings with JSON dictionaries.
      "},{"location":"dev/data-format/#lowercasestring","title":"LowercaseString","text":"
      Like string, but only allows lower case characters.
       
      Sanitation lowers all characters.
      "},{"location":"dev/data-format/#registry","title":"Registry","text":"
      Registry type. Derived from UppercaseString.
       
      Only valid values: AFRINIC, APNIC, ARIN, LACNIC, RIPE. RIPE-NCC and RIPENCC are normalized to RIPE.
      "},{"location":"dev/data-format/#string","title":"String","text":"
      Any non-empty string without leading or trailing whitespace.
      "},{"location":"dev/data-format/#tlp","title":"TLP","text":"
      TLP level type. Derived from UppercaseString.
       
      Only valid values: WHITE, GREEN, AMBER, RED.
       
      Accepted for sanitation are different cases and the prefix 'tlp:'.
      "},{"location":"dev/data-format/#url","title":"URL","text":"
      URI type. Local and remote.
       
      Sanitation converts hxxp and hxxps to http and https. For local URIs (file) a missing host is replaced by localhost.
       
      Valid values must have the host (network location part).
      "},{"location":"dev/data-format/#uppercasestring","title":"UppercaseString","text":"
      Like string, but only allows upper case characters.
       
      Sanitation uppers all characters.
      "},{"location":"dev/documentation/","title":"Documentation","text":""},{"location":"dev/documentation/#documentation","title":"Documentation","text":"
      The documentation is automatically published to https://docs.intelmq.org at every push to the develop branch of the repository.
       
      To build the documentation you need additional packages:
       
      pip3 install .[development]\n
       
      Then use the Makefile to build the documentation using mkdocs:
       
      make docs\n
       
      Some parts of the documentation are automatically generated using dedicated scripts. You can find them in the Makefile.
      "},{"location":"dev/environment/","title":"Environment","text":""},{"location":"dev/environment/#development-environment","title":"Development Environment","text":""},{"location":"dev/environment/#directories","title":"Directories","text":"
      For development purposes, you need two directories:
       
         
      • directory with the local source code repository
        •  
      • root directory of the IntelMQ installation
        •  
       
      The default root directory of the IntelMQ installation is /opt/intelmq. This directory is used for configurations (/opt/intelmq/etc), local states (/opt/intelmq/var/lib) and logs (/opt/intelmq/var/log). If you want to change it, please set the INTELMQ_ROOT_DIR environment variable with a desired location.
       
      For repository directory, you can use any path that is accessible by users you use to run IntelMQ. For globally installed IntelMQ, the directory has to be readable by other unprivileged users (e.g. home directories on Fedora can't be read by other users by default).
       
      To keep commands in the guide universal, we will use environmental variables for repository and installation paths. You can set them with following commands:
       
      # Adjust paths if you want to use non-standard directories\nexport INTELMQ_REPO=/opt/dev_intelmq\nexport INTELMQ_ROOT_DIR=/opt/intelmq\n
       
      Note
       
      If using non-default installation directory, remember to keep the root directory variable set for every run of IntelMQ commands. If you don't, then the default location /opt/intelmq will be used.
      "},{"location":"dev/environment/#installation","title":"Installation","text":"
      Developers can create a fork repository of IntelMQ in order to commit the new code to this repository and then be able to do pull requests to the main repository. Otherwise you can just use the 'certtools' as username below.
       
      The following instructions will use pip3 -e, which gives you a so called editable installation. No code is copied in the libraries directories, there's just a link to your code. However, configuration files still required to be moved to /opt/intelmq as the instructions show.
       
      The traditional way to work with IntelMQ is to install it globally and have a separated user for running it. If you wish to separate your machine Python's libraries, e.g. for development purposes, you could alternatively use a Python virtual environment and your local user to run IntelMQ. Please use your preferred way from instructions below.
      "},{"location":"dev/environment/#using-globally-installed-intelmq","title":"Using globally installed IntelMQ","text":"
      sudo -s\n\ngit clone https://github.com/<your username>/intelmq.git $INTELMQ_REPO\ncd $INTELMQ_REPO\n\npip3 install -e .\n\nuseradd -d $INTELMQ_ROOT_DIR -U -s /bin/bash intelmq\n\nintelmqsetup\n
      "},{"location":"dev/environment/#using-virtual-environment","title":"Using virtual environment","text":"
      git clone https://github.com/<your username>/intelmq.git $INTELMQ_REPO\ncd $INTELMQ_REPO\n\npython -m venv .venv\nsource .venv/bin/activate\n\npip install -e .\n\n# If you use a non-local directory as INTELMQ_ROOT_DIR, use following\n# command to create it and change the ownership.\nsudo install -g `whoami` -o `whoami` -d $INTELMQ_ROOT_DIR\n# For local directory, just create it with mkdir:\nmkdir $INTELMQ_ROOT_DIR\n\nintelmqsetup --skip-ownership\n
       
      Note
       
      Please do not forget that configuration files, log files will be available on $INTELMQ_ROOT_DIR. However, if your development is somehow related to any shipped configuration file, you need to apply the changes in your repository $INTELMQ_REPO/intelmq/etc/.
      "},{"location":"dev/environment/#additional-services","title":"Additional services","text":"
      Some features require additional services, like message queue or database. The commonly used services are gained for development purposes in the Docker Compose file in contrib/development-tools/docker-compose-common-services.yaml in the repository. You can use them to run services on your machine in a docker containers, or decide to configure them in an another way. To run them using Docker Compose, use following command from the main repository directory:
       
      # For older Docker versions, you may need to use `docker-compose` command\ndocker compose -f contrib/development-tools/docker-compose-common-services.yaml up -d\n
       
      This will start in the background containers with Redis, RabbitMQ, PostgreSQL and MongoDB.
      "},{"location":"dev/environment/#how-to-develop","title":"How to develop","text":"
      After you successfully setup your IntelMQ development environment, you can perform any development on any .py file on $INTELMQ_REPO. After you change, you can use the normal procedure to run the bots:
       
      su - intelmq # Use for global installation\nsource .venv/bin/activate # Use for virtual environment installation\n\nintelmqctl start spamhaus-drop-collector\n\ntail -f $INTELMQ_ROOT_DIR/var/log/spamhaus-drop-collector.log\n
       
      You can also add new bots, creating the new .py file on the proper directory inside cd $INTELMQ_REPO/intelmq. However, your IntelMQ installation with pip3 needs to be updated. Please check the following section.
      "},{"location":"dev/environment/#update","title":"Update","text":"
      In case you developed a new bot, you need to update your current development installation. In order to do that, please follow this procedure:
       
         
      1. Make sure that you have your new bot in the right place.
        2.  
      2.  
        Update pip metadata and new executables: 
        sudo -s # Use for global installation\nsource .venv/bin/activate # Use for virtual environment installation\n\ncd /opt/dev_intelmq\npip3 install -e .\n
         
        3.  
      3.  
        If you're using the global installation, an additional step of changing permissions and ownership is necessary: 
        find $INTELMQ_ROOT_DIR/ -type d -exec chmod 0770 {} \\+\nfind $INTELMQ_ROOT_DIR/ -type f -exec chmod 0660 {} \\+\nchown -R intelmq.intelmq $INTELMQ_ROOT_DIR\n## if you use the intelmq manager (adapt the webservers' group if needed):\nchown intelmq.www-data $INTELMQ_ROOT_DIR/etc/*.conf\n
         
        4.  
       
      Now you can test run your new bot following this procedure:
       
      su - intelmq              # Use for global installation\nsource .venv/bin/activate # Use for virtual environment installation\n\nintelmqctl start <bot_id>\n
      "},{"location":"dev/extensions-packages/","title":"Extensions Packages","text":""},{"location":"dev/extensions-packages/#creating-extensions-packages","title":"Creating extensions packages","text":"
      IntelMQ supports adding additional bots using your own independent packages. You can use this to add a new integration that is special to you, or cannot be integrated into the main IntelMQ repository for some reason.
      "},{"location":"dev/extensions-packages/#building-an-extension-package","title":"Building an extension package","text":"
      A simple example of the package can be found in contrib/example-extension-package. To make your custom bots work with IntelMQ, you need to ensure that
       
         
      • your bot's module exposes a BOT object of the class inherited from intelmq.lib.bot.Bot    or its subclasses,
        •  
      • your package registers an entry point    in the console_scripts group with a name starting with intelmq.bots. followed by    the name of the group (collectors, experts, outputs, parsers), and then your original name.    The entry point must point to the BOT.run method,
        •  
      • the module in which the bot resides must be importable by IntelMQ (e.g. installed in the same    virtualenv, if you use them).
        •  
       
      Apart from these requirements, your package can use any of the usual package features. We strongly recommend following the same principles and main guidelines as the official bots. This will ensure the same experience when using official and additional bots.
      "},{"location":"dev/extensions-packages/#naming-convention","title":"Naming convention","text":"
      Building your own extensions gives you a lot of freedom, but it's important to know that if your bot's entry point uses the same name as another bot, it may not be possible to use it, or to determine which one is being used. For this reason, we recommend that you start the name of your bot with an with an organization identifier and then the bot name.
       
      For example, if I create a collector bot for feed source Special and run it on behalf of the organization Awesome, the suggested entry point might be intelmq.bots.collectors.awesome.special. Note that the structure of your package doesn't matter, as long as it can be imported properly.
       
      For example, I could create a package called awesome-bots with the following file structure
       
         awesome_bots\n   \u251c\u2500\u2500 pyproject.toml\n   \u2514\u2500\u2500 awesome_bots\n        \u251c\u2500\u2500 __init__.py\n        \u2514\u2500\u2500 special.py\n
       
      The pyproject.toml file would then have the following section:
       
         [project.scripts]\n   intelmq.bots.collectors.awesome.special = \"awesome_bots.special:BOT.run\"\n
       
      Once you have installed your package, you can run intelmqctl list bots to check if your bot was properly registered.
      "},{"location":"dev/guidelines/","title":"Guidelines","text":""},{"location":"dev/guidelines/#development-guidelines","title":"Development Guidelines","text":""},{"location":"dev/guidelines/#coding-rules","title":"Coding-Rules","text":"
      Most important: KEEP IT SIMPLE! This can not be over-estimated. Feature creep can destroy any good software project. But if new folks can not understand what you wrote in 10-15 minutes, it is not good. It's not about the performance, etc. It's about readability.
       
      In general, we follow PEP8. We recommend reading it before committing code.
       
      There are some exceptions: sometimes it does not make sense to check for every PEP8 error (such as whitespace indentation when you want to make a dict=() assignment look pretty. Therefore, we do have some exceptions defined in the setup.cfg file.
       
      We support Python 3 only.
      "},{"location":"dev/guidelines/#unicode","title":"Unicode","text":"
         
      • Each internal object in IntelMQ (Event, Report, etc) that has strings, their strings MUST be in UTF-8 Unicode format.
        •  
      • Any data received from external sources MUST be transformed into UTF-8 Unicode format before add it to IntelMQ   objects.
        •  
      "},{"location":"dev/guidelines/#back-end-independence-and-compatibility","title":"Back-end independence and Compatibility","text":"
      Any component of the IntelMQ MUST be independent of the message queue technology (Redis, RabbitMQ, etc...).
      "},{"location":"dev/guidelines/#license-header","title":"License Header","text":"
      Please add a license and copyright header to your bots. There is a Github action that tests for reuse compliance of your code files.
      "},{"location":"dev/guidelines/#intelmq-data-format-rules","title":"IntelMQ Data Format Rules","text":"
      Any component of IntelMQ MUST respect the IntelMQ Data Format.
      "},{"location":"dev/guidelines/#code-submission-rules","title":"Code Submission Rules","text":""},{"location":"dev/guidelines/#releases-repositories-and-branches","title":"Releases, Repositories and Branches","text":"
         
      • The main repository is in github.com/certtools/intelmq.
        •  
      • We use semantic versioning.
        •  
      • If you contribute something, please fork the repository, create a separate branch and use this for pull requests, see section below.
        •  
      • There are a couple of forks which might be regularly merged into the main repository. They are independent and can have incompatible changes and can deviate from the upstream repository.
        •  
      "},{"location":"dev/guidelines/#branching-model","title":"Branching model","text":"
         
      • \"master\" is the stable branch. It hold the latest stable release. Non-developers should only work on this branch. The recommended log level is WARNING. Code is only added by merges from the maintenance branches.
        •  
      • \"maintenance/a.b.x\" branches accumulate (cherry-picked) patches for a maintenance release (a.b.x). Recommended for   experienced users which deploy intelmq themselves. No new features will be added to these branches.
        •  
      • \"develop\" is the development branch for the next stable release   (a.x). New features must go there. Developers may want to work on this branch. This branch also holds all patches from   maintenance releases if applicable. The recommended log level is DEBUG.
        •  
      • Separate branches to develop features or bug fixes may be used by any contributor.
        •  
      "},{"location":"dev/guidelines/#how-to-contribute","title":"How to Contribute","text":"
         
      • Make separate pull requests / branches on GitHub for changes. This allows us to discuss things via GitHub.
        •  
      • We prefer one Pull Request per feature or change. If you have a bunch of small fixes, please don't create one PR per fix :)
        •  
      • Only very small and changes (docs, ...) might be committed directly to development branches without Pull Request by the core-team.
        •  
      • Keep the balance between atomic commits and keeping the amount of commits per PR small. You can use interactive   rebasing to squash multiple small commits into one (rebase -i [base-branch]). Only do rebasing if the code you are rebasing is yet not used by others or is already merged - because then others may need to run into conflicts.
        •  
      • Make sure your PR is merge able in the develop branch and all tests are successful.
        •  
      • If possible sign your commits with GPG.
        •  
      "},{"location":"dev/guidelines/#workflow","title":"Workflow","text":"
      We assume here, that origin is your own fork. We first add the upstream repository:
       
       git remote add upstream https://github.com/certtools/intelmq.git\n
       
      Syncing develop:
       
       git checkout develop\n git pull upstream develop\n git push origin develop\n
       
      You can do the same with the branches master and maintenance.
       
      Create a separate feature-branch to work on, sync develop with upstream. Create working branch from develop:
       
       git checkout develop\n git checkout -b bugfix\n# your work\n git commit\n
       
      Or, for bugfixes create a separate bugfix-branch to work on, sync maintenance with upstream. Create working branch from maintenance:
       
      git checkout maintenance\ngit checkout -b new-feature\n# your work\ngit commit\n
       
      Getting upstream's changes for master or any other branch:
       
      git checkout develop\ngit pull upstream develop\ngit push origin develop\n
       
      There are 2 possibilities to get upstream's commits into your branch. Rebasing and Merging. Using rebasing, your history is rewritten, putting your changes on top of all other commits. You can use this if your changes are not published yet (or only in your fork).
       
      git checkout bugfix\ngit rebase develop\n
       
      Using the -i flag for rebase enables interactive rebasing. You can then remove, reorder and squash commits, rewrite commit messages, beginning with the given branch, e.g. develop.
       
      Or using merging. This doesn't break the history. It's considered more , but also pollutes the history with merge commits.
       
      git checkout bugfix\ngit merge develop\n
       
      You can then create a PR with your branch bugfix to our upstream repository, using GitHub's web interface.
      "},{"location":"dev/guidelines/#commit-messages","title":"Commit Messages","text":"
      If it fixes an existing issue, please use GitHub syntax, e.g.: fixes certtools/intelmq#<IssueID>
      "},{"location":"dev/guidelines/#prepare-for-discussion-in-github","title":"Prepare for Discussion in GitHub","text":"
      If we don't discuss it, it's probably not tested.
      "},{"location":"dev/guidelines/#license-and-author-files","title":"License and Author files","text":"
      License and Authors files can be found at the root of repository.
       
         
      • License file MUST NOT be modified except by the explicit written permission by CNCS/CERT.PT or CERT.at
        •  
      • Credit to the authors file must be always retained. When a new contributor (person and/or organization) improves in   some way the repository content (code or documentation), he or she might add his name to the list of contributors.
        •  
       
      License and authors must be only listed in an external file but not inside the code files.
      "},{"location":"dev/intro/","title":"Intro","text":""},{"location":"dev/intro/#intro","title":"Intro","text":"
      This guide is for developers of IntelMQ. It explains the code architecture, coding guidelines as well as ways you can contribute code or documentation. If you have not done so, please read the User Guide and the Administrator Guide first. Once you feel comfortable running IntelMQ with open source bots and you feel adventurous enough to contribute to the project, this guide is for you. It does not matter if you are an experienced Python programmer or just a beginner. There is a lot of examples to help you out.
       
      However, before we go into the details, it is important to observe and internalize some overall project goals.
      "},{"location":"dev/intro/#goals","title":"Goals","text":"
      It is important, that all developers agree and stick to these meta-guidelines. IntelMQ tries to:
       
         
      • Be well tested. For developers this means, we expect you to write unit tests for bots. Every time.
        •  
      • Reduce the complexity of system administration.
        •  
      • Reduce the complexity of writing new bots for new data feeds.
        •  
      • Make your code easily and pleasantly readable.
        •  
      • Reduce the probability of events lost in all process with persistence functionality (even system crash).
        •  
      • Strictly adhere to the existing format for keys and values in events.
        •  
      • Always use JSON format for all messages internally.
        •  
      • Help and support the interconnection between IntelMQ and existing tools like AbuseHelper, CIF, etc. or new tools (in other words: we will not accept data-silos!).
        •  
      • Provide an easy way to store data into log collectors such as ElasticSearch or Splunk.
        •  
      • Provide an easy way to create your own black-lists.
        •  
      • Provide easy to understand interfaces with other systems via HTTP RESTFUL API.
        •  
       
      The main take away point from the list above is: things MUST stay intuitive and easy. How do you ultimately test if things are still easy? Let them new programmers test-drive your features and if it is not understandable in 15 minutes, go back to the drawing board.
       
      Similarly, if code does not get accepted upstream by the main developers, it is usually only because of the ease-of-use argument. Do not give up, go back to the drawing board, and re-submit again.
      "},{"location":"dev/intro/#mailing-list","title":"Mailing list","text":"
      There is a separate mailing list for developers to discuss development topics: The IntelMQ-DevArchive is public as well.
      "},{"location":"dev/intro/#github","title":"GitHub","text":"
      The ideal way to propose changes and additions to IntelMQ is to open a Pull Request on GitHub.
      "},{"location":"dev/library/","title":"Use as Library","text":""},{"location":"dev/library/#running-intelmq-as-library","title":"Running IntelMQ as Library","text":""},{"location":"dev/library/#introduction","title":"Introduction","text":"
      The feature is specified in IEP007.
      "},{"location":"dev/library/#quickstart","title":"Quickstart","text":"
      First, import the Python module and a helper. More about the BotLibSettings later.
       
      from intelmq.lib.bot import BotLibSettings\nfrom intelmq.bots.experts.domain_suffix.expert import DomainSuffixExpertBot\n
       
      Then we need to initialize the bot's instance. We pass two parameters:
       
         
      • bot_id: The id of the bot
        •  
      • settings: A Python dictionary of runtime configuration parameters, see runtime-configuration. The bot first loads the runtime configuration file if it exists. Then we update them with the BotLibSettings which are some accumulated settings disabling the logging to files and configure the pipeline so that we can send and receive messages directly to/from the bot. Last by not least, the actual bot parameters, taking the highest priority.
        •  
       
      domain_suffix = DomainSuffixExpertBot('domain-suffix',  # bot id\nsettings=BotLibSettings | {\n'field': 'fqdn',\n'suffix_file': '/usr/share/publicsuffix/public_suffix_list.dat'}\n
       
      As the bot is not fully initialized, we can process messages now. Inserting a message as dictionary:
       
      queues = domain_suffix.process_message({'source.fqdn': 'www.example.com'})\n
       
      The return value is a dictionary of queues, e.g. the output queue and the error queue. More details below.
       
      The methods accepts multiple messages as positional argument:
       
      domain_suffix.process_message(\n    {'source.fqdn': 'www.example.com'},\n    {'source.fqdn': 'www.example.net'}\n)\ndomain_suffix.process_message(*[\n    {'source.fqdn': 'www.example.com'},\n    {'source.fqdn': 'www.example.net'}\n])\n
       
      Select the output queue (as defined in destination_queues), first message, access the field source.domain_suffix: 
      >>> output['output'][0]['source.domain_suffix']\n'com'\n
      "},{"location":"dev/library/#configuration","title":"Configuration","text":"
      Configuration files are not required to run IntelMQ as library. Contrary to IntelMQ normal behavior, if the files runtime.yaml and harmonization.conf do not exist, IntelMQ won't raise any errors. For the harmonization configuration, internal defaults are loaded.
      "},{"location":"dev/release/","title":"Release","text":""},{"location":"dev/release/#release-procedure","title":"Release procedure","text":"
      General assumption: You are working on branch maintenance, the next version is a bug fix release. For feature releases it is slightly different.
      "},{"location":"dev/release/#check-before","title":"Check before","text":"
         
      • Make sure the current state is really final ;) You can test most of     the steps described here locally before doing it real.
        •  
      • Check the upgrade functions in intelmq/lib/upgrades.py.
        •  
      • Close the milestone on GitHub and move any open issues to the next     one.
        •  
      • docs/admin/installation/linux-packages.md: Update supported operating systems.
        •  
      "},{"location":"dev/release/#documentation","title":"Documentation","text":"
      These apply to all projects:
       
         
      • CHANGELOG.MD and NEWS.MD: Update the latest header, fix the order, remove empty sections and (re)group the entries if necessary.
        •  
      • debian/changelog: Insert a new section for the new version with the tool dch or update the version of the existing last item if yet unreleased. Don't forget the revision after the version number!
        •  
      "},{"location":"dev/release/#intelmq","title":"IntelMQ","text":"
         
      • intelmq/version.py: Update the version.
        •  
      "},{"location":"dev/release/#intelmq-api","title":"IntelMQ API","text":"
         
      • intelmq_api/version.py: Update the version.
        •  
      "},{"location":"dev/release/#intelmq-manager","title":"IntelMQ Manager","text":"
         
      • intelmq_manager/version.py: Update the version.
        •  
      • intelmq_manager/static/js/about.js: Update the version.
        •  
      "},{"location":"dev/release/#commit-push-review-and-merge","title":"Commit, push, review and merge","text":"
      Commit your changes in a separate branch, the final commit message should start with REL:. Push and create a pull request to the develop branch. Someone else should review the changes. Eventually fix them, make sure the REL: is the last commit, you can also push that one at last, after the reviews.
       
      Why a separate branch? Because if problems show up, you can still force-push to that one, keeping the release commit the latest one.
      "},{"location":"dev/release/#tag-and-release","title":"Tag and release","text":"
      Tag the commit with git tag -s version HEAD, merge it into develop, push the branches and the tag. The tag is just a.b.c, not prefixed with v (that was necessary only with SVN a long time ago...).
       
      Go to https://github.com/certtools/intelmq/tags and enter the release notes (from the CHANGELOG) for the new tag, then it's considered a release by GitHub.
      "},{"location":"dev/release/#tarballs-and-pypi","title":"Tarballs and PyPI","text":"
         
      • Build the source and binary (wheel) distribution:
        •  
       
      rm -r build/\npython3 setup.py sdist bdist_wheel\n
       
         
      • Upload the files including signatures to PyPI with e.g. twine: twine upload -u __token__ -p $APITOKEN dist/intelmq... (or set the API Token in .pypirc).
        •  
      "},{"location":"dev/release/#documentation_1","title":"Documentation","text":"
      Since using mkdocs (see https://docs.intelmq.org) nothing needs to be done anymore.
      "},{"location":"dev/release/#packages","title":"Packages","text":"
      We are currently using the public Open Build Service instance of openSUSE: http://build.opensuse.org/project/show/home:sebix:intelmq
       
      First, test all the steps first with the unstable-repository and check that at least installations succeed.
       
         
      • Create the tarballs with the script create-archives.sh.
        •  
      • Update the dsc and spec files for new filenames and versions.
        •  
      • Update the .changes file
        •  
      • Build locally for all distributions.
        •  
      • Commit.
        •  
      "},{"location":"dev/release/#docker-image","title":"Docker Image","text":"
      Releasing a new Docker image is very easy.
       
         
      • Clone IntelMQ Docker Repository with git clone https://github.com/certat/intelmq-docker.git --recursive as this repository contains submodules
        •  
      • If the intelmq-docker repository is not updated yet, use git pull --recurse-submodules to pull the latest changes from their respective repository.
        •  
      • Run ./build.sh, check your console if the build was successful.
        •  
      • Run ./test.sh - It will run nosetests3 with the exotic flag. All     errors/warnings will be displayed.
        •  
      • Change the build_version in publish.sh to the new version you     want to release.
        •  
      • Change the namespace variable in publish.sh.
        •  
      • If no error/warning was shown, you can release with ./publish.sh.
        •  
      • Update the DockerHub ReadMe and add the latest version.
        •  
      • Commit and push the updates to the intelmq-docker repository
        •  
      "},{"location":"dev/release/#announcements","title":"Announcements","text":"
      Announce the new version at the mailinglists intelmq-users, intelmq-dev. For bigger releases, probably also at IHAP, Twitter, etc. Ask your favorite social media consultant.
      "},{"location":"dev/release/#prepare-new-version","title":"Prepare new version","text":"
      Increase the version in intelmq/version.py and declare it as alpha version. Add the new version in intelmq/lib/upgrades.py. Add a new entry in debian/changelog with dch -v [version] -c debian/changelog.
       
      Add new entries to CHANGELOG.md and NEWS.md.
      "},{"location":"dev/release/#intelmq_1","title":"IntelMQ","text":"
      For CHANGELOG.md:
       
      ### Configuration\n\n### Core\n\n### Development\n\n### Data Format\n\n### Bots\n#### Collectors\n\n#### Parsers\n\n#### Experts\n\n#### Outputs\n\n### Documentation\n\n### Packaging\n\n### Tests\n\n### Tools\n\n### Contrib\n\n### Known issues\n
       
      And for NEWS.md:
       
      ### Requirements\n\n### Tools\n\n### Data Format\n\n### Configuration\n\n### Libraries\n\n### Postgres databases\n
      "},{"location":"dev/release/#intelmq-api_1","title":"IntelMQ API","text":"
      An empty section of CHANGELOG.rst.
      "},{"location":"dev/release/#intelmq-manager_1","title":"IntelMQ Manager","text":"
      For CHANGELOG.md:
       
      ### Pages\n\n#### Landing page\n\n#### Configuration\n\n#### Management\n\n#### Monitor\n\n#### Check\n\n### Documentation\n\n### Third-party libraries\n\n### Packaging\n\n### Known issues\n
       
      And an empty section in the NEWS.md file.
      "},{"location":"dev/structure/","title":"Structure","text":""},{"location":"dev/structure/#system-overview","title":"System Overview","text":"
      In the intelmq/lib/ directory you can find some libraries:
       
         
      • Bots: Defines base structure for bots and handling of startup, stop,     messages etc.
        •  
      • Cache: For some expert bots it does make sense to cache external     lookup results. Redis is used here.
        •  
      • Harmonization: For defined types, checks and sanitation methods are     implemented.
        •  
      • Message: Defines Events and Reports classes, uses harmonization to     check validity of keys and values according to config.
        •  
      • Pipeline: Writes messages to message queues. Implemented for     productions use is only Redis, AMQP is beta.
        •  
      • Test: Base class for bot tests with predefined test and assert     methods.
        •  
      • Utils: Utility functions used by system components.
        •  
      "},{"location":"dev/structure/#code-architecture","title":"Code Architecture","text":""},{"location":"dev/testing/","title":"Testing","text":""},{"location":"dev/testing/#testing","title":"Testing","text":""},{"location":"dev/testing/#additional-test-requirements","title":"Additional test requirements","text":"
      Libraries required for tests are listed in the setup.py file. You can install them with pip:
       
      pip3 install -e .[development]\n
       
      or the package management of your operating system.
      "},{"location":"dev/testing/#run-the-tests","title":"Run the tests","text":"
      All changes have to be tested and new contributions should be accompanied by according unit tests. Please do not run the tests as root just like any other IntelMQ component for security reasons. Any other unprivileged user is possible.
       
      You can run the tests by changing to the directory with IntelMQ repository and running either unittest or pytest. For virtual environment installation, please activate it and omit the sudo -u from examples below:
       
      cd $INTELMQ_REPO\nsudo -u intelmq python3 -m unittest {discover|filename}  # or\nsudo -u intelmq pytest [filename]\nsudo -u intelmq python3 setup.py test  # uses a build environment (no external dependencies)\n
       
      Some bots need local databases to succeed. If you only want to test one explicit test file, give the file path as argument.
       
      There are multiple GitHub Action Workflows setup for automatic testing, which are triggered on pull requests. You can also easily activate them for your forks.
      "},{"location":"dev/testing/#environment-variables","title":"Environment variables","text":"
      There are a bunch of environment variables which switch on/off some tests:
       Environment\u00a0Variable\u00a0Name Description INTELMQ_TEST_DATABASES databases such as postgres, elasticsearch, mongodb are not tested by default. Set this environment variable to 1 to test those bots. These tests need preparation, e.g. running databases with users and certain passwords etc. Have a look at the .github/workflows/unittests.yml and the corresponding .github/workflows/scripts/setup-full.sh in IntelMQ's repository for steps to set databases up. INTELMQ_SKIP_INTERNET tests requiring internet connection will be skipped if this is set to 1. INTELMQ_SKIP_REDIS redis-related tests are ran by default, set this to 1 to skip those. INTELMQ_TEST_EXOTIC some bots and tests require libraries which may not be available, those are skipped by default. To run them, set this to 1. INTELMQ_TEST_REDIS_PASSWORD Set this value to the password for the local redis database if needed. INTELMQ_LOOKYLOO_TEST Set this value to run the lookyloo tests. Public lookyloo instance will be used as default. INTELMQ_TEST_INSTALLATION Set this value to run tests which require a local IntelMQ installation, such as for testing the command lines tools relying on configuration files, dump files etc. 
      For example, to run all tests you can use:
       
      INTELMQ_TEST_DATABASES=1 INTELMQ_TEST_EXOTIC=1 pytest intelmq/tests/\n
      "},{"location":"dev/testing/#configuration-test-files","title":"Configuration test files","text":"
      The tests use the configuration files in your working directory, not those installed in /opt/intelmq/etc/ or /etc/. You can run the tests for a locally changed intelmq without affecting an installation or requiring root to run them.
      "},{"location":"tutorials/intelmq-manager/","title":"Using IntelMQ Manager","text":""},{"location":"tutorials/intelmq-manager/#tutorial-on-using-intelmq-manager","title":"Tutorial on using IntelMQ Manager","text":"
      Bug
       
      This section of the documentation is currently incomplete and will be updated later.
      "},{"location":"unsorted/botnet-concept/","title":"Botnet concept","text":""},{"location":"unsorted/botnet-concept/#botnet-concept","title":"Botnet Concept","text":"
      The \\\"botnet\\\" represents all currently configured bots which are explicitly enabled. It is, in essence, the graph of the bots which are connected together via their input source queues and destination queues.
       
      To get an overview which bots are running, use intelmqctl status or use the IntelMQ Manager. Set \"enabled\": true in the runtime configuration to add a bot to the botnet. By default, bots will be configured as \"enabled\": true. See bots{.interpreted-text role=\"doc\"} for more details on configuration.
       
      Disabled bots can still be started explicitly using intelmqctl start <bot_id>, but will remain in the state disabled if stopped (and not be implicitly enabled by the start command). They are not started by intelmqctl start in analogy to the behavior of widely used initialization systems.
      "},{"location":"unsorted/intelmq-3.0-architecture/","title":"Intelmq 3.0 architecture","text":""},{"location":"unsorted/intelmq-3.0-architecture/#idea-list-and-architecture-of-intelmq-30","title":"Idea list and architecture of IntelMQ 3.0","text":"
      Authors: Aaron Kaplan kaplan@cert.at, Sebastian Wagner wagner@cert.at
      "},{"location":"unsorted/intelmq-3.0-architecture/#use-cases","title":"Use-cases","text":"
      XXX fill in a complete list of use cases XXX
      "},{"location":"unsorted/intelmq-3.0-architecture/#certs","title":"CERTs","text":"
      No direct access to networks in constituency.
      "},{"location":"unsorted/intelmq-3.0-architecture/#data-collection","title":"Data collection","text":""},{"location":"unsorted/intelmq-3.0-architecture/#distribution-of-information","title":"Distribution of information","text":""},{"location":"unsorted/intelmq-3.0-architecture/#national-cert","title":"National CERT","text":"
      Work is based heavily on Geolocation
      "},{"location":"unsorted/intelmq-3.0-architecture/#sector-cert","title":"Sector CERT","text":"
      Work is based on known constituents, sector information, lists of IP address ranges and domains, company & organisation names.
      "},{"location":"unsorted/intelmq-3.0-architecture/#socs-and-nocs","title":"SOCs and NOCs","text":"
      Goal is the protection of internal known networks only. Direct access to the networks.
       
      Involves collecting information from internal infrastructure, matching IoCs to internal infrastructure, using IoCs for active protection.
      "},{"location":"unsorted/intelmq-3.0-architecture/#data-science-and-research","title":"Data science and research","text":""},{"location":"unsorted/intelmq-3.0-architecture/#users","title":"Users","text":"
      XXX fill in a complete list of use cases XXX
      "},{"location":"unsorted/intelmq-3.0-architecture/#restful-api","title":"RESTful API","text":"
      For automation purposes, we will need a typical RESTful API to manage, control, monitor the IntelMQ \"botnet\" and read and set configs. See #1424
      "},{"location":"unsorted/intelmq-3.0-architecture/#ux","title":"UX","text":""},{"location":"unsorted/intelmq-3.0-architecture/#devops-sysadmin-perspective","title":"Devops/ Sysadmin perspective","text":""},{"location":"unsorted/intelmq-3.0-architecture/#docker","title":"Docker","text":"
      Task: create a setup where each bot MAY run in a docker container
       
      Background: It might make sense to  be able to run each bot in a docker container since it fits with a lot of new paradigms in orchestration. With a proper template, each bot running in a docker container could send its logs to some central logger (for example splunk or similar) and the sysadmin/devops teams which are already using these systems for monitoring alerts can properly fit the IntelMQ logs and alerts to their regular daily routine. Docker also allows the sysadmin/devops folks to centrally manage the system.
       
      Think about: how do we integrate the pipeline graph?
       
      Category: this feature should be OPTIONAL.
      "},{"location":"unsorted/intelmq-3.0-architecture/#tutorials-and-vms-dockers","title":"Tutorials and VMs / dockers","text":"
      Task: create tutorials with VMs/docker images.
       
      Background: We are missing good tutorials (\"playbooks\") on how to run certain workflows via IntelMQ. Ideally, we would offer ready-made VMs/docker images where people who want to try out IntelMQ (and consequently adapt the setup to their own needs). This also helps teachers/presenters who want to demo IntelMQ.
       
      Specifically we would like to have:   * how to process shadowserver feeds   * how to process shodan data   * how to process n6 data
       
      Think about: shadowserver already created some training material. Build on this.
       
      Category: OPTIONAL component, but highly needed.
      "},{"location":"unsorted/intelmq-3.0-architecture/#architecture","title":"Architecture","text":""},{"location":"unsorted/intelmq-3.0-architecture/#message-queue","title":"Message queue","text":"
      Task: Create a Kafka MQ backend: add Kafka as a replaceable MQ for IntelMQ 3.0
       
      Background: IntelMQ 2.0 supports AMQP (RabbitMQ) next to redis as a message queue. Many organisations use Kafka internally. Support connecting to their other work flows.
       
      Think about: Using Apache Pulsar
       
      Category: SHOULD
      "},{"location":"unsorted/intelmq-3.0-architecture/#notification-settings","title":"Notification settings","text":"
      Task: Keep notification settings per event: Where to (destination mail/host address), how (protocol, authentication (SSL client certificate), etc), how often/time information (intervals etc.)
       
      Background: CERTs (and potentially other groups of users) need to specify where the events should be sent to, how often etc. Currently only destination email addresses can be saved (source.abuse_contact), which is not enough for most use-cases. There exist some custom solutions (e.g. notify boolean at cert.at (to be changed), extra.processing dictionary at BSI), but no least common denominator.
       
      See also https://github.com/certtools/intelmq/issues/758
       
      Category: this feature should be OPTIONAL but is NEEDED by several users.
      "},{"location":"unsorted/intelmq-3.0-architecture/#configuration-parameter-handling-in-bots-and-a-bots-unified-documentation","title":"Configuration parameter handling in Bots and a bot's unified documentation","text":"
      Task: Handle bots' configuration parameters by the core, providing type sanitation, checks, default values and documentation.
       
      Background: Currently every bot needs to handle these issues itself, but many of these checks could be done centrally in a generic way. At upgrades, new configuration might get introduced and the bots need to provide defaults values although they are available in BOTS. Error handling on parameters must be done for every bot on itself. Documentation is not available to the Bots, not available in BOTS and the Manager. There are 3 places for parameters where the available information is spread: BOTS, Bots.md and the bots' code.
      "},{"location":"unsorted/intelmq-3.0-architecture/#automatic-monitoring-management-handling-full-load-situations","title":"Automatic Monitoring & Management: Handling full load situations","text":"
      Task: Create a solution to prevent system over-loading (only for Redis).
       
      Background: If too much data is ingested, collected or enriched, the system can easily run out of memory. This quickly causes major operation troubles and data loss, needing manual intervention.
       
      See also: https://github.com/certtools/intelmq/issues/709
      "},{"location":"unsorted/intelmq-3.0-architecture/#making-intelmq-plug-able-and-getting-rid-of-bots","title":"Making intelmq plug-able and getting rid of BOTS","text":"
      Task: Allow installation of IntelMQ bots, meaning the deprecation of the centralized BOTS file and a generated documentation.
       
      Background: Adapting IntelMQ to specific needs also means the development of specific bots which might not part of the public repository. Adding them to an existing IntelMQ installation is currently only possible by cloning the repository and adding the code there, not by just providing/installing the required code (because of BOTS and central documentation).
       
      See also https://github.com/certtools/intelmq/issues/972
      "},{"location":"unsorted/intelmq-3.0-architecture/#exposing-a-plug-in-or-hooking-api","title":"Exposing a plug-in or hooking API","text":"
      Task: Provide an hooking API for the core classes.
       
      Background: Adapting IntelMQ to specific can require adaptions in the Core classes' code. Instead of making the changes/extensions in the core itself, we can provide a hook system allowing to call (or replace?) functions at specific steps. For example custom monitoring.
      "},{"location":"unsorted/intelmq-3.0-architecture/#grouping-of-events","title":"Grouping of events","text":"
      Task: Provide possibilities to assign an event to a group of events.
       
      Background: Several IoCs part of one MISP Event. Grouping of similar events to one group for outputs (e.g. one CSV file per Network).
       
      See also: https://github.com/certtools/intelmq/issues/751
      "},{"location":"unsorted/intelmq-3.0-architecture/#data-format-multiple-values","title":"Data Format: Multiple values","text":"
      Task: Allow multiple values for (some) fields in the data format.
       
      Background: In some cases one value per field is not enough, for example for Domain -> IP address lookups. Other formats like IDEA and n6 support this.
       
      See also: https://github.com/certtools/intelmq/issues/543 https://github.com/certtools/intelmq/issues/373
      "},{"location":"unsorted/intelmqctl-more/","title":"Intelmqctl more","text":""},{"location":"unsorted/intelmqctl-more/#command-line-interface-intelmqctl","title":"Command-line interface: intelmqctl","text":"
      Syntax see intelmqctl -h
       
         
      • Starting a bot: intelmqctl start bot-id
        •  
      • Stopping a bot: intelmqctl stop bot-id
        •  
      • Reloading a bot: intelmqctl reload bot-id
        •  
      • Restarting a bot: intelmqctl restart bot-id
        •  
      • Get status of a bot: intelmqctl status bot-id
        •  
      • Run a bot directly for debugging purpose and temporarily leverage the logging level to DEBUG: intelmqctl run bot-id
        •  
      • Get a pdb (or ipdb if installed) live console.   intelmqctl run bot-id console
        •  
      • See the message that waits in the input queue.   intelmqctl run bot-id message get
        •  
      • See additional help for further explanation.   intelmqctl run bot-id --help
        •  
      • Starting the botnet (all bots): intelmqctl start
        •  
      • Starting a group of bots: intelmqctl start --group experts
        •  
      • Get a list of all configured bots: intelmqctl list bots
        •  
      • Get a list of all queues: intelmqctl list queues If -q is given, only queues with more than one item are listed.
        •  
      • Get a list of all queues and status of the bots:   intelmqctl list queues-and-status
        •  
      • Clear a queue: intelmqctl clear queue-id
        •  
      • Get logs of a bot: intelmqctl log bot-id number-of-lines log-level   Reads the last lines from bot log. Log level should be one of DEBUG, INFO, ERROR or CRITICAL. Default is INFO. Number   of lines defaults to 10, -1 gives all. Result can be longer due to our logging format!
        •  
      • Upgrade from a previous version: intelmqctl upgrade-config Make a backup of your configuration first, also including   bot's configuration files.
        •  
      "},{"location":"unsorted/intelmqctl-more/#reloading","title":"Reloading","text":"
      Whilst restart is a mere stop & start, performing intelmqctl reload <bot_id> will not stop the bot, permitting it to keep the state: the same common behavior as for ( Linux) daemons. It will initialize again (including reading all configuration again) after the current action is finished. Also, the rate limit/sleep is continued (with the new time) and not interrupted like with the restart command. So if you have a collector with a rate limit of 24 h, the reload does not trigger a new fetching of the source at the time of the reload, but just 24 h after the last run -- with the new configuration. Which state the bots are keeping depends on the bots of course.
      "},{"location":"unsorted/intelmqctl-more/#forcing-reset-pipeline-and-cache-be-careful","title":"Forcing reset pipeline and cache (be careful)","text":"
      If you are using the default broker (Redis), in some test situations you may need to quickly clear all pipelines and caches. Use the following procedure:
       
      redis-cli FLUSHDB\nredis-cli FLUSHALL\n
      "},{"location":"unsorted/intelmqctl-more/#management","title":"Management","text":"
      IntelMQ has a modular structure consisting of bots. There are four types of bots:
       
         
      • collector bots retrieve data from internal or external sources, the output are *   reports* consisting of many individual data sets / log lines.
        •  
      • parser bots parse the (report) data by splitting it into individual events (log lines) and giving them a defined   structure, see also /dev/data-format for the list of fields an event may be split up into.
        •  
      • expert bots enrich the existing events by e.g. lookup up information such as DNS reverse records, geographic   location information (country code) or abuse contacts for an IP address or domain name.
        •  
      • output bots write events to files, databases, (REST)-APIs or any other data sink that you might want to write to.
        •  
       
      Each bot has one source queue (except collectors) and can have multiple destination queues (except outputs). But multiple bots can write to the same pipeline (queue), resulting in multiple inputs for the next bot.
       
      Every bot runs in a separate process. A bot is identifiable by a bot id.
       
      Currently only one instance (i.e. with the same bot id) of a bot can run at the same time. Concepts for multiprocessing are being discussed, see this issue: Multiprocessing per queue is not supported #186 <186>. Currently you can run multiple processes of the same bot ( with different bot ids) in parallel.
       
      Example: multiple gethostbyname bots (with different bot ids) may run in parallel, with the same input queue and sending to the same output queue. Note that the bot providing the input queue must have the load_balance option set to true.
      "},{"location":"user/abuse-contacts/","title":"Abuse Contacts","text":""},{"location":"user/abuse-contacts/#abuse-contact-look-ups","title":"Abuse-contact look-ups","text":"
      The right decision whom to contact about a specific incident is vital to get the incident resolved as quick as possible. Different types of events may required different abuse-contact to be selected. For example, issues about a device, e.g. a vulnerability in the operating system or an application, is better sent to the hoster which can inform the server administrator. For website-related issues, like defacements or phishing, the domain owner (maintaining the content of the website) could be the better and more direct contact. Additionally, different CERT's have different approaches and different contact databases. Multiple information sources have different information, and some sources are more accurate than others. IntelMQ can query multiple sources of abuse-contacts and combine them. Internal databases, like a Constituency Portal provide high-quality and first-hand contact information. The RIPE document Sources of Abuse Contact Information for Abuse Handlers contains a good summary of the complex of themes.
      "},{"location":"user/abuse-contacts/#sources-for-abuse-contacts","title":"Sources for abuse-contacts","text":"
      All these bots add the queried contacts to the IntelMQ events in the field source.abuse_contact if not state otherwise in the documentation.
      "},{"location":"user/abuse-contacts/#sources-for-domain-based-abuse-contacts","title":"Sources for domain-based abuse-contacts","text":"
      These bots are suitable for domain-based abuse-contact look-ups.
       
         
      • intelmq.bots.experts.rdap.expert expert queries private and public RDAP servers for source.fqdn and add the contact information to the event as source.abuse_contact.
        •  
      • intelmq.bots.experts.trusted_introducer_lookup.expert expert queries a locally cached Trusted Introducer team directory for the TLD or domain (first match) of source.fqdn.
        •  
      "},{"location":"user/abuse-contacts/#sources-for-ip-address-based-abuse-contacts","title":"Sources for IP address-based abuse-contacts","text":"
      These bots are suitable for IP address and ASN based abuse-contact look-ups.
       
         
      • intelmq.bots.experts.abusix.expert expert queries the online Abusix service.
        •  
      • intelmq.bots.experts.do_portal.expert expert queries an instance of the do-portal software (deprecated).
        •  
      • intelmq.bots.experts.tuency.expert expert queries an instance of the tuency Constituency Portal for the IP address. The Portal also takes into account any notification rules, which are saved   additionally in the event.
        •  
      • intelmq.bots.experts.ripe.expert expert queries the online RIPE database for IP-Address and AS contacts.
        •  
      • intelmq.bots.experts.trusted_introducer_lookup.expert expert queries a locally   cached Trusted Introducer team directory   for the Autonomous system source.asn.
        •  
      "},{"location":"user/abuse-contacts/#generic-sources-for-abuse-contacts","title":"Generic sources for abuse-contacts","text":"
         
      • intelmq.bots.experts.generic_db_lookup.expert expert for local data sources, like   database tables mapping ASNs to abuse-contact or Country Codes to abuse-contact.
        •  
      • intelmq.bots.experts.uwhoisd.expert expert for fetching whois-data, not extracting   abuse-contact information
        •  
      "},{"location":"user/abuse-contacts/#helpful-other-bots-for-pre-processing","title":"Helpful other bots for pre-processing","text":"
         
      • intelmq.bots.experts.asn_lookup.expert queries locally cached database to lookup ASN.
        •  
      • intelmq.bots.experts.cymru_whois.expert to lookup ASN, Geolocation, and BGP prefix   for *.ip.
        •  
      • intelmq.bots.experts.domain_suffix.expert to lookup the public suffix of the domain   in *.fqdn.
        •  
      • intelmq.bots.experts.format_field.expert
        •  
      • intelmq.bots.experts.gethostbyname.expert resolve *.ip from *.fqdn.
        •  
      • intelmq.bots.experts.maxmind_geoip.expert to lookup Geolocation information for *.ip   .
        •  
      • intelmq.bots.experts.reverse_dns.expert to resolve *.reverse_dns from *.ip.
        •  
      • intelmq.bots.experts.ripe.expert to lookup *.asn and Geolocation information   for *.ip.
        •  
      • intelmq.bots.experts.tor_nodes.expert for filtering out TOR nodes.
        •  
      • intelmq.bots.experts.url2fqdn.expert to extract *.fqdn/*.ip from *.url.
        •  
      "},{"location":"user/abuse-contacts/#combining-the-lookup-approaches","title":"Combining the lookup approaches","text":"
      In order to get the best contact, it may be necessary to combine multiple abuse-contact sources. IntelMQ's modularity provides methods to arrange and configure the bots as needed. Among others, the following bots can help in getting the best result:
       
         
      • intelmq.bots.experts.filter.expert Your lookup process may be different for different types of data. E.g. website-related issues may be better addressed at the domain owner and device-related issues may be better addressed to the hosting provider.
        •  
      • intelmq.bots.experts.modify.expert Allows you to set values based on filter and also format values based on the value of other fields.
        •  
      • intelmq.bots.experts.sieve.expert Very powerful expert which allows filtering, routing (to different subsequent bots) based on if-expressions . It support set-operations (field value is in list) as well as sub-network operations for IP address networks in CIDR notation for the expression-part. You can as well set the abuse-contact directly.
        •  
      "},{"location":"user/api/","title":"API","text":""},{"location":"user/api/#using-intelmq-api","title":"Using IntelMQ API","text":"
      Bug
       
      This section of the documentation is currently incomplete and will be added later.
      "},{"location":"user/api/#usage-from-programs","title":"Usage from programs","text":"
      The IntelMQ API can also be used from programs, not just browsers. To do so, first send a POST-Request with JSON-formatted data to http://localhost/intelmq/v1/api/login/
       
      {\n    \"username\": \"$your_username\",\n    \"password\": \"$your_password\"\n}\n
       
      With valid credentials, the JSON-formatted response contains the login_token. This token can be used like an API key in the Authorization header for the next API calls:
       
      Authorization: $login_token\n
       
      Here is a full example using curl:
       
         
      1.  
        Authentication step:    
        curl --location --request POST \"http://localhost/intelmq/v1/api/login/\" \\\n     --header \"Content-Type: application/x-www-form-urlencoded\" \\\n     --data-urlencode \"username=$username\"\\\n     --data-urlencode \"password=$password\"\n
         
        {\"login_token\":\"68b329da9893e34099c7d8ad5cb9c940\",\"username\":\"$username\"}\n
         
        2.  
      2.  
        Using the login token to fetch data:    
        curl --location \"http://localhost/intelmq/v1/api/version\" \\\n     --header \"Authorization: 68b329da9893e34099c7d8ad5cb9c940\"\n
         
        {\"intelmq\":\"3.0.0rc1\",\"intelmq-manager\":\"2.3.1\"}\n
         
        3.  
       
      The same approach also works for Ansible, as you can see here:
       
         
      1. https://github.com/schacht-certat/intelmq-vagrant/blob/7082719609c0aafc9324942a8775cf2f8813703d/ansible/tasks/api/00_registerauth.yml#L1-L9
        2.  
      2. https://github.com/schacht-certat/intelmq-vagrant/blob/7082719609c0aafc9324942a8775cf2f8813703d/ansible/tasks/api/02_queuestatus.yml#L1-L5
        3.  
      "},{"location":"user/bots/","title":"Bots","text":""},{"location":"user/bots/#bots-inventory","title":"Bots Inventory","text":"
      This document contains complete reference of bots implemented by IntelMQ and how to configure them from the users perspective (meaning via IntelMQ Manager). Some of the bots are intended for general use and some of them are for processing particular data sources.
      "},{"location":"user/bots/#individual-bot-configuration","title":"Individual Bot Configuration","text":"
      Each bot has it's own configuration. The configuration consists of two types of parameters:
       
         
      •  
        Generic parameters that are common to all the bots and need to be set for each bot.
         
        •  
      •  
        Runtime parameters are needed by the bot itself during runtime. Some of these parameters can be inherited from the global configuration (which is applied to all the bots), but can be overridden in the individual bot configuration.
         
        •  
      "},{"location":"user/bots/#generic-parameters","title":"Generic Parameters","text":"
      These parameters must be set for each bot (at least the required ones).
      "},{"location":"user/bots/#id","title":"id","text":"
      (required, string) This must be a unique identifier. Commonly it looks something like this: abusech-feodo-tracker-collector. It is safer to avoid using spaces.
      "},{"location":"user/bots/#name","title":"name","text":"
      (required, string) Human readable name of the bot.
      "},{"location":"user/bots/#description","title":"description","text":"
      (required, string) The description of the bot.
      "},{"location":"user/bots/#module","title":"module","text":"
      (required, string) The executable (should be in PATH environment variable) which will be started.
      "},{"location":"user/bots/#group","title":"group","text":"
      (optional, string) The group of the bot. Can be Collector, Parser, Expert or Output. Only used for visualization by other tools.
      "},{"location":"user/bots/#enabled","title":"enabled","text":"
      (optional, boolean) Whether the bot will start when the whole botnet is started. You can still start a disabled bot explicitly. Defaults to true.
      "},{"location":"user/bots/#run_mode","title":"run_mode","text":"
      (optional, string) There are two run modes, continuous or scheduled. In the first case, the bot will be running forever until stopped or exits because of errors (depending on the configuration). In the latter case, the bot will stop after one successful run. This is especially useful when scheduling bots via cron or systemd. Check Configuration section for more details. Defaults to continuous.
      "},{"location":"user/bots/#http-parameters","title":"HTTP Parameters","text":"
      Common HTTP runtime parameters used in multiple bots.
      "},{"location":"user/bots/#http_timeout_sec","title":"http_timeout_sec","text":"
      (optional, float) A tuple of floats or only one float describing the timeout (seconds) of the HTTP connection. Can be a tuple of two floats (read and connect timeout) or just one float (applies for both timeouts). See also https://requests.readthedocs.io/en/master/user/advanced/#timeouts. Defaults to 30.
      "},{"location":"user/bots/#http_timeout_max_tries","title":"http_timeout_max_tries","text":"
      (optional, integer) An integer depicting how many times a connection is retried, when a timeout occurred. Defaults to 3.
      "},{"location":"user/bots/#http_username","title":"http_username","text":"
      (optional, string) Username for basic HTTP authentication.
      "},{"location":"user/bots/#http_password","title":"http_password","text":"
      (optional, string) Password for basic HTTP authentication.
      "},{"location":"user/bots/#http_proxy","title":"http_proxy","text":"
      (optional, string) Proxy to use for HTTP.
      "},{"location":"user/bots/#https_proxy","title":"https_proxy","text":"
      (optional, string) Proxy to use for HTTPS.
      "},{"location":"user/bots/#http_user_agent","title":"http_user_agent","text":"
      (optional, string) User-Agent to be used for HTTP requests.
      "},{"location":"user/bots/#http_verify_cert","title":"http_verify_cert","text":"
      (optional, boolean/string) Path to trusted CA bundle or directory, false to ignore verifying SSL certificates, or true to verify SSL certificates. Defaults to true.
      "},{"location":"user/bots/#ssl_client_certificate","title":"ssl_client_certificate","text":"
      (optional, string) Path to client certificate to use for TLS connections.
      "},{"location":"user/bots/#ssl_ca_certificate","title":"ssl_ca_certificate","text":"
      (optional, string) Path to trusted CA certificate. Only used by some bots.
      "},{"location":"user/bots/#cache-parameters","title":"Cache Parameters","text":"
      Common Redis cache runtime parameters used in multiple bots (mainly lookup experts).
      "},{"location":"user/bots/#redis_cache_host","title":"redis_cache_host","text":"
      (required, string) Hostname of the Redis database.
      "},{"location":"user/bots/#redis_cache_port","title":"redis_cache_port","text":"
      (required, string) Port of the Redis database.
      "},{"location":"user/bots/#redis_cache_db","title":"redis_cache_db","text":"
      (required, integer) Database number.
      "},{"location":"user/bots/#redis_cache_ttl","title":"redis_cache_ttl","text":"
      (required, integer) TTL used for caching.
      "},{"location":"user/bots/#redis_cache_password","title":"redis_cache_password","text":"
      (optional, string) Password for the Redis database.
      "},{"location":"user/bots/#collector-bots","title":"Collector Bots","text":"
      Multihreading is disabled for all Collectors, as this would lead to duplicated data.
      "},{"location":"user/bots/#feed-parameters","title":"Feed Parameters","text":"
      These runtime parameters must be set for each collector bot (at least the required ones).
      "},{"location":"user/bots/#name_1","title":"name","text":"
      (required, string) Name of the feed (feed.name).
      "},{"location":"user/bots/#accuracy","title":"accuracy","text":"
      (optional, float) Accuracy of the data from the feed (feed.accuracy).
      "},{"location":"user/bots/#code","title":"code","text":"
      (optional, string) Code for the feed (feed.code).
      "},{"location":"user/bots/#documentation","title":"documentation","text":"
      (optional, string) Link to documentation for the feed (feed.documentation).
      "},{"location":"user/bots/#provider","title":"provider","text":"
      (optional, string) Name of the provider of the feed (feed.provider).
      "},{"location":"user/bots/#rate_limit","title":"rate_limit","text":"
      (optional, integer) Time interval (in seconds) between fetching data if applicable. Defaults to 0.
      "},{"location":"user/bots/#alien-vault-otx","title":"Alien Vault OTX","text":"
      Collects report messages from Alien Vault OTX.
       
      Module: intelmq.bots.collectors.alienvault_otx.collector
       
      Requirements
       
      Install the library from GitHub, as there is no package in PyPi:
       
      pip3 install -r intelmq/bots/collectors/alienvault_otx/REQUIREMENTS.txt\n
       
      Parameters (also expects feed parameters):
       
      api_key
       
      (required, string) API Key
       
      modified_pulses_only
       
      (optional, boolean) Whether to get only modified pulses instead of all. Defaults to false.
       
      interval
       
      (optional, integer) When modified_pulses_only is set, define the time in hours (integer value) to get modified pulses since then. Defaults to 24 (hours).
      "},{"location":"user/bots/#amqp","title":"AMQP","text":"
      This bot collects data from (remote) AMQP servers, for both IntelMQ as well as external data. Currently only fetching from a queue is supported can be extended in the future. Messages will be acknowledge at AMQP after it is sent to the pipeline. Requires the pika library, minimum version 1.0.0.
       
      Module: intelmq.bots.collectors.amqp.collector_amqp
       
      Parameters (also expects feed parameters):
       
      connection_host
       
      (optional, string) Hostname of the AMQP server. Defaults to 127.0.0.1.
       
      connection_port
       
      (optional, integer) Port of the AMQP server. Defaults to 5672.
       
      connection_attempts
       
      (optional, integer) The number of connection attempts to the defined server. Defaults to 3.
       
      connection_heartbeat
       
      (optional, integer) Heartbeat to server (seconds). Defaults to 3600.
       
      connection_vhost
       
      (optional, string) Virtual host to connect, on an HTTP(S) connection would be . 
      expect_intelmq_message
       
      (optional, boolean) This parameter denotes whether the the data is from IntelMQ or not. If true, then the data can be any Report or Event and will be passed to the next bot as is. Otherwise a new Report is created with the raw data. Defaults to false.
       
      queue_name
       
      (optional, string) The name of the queue to fetch the data from.
       
      username
       
      (optional, string) Username for authentication to the AMQP server.
       
      password
       
      (optional, string) Password for authentication to the AMQP server.
       
      use_ssl
       
      (optional, boolean) Use of TLS for the connection. Make sure to also set the correct port. Defaults to false.
      "},{"location":"user/bots/#api","title":"API","text":"
      This bot collects data from HTTP or Socket REST API. The API is available at /intelmq/push when the HTTP interface is used. Requires the tornado library.
       
      Module: intelmq.bots.collectors.api.collector
       
      Parameters (also expects feed parameters):
       
      port
       
      (optional, integer) The local port at which the API is available. Defaults to 5000.
       
      use_socket
       
      (optional, boolean) If true, the socket will be opened at the location given with socket_path. Defaults to false.
       
      socket_path
       
      (optional, string) Location of the socket. Defaults to /tmp/imq_api_default_socket.
       
      socket_perms
       
      (optional, octal integer) Unix permissions to grant to the socket file. Default: 600
       
      socket_group
       
      (optional, string) Name of group to change group ownership of socket file to.
      "},{"location":"user/bots/#generic-url-fetcher","title":"Generic URL Fetcher","text":"
      This bot collects data from remote hosts using HTTP protocol. If the HTTP response' status code is not 2xx, this is treated as error. In Debug logging level, the request's and response's headers and body are logged for further inspection.
       
      Module: intelmq.bots.collectors.http.collector_http
       
      Parameters (also expects feed parameters and HTTP parameters):
       
      http_url
       
      (required, string) Location of the resource to download.
       
      http_url_formatting
       
      (optional, boolean/object) When true, {time[format]} will be replaced by the current time in local timezone formatted by the given format. E.g. if the URL is http://localhost/{time[%Y]}, then the resulting URL is http://localhost/2019 for the year 2019. ( Python's Format Specification Mini-Language is used for this.). You may use a JSON specifying time-delta parameters to shift the current time accordingly. For example use days: -1 for the yesterday's date; the URL http://localhost/{time[%Y-%m-%d]} will get translated to http://localhost/2018-12-31 for the 1st Jan of 2019. Defaults to false.
       
      extract_files
       
      (optional, boolean/array of strings) If true, the retrieved (compressed) file or archived will be uncompressed/unpacked and the files are extracted. If the parameter is a list of strings, only the files matching the filenames are extracted. Extraction handles gzipped files and both compressed and uncompressed tar-archives as well as zip archives. For extracted files, every extracted file is sent in it's own report. Every report has a field named extra.file_name with the file name in the archive the content was extracted from. Defaults to false.
       
      verify_pgp_signatures
       
      (optional, boolean) When true, signature file is downloaded and report file is checked. On error (missing signature, mismatch, ...), the error is logged and the report is not processed. Public key has to be imported in local keyring. This requires the python-gnupg library. Defaults to false.
       
      signature_url
       
      (optional, string) Location of the signature file for the downloaded content.
       
      signature_url_formatting
       
      (optional, boolean/object) Same as http_url_formatting. Defaults to false.
       
      gpg_keyring
       
      (optional, string) If specified, the string represents path to keyring file. Otherwise the PGP keyring file of the current intelmq user is used.
      "},{"location":"user/bots/#generic-url-stream-fetcher","title":"Generic URL Stream Fetcher","text":"
      Opens a streaming connection to the URL and collects the received lines.
       
      If the stream is interrupted, the connection will be aborted using the timeout parameter. No error will be logged if the number of consecutive connection fails does not reach the parameter error_max_retries. Instead of errors, an INFO message is logged. This is a measurement against too frequent ERROR logging messages. The consecutive connection fails are reset if a data line has been successfully transferred. If the consecutive connection fails reaches the parameter error_max_retries, an exception will be thrown and rate_limit applies, if not null.
       
      Module: intelmq.bots.collectors.http.collector_http_stream
       
      Parameters (also expects feed parameters and HTTP parameters):
       
      Uses the same parameters as Generic URL Fetcher. The parameter http_timeout_max_tries is of no use in this collector.
       
      strip_lines
       
      (optional, boolean) Whether the single lines should be stripped (removing whitespace from the beginning and the end of the line) or not. Defaults to true.
      "},{"location":"user/bots/#generic-mail-url-fetcher","title":"Generic Mail URL Fetcher","text":"
      Extracts URLs from e-mail messages and downloads the content from the URLs. It uses the imbox library.
       
      The resulting reports contain the following special fields:
       
         
      • feed.url: The URL the data was downloaded from.
        •  
      • extra.email_date: The content of the email's Date header.
        •  
      • extra.email_subject: The subject of the email.
        •  
      • extra.email_from: The email's from address.
        •  
      • extra.email_message_id: The email's message ID.
        •  
      • extra.file_name: The file name of the downloaded file (extracted from the HTTP Response Headers if possible).
        •  
       
      The fields can be used by parsers to identify the feed and are not automatically passed on to events.
       
      Chunking
       
      For line-based inputs the bot can split up large reports into smaller chunks. This is particularly important for setups that use Redis as a message queue which has a per-message size limitation of 512 MB. To configure chunking, set chunk_size to a value in bytes. chunk_replicate_header determines whether the header line should be repeated for each chunk that is passed on to a parser bot. Specifically, to configure a large file input to work around Redis size limitation set chunk_size to something like 384000000 (~384 MB).
       
      Module: intelmq.bots.collectors.mail.collector_mail_url
       
      Parameters (also expects feed parameters and HTTP parameters):
       
      mail_host
       
      (required, string) Hostname of the mail server.
       
      mail_port
       
      (optional, integer) IMAP server port: 143 without TLS, 993 with TLS. Defaults to 143.
       
      mail_user
       
      (required, string) Username of the email account.
       
      mail_password
       
      (required, string) Password associated with the user account.
       
      mail_ssl
       
      (optional, boolean) Whether the mail server uses TLS or not. Defaults to true.
       
      mail_starttls
       
      (optional, boolean) Whether the mail server uses STARTTLS or not. Defaults to false.
       
      folder
       
      (optional, string) Folder in which to look for e-mail messages. Defaults to INBOX.
       
      subject_regex
       
      (optional, string) Regular expression to look for in the e-mail subject.
       
      url_regex
       
      (optional, string) Regular expression of the feed URL to look for in the e-mail body.
       
      sent_from
       
      (optional, string) Filter messages by the sender.
       
      sent_to
       
      (optional, string) Filter messages by the recipient.
       
      ssl_ca_certificate
       
      (optional, string) Path to trusted CA certificate. Applies only to IMAP connections, not HTTP. If the provided certificate is not found, the IMAP connection will fail on handshake. Defaults to no certificate.
      "},{"location":"user/bots/#generic-mail-attachment-fetcher","title":"Generic Mail Attachment Fetcher","text":"
      This bot collects messages from mailboxes and downloads the attachments. It uses the imbox library.
       
      The resulting reports contains the following special fields:
       
         
      • extra.email_date: The content of the email's Date header
        •  
      • extra.email_subject: The subject of the email
        •  
      • extra.email_from: The email's from address
        •  
      • extra.email_message_id: The email's message ID
        •  
      • extra.file_name: The file name of the attachment or the file name in the attached archive if attachment is to   uncompress.
        •  
       
      The fields can be used by parsers to identify the feed and are not automatically passed on to events.
       
      Module: intelmq.bots.collectors.mail.collector_mail_attach
       
      Parameters (also expects feed parameters):
       
      mail_host
       
      (required, string) Hostname of the mail server.
       
      mail_port
       
      (optional, integer) IMAP server port: 143 without TLS, 993 with TLS. Default depends on SSL setting.
       
      mail_user
       
      (required, string) Username of the email account.
       
      mail_password
       
      (required, string) Password associated with the user account.
       
      mail_ssl
       
      (optional, boolean) Whether the mail server uses TLS or not. Defaults to true.
       
      mail_starttls
       
      (optional, boolean) Whether to use STARTTLS before authenticating to the server. Defaults to false.
       
      folder
       
      (optional, string) Folder in which to look for e-mail messages. Defaults to INBOX.
       
      subject_regex
       
      (optional, string) Regular expression to look for in the e-mail subject.
       
      attach_regex
       
      (optional, string) All attachments which match this regular expression will be processed. Defaults to csv.zip.
       
      extract_files
       
      (optional, boolean) Whether to extract compress files from the attachment. Defaults to true.
       
      sent_from
       
      (optional, string) Only process messages sent from this address. Defaults to null (any sender).
       
      sent_to
       
      (optional, string) Only process messages sent to this address. Defaults to null (any recipient).
       
      ssl_ca_certificate
       
      (optional, string) Path to trusted CA certificate. Applies only to IMAP connections, not HTTP. If the provided certificate is not found, the IMAP connection will fail on handshake. By default, no certificate is used.
      "},{"location":"user/bots/#generic-mail-body-fetcher","title":"Generic Mail Body Fetcher","text":"
      This bot collect messages from mailboxes, forwards the bodies as reports. Each non-empty body with the matching content type is sent as individual report.
       
      The resulting reports contains the following special fields:
       
         
      • extra.email_date: The content of the email's Date header
        •  
      • extra.email_subject: The subject of the email
        •  
      • extra.email_from: The email's from address
        •  
      • extra.email_message_id: The email's message ID
        •  
       
      Module: intelmq.bots.collectors.mail.collector_mail_body
       
      Parameters (also expects feed parameters):
       
      mail_host
       
      (required, string) Hostname of the mail server.
       
      mail_port
       
      (optional, integer) IMAP server port: 143 without TLS, 993 with TLS. Defaults to 143.
       
      mail_user
       
      (required, string) Username of the email account.
       
      mail_password
       
      (required, string) Password associated with the user account.
       
      mail_ssl
       
      (optional, boolean) Whether the mail server uses TLS or not. Defaults to true.
       
      folder
       
      (optional, string) Folder in which to look for e-mail messages. Defaults to INBOX.
       
      subject_regex
       
      (optional, string) Regular expression to look for in the e-mail subject.
       
      url_regex
       
      (optional, string) Regular expression of the feed URL to look for in the e-mail body.
       
      sent_from
       
      (optional, string) Filter messages by the sender.
       
      sent_to
       
      (optional, string) Filter messages by the recipient.
       
      ssl_ca_certificate
       
      (optional, string) Path to trusted CA certificate. Applies only to IMAP connections, not HTTP. If the provided certificate is not found, the IMAP connection will fail on handshake. Defaults to no certificate.
       
      content_types
       
      (optional, boolean/array of strings) Which bodies to use based on the content_type. Defaults to true (same as ['html', 'plain']) for all:
       
         
      • string with comma separated values, e.g. ['html', 'plain']
        •  
      • true, false, null: Same as default value - string, e.g. plain
        •  
      "},{"location":"user/bots/#github-api","title":"Github API","text":"
      Collects files matched by regular expression from GitHub repository via the GitHub API. Optionally with GitHub credentials, which are used as the Basic HTTP authentication.
       
      Workflow
       
      The optional authentication parameters provide a high limit of the GitHub API requests. With the git hub user authentication, the requests are rate limited to 5000 per hour, otherwise to 60 requests per hour.
       
      The collector recursively searches for regex-defined files in the provided repository. Additionally it adds extra file metadata defined by the extra_fields.
       
      The bot always sets the url, from which downloaded the file, as feed.url.
       
      Module: intelmq.bots.collectors.github_api.collector_github_contents_api
       
      Parameters (also expects feed parameters):
       
      personal_access_token
       
      (required, string) GitHub account personal access token GitHub documentation: Creating a personal access token
       
      repository
       
      (required, string) GitHub target repository (<USER>/<REPOSITORY>)
       
      regex
       
      (optional, string) Valid regular expression of target files within the repository. Defaults to .*.json.
       
      extra_fields
       
      (optional, array of strings) Comma-separated list of extra fields from GitHub contents API.
      "},{"location":"user/bots/#file","title":"File","text":"
      This bot is capable of reading files from the local file-system. This is handy for testing purposes, or when you need to react to spontaneous events. In combination with the Generic CSV parser this should work great.
       
      The resulting reports contains the following special fields:
       
         
      • feed.url: The URI using the file:// scheme and localhost, with the full path to the processed file.
        •  
      • extra.file_name: The file name (without path) of the processed file.
        •  
       
      Chunking
       
      Additionally, for line-based inputs the bot can split up large reports into smaller chunks.
       
      This is particularly important for setups that use Redis as a message queue which has a per-message size limitation of 512 MB.
       
      To configure chunking, set chunk_size to a value in bytes. chunk_replicate_header determines whether the header line should be repeated for each chunk that is passed on to a parser bot.
       
      Specifically, to configure a large file input to work around Redis' size limitation set chunk_size to something like 384000, i.e., ~384 MB.
       
      Workflow
       
      The bot loops over all files in path and tests if their file name matches *postfix, e.g. *.csv. If yes, the file will be read and inserted into the queue.
       
      If delete_file is set, the file will be deleted after processing. If deletion is not possible, the bot will stop.
       
      To prevent data loss, the bot also stops when no postfix is set and delete_file was set. This cannot be overridden.
       
      The bot always sets the file name as feed.url.
       
      Module: intelmq.bots.collectors.file.collector_file
       
      Parameters (also expects feed parameters):
       
      path
       
      (required, string) Path to file.
       
      postfix
       
      (required, string) The postfix (file ending) of the files to look for. For example [.csv].
       
      delete_file
       
      (optional, boolean) Whether to delete the file after reading. Defaults to false.
      "},{"location":"user/bots/#fireeye","title":"FireEye","text":"
      This bot is capable of collecting hashes and URLs from a FireEye MAS appliance.
       
      The Python library xmltodict is required to run this bot.
       
      Workflow
       
      The bot collects all alerts which occurred during specified duration. After this we make a second call and check if there is additional information like domains and hashes available. After collecting the openioc data we send this information to the Fireeye parser.
       
      Module: intelmq.bots.collectors.fireeye.collector_fireeye
       
      Parameters (also expects feed parameters):
       
      host
       
      (required, string) DNS name of the target appliance.
       
      request_duration
       
      (required, string) Allowed values: 24_hours or 48_hours. Length of the query in past eg. collect alerts from last 24hours/48hours.
       
      http_username
       
      (required, string) Password for authentication.
       
      http_password
       
      (required, string) Username for authentication.
      "},{"location":"user/bots/#kafka","title":"Kafka","text":"
      Requires the kafka python library.
       
      Module: intelmq.bots.collectors.kafka.collector
       
      Parameters (also expects feed parameters):
       
      topic
       
      (required, string) Kafka topic the collector should get messages from.
       
      bootstrap_servers
       
      (required, string) Kafka server(s) and port the collector should connect to. Defaults to localhost:9092
       
      ssl_check_hostname
       
      (optional, boolean) Whether to verify TLS certificates. Defaults to true.
       
      ssl_client_certificate
       
      (optional, string) Path to client certificate to use for TLS connections.
       
      ssl_ca_certificate
       
      (optional, string) Path to trusted CA certificate.
      "},{"location":"user/bots/#misp-generic","title":"MISP Generic","text":"
      Collects messages from MISP, a malware information sharing platform server.
       
      Workflow
       
      This collector will search for events on a MISP server that have a [to_process] tag attached to them (see the [misp_tag_to_process] parameter) and collect them for processing by IntelMQ. Once the MISP event has been processed the [to_process] tag is removed from the MISP event and a [processed] tag is then attached (see the [misp_tag_processed] parameter).
       
      NB. The MISP tags must be configured to be 'exportable' otherwise they will not be retrieved by the collector.
       
      Module: intelmq.bots.collectors.misp.collector
       
      Parameters (also expects feed parameters):
       
      misp_url
       
      (required, string) URL of MISP server (with trailing '/').
       
      misp_key
       
      (required, string) MISP Authkey.
       
      misp_tag_to_process
       
      (required, string) MISP tag for events to be processed.
       
      misp_tag_processed
       
      (optional, string) MISP tag for processed events.
       
      http_verify_cert
       
      (optional, boolean) Verify the TLS certificate of the server. Defaults to true.
      "},{"location":"user/bots/#request-tracker","title":"Request Tracker","text":"
      Request Tracker Collector fetches attachments from an RTIR instance.
       
      This rt bot will connect to RT and inspect the given search_queue for tickets matching all criteria in search_*, Any matches will be inspected. For each match, all (RT-) attachments of the matching RT tickets are iterated over and within this loop, the first matching filename in the attachment is processed. If none of the filename matches apply, the contents of the first (RT-) \"history\" item is matched against the regular expression for the URL (url_regex).
       
      The parameter http_timeout_max_tries is of no use in this collector.
       
      Search
       
      The parameters prefixed with search_ allow configuring the ticket search.
       
      Empty strings and null as value for search parameters are ignored.
       
      File downloads
       
      Attachments can be optionally unzipped, remote files are downloaded with the http_* settings applied.
       
      If url_regex or attachment_regex are empty strings, false or null, they are ignored.
       
      Ticket processing
       
      Optionally, the RT bot can \"take\" RT tickets (i.e. the user is assigned this ticket now) and/or the status can be changed (leave set_status empty in case you don't want to change the status). Please note however that you MUST do one of the following: either \"take\" the ticket or set the status (set_status). Otherwise, the search will find the ticket every time and get stuck in an endless loop.
       
      In case a resource needs to be fetched and this resource is permanently not available (status code is 4xx), the ticket status will be set according to the configuration to avoid processing the ticket over and over. For temporary failures the status is not modified, instead the ticket will be skipped in this run.
       
      Time search
       
      To find only tickets newer than a given absolute or relative time, you can use the search_not_older_than parameter. Absolute time specification can be anything parseable by dateutil, best use a ISO format.
       
      Relative must be in this format: [NUMBER] [TIMESPAN]s, e.g. 3 days. Timespan can be hour, day, week, month or year. Trailing 's' is supported for all timespans. Relative times are subtracted from the current time directly before the search is performed.
       
      The resulting reports contains the following special fields:
       
         
      • rtir_id: The ticket ID
        •  
      • extra.email_subject and extra.ticket_subject: The subject of the ticket
        •  
      • extra.email_from and extra.ticket_requestors: Comma separated list of the ticket's requestor's email addresses.
        •  
      • extra.ticket_owner: The ticket's owner name
        •  
      • extra.ticket_status: The ticket's status
        •  
      • extra.ticket_queue: The ticket's queue
        •  
      • extra.file_name: The name of the extracted file, the name of the downloaded file or the attachments' filename without .gz postfix.
        •  
      • time.observation: The creation time of the ticket or attachment.
        •  
       
      Requirements
       
      You need the rt-library >= 1.9 and < 3.0 from from nic.cz, available via pypi: pip3 install rt<3
       
      Module: intelmq.bots.collectors.rt.collector_rt
       
      Parameters (also expects feed parameters and HTTP parameters):
       
      extract_attachment
       
      (optional, boolean/array of strings) See documentation of the Generic URL Fetcher parameter extract_files for more details.
       
      extract_download
       
      (optional, boolean/array of strings) See documentation of the Generic URL Fetcher parameter extract_files for more details.
       
      uri
       
      (optional, string) URL of the REST interface of the RT. Defaults to http://localhost/rt/REST/1.0.
       
      user
       
      (optional, string) RT username. Defaults to intelmq.
       
      password
       
      (optional, string) RT password. Defaults to password.
       
      search_not_older_than
       
      (optional, string) Absolute time (use ISO format) or relative time, e.g. 3 days.
       
      search_owner
       
      (optional, string) Owner of the ticket to search for. Defaults to nobody.
       
      search_queue
       
      (optional, string) Queue of the ticket to search for. Defaults to Incident Reports.
       
      search_requestor
       
      (optional, string) E-mail address of the requestor.
       
      search_status
       
      (optional, string) Status of the ticket to search for. Defaults to new.
       
      search_subject_like
       
      (optional, string/array of strings) Part of the subject of the ticket to search for. Defaults to \"Report\".
       
      search_subject_notlike
       
      (optional, string/array of strings) Exclude subject containing given value, use list for multiple excluding values.
       
      set_status
       
      (optional, string) Status to set the ticket to after processing. Use false or null to keep current status. Defaults to open.
       
      take_ticket
       
      (optional, boolean) Whether to take the ticket. Defaults to true.
       
      url_regex
       
      (optional, string) Regular expression of an URL to search for in the ticket. Defaults to https://dl.shadowserver.org/[a-zA-Z0-9?_-]*.
       
      attachment_regex
       
      (optional, string) Eegular expression of an attachment in the ticket. Defaults to \\.csv\\.zip$.
      "},{"location":"user/bots/#rsync","title":"Rsync","text":"
      This bot downloads a file via rsync and then load data from downloaded file. Downloaded file is located in var/lib/bots/rsync_collector.
       
      Requires the rsync executable.
       
      Module: intelmq.bots.collectors.rsync.collector_rsync
       
      Parameters (also expects feed parameters):
       
      file
       
      (required, string) The filename to process, combined with rsync_path.
       
      rsync_path
       
      (required, string) Path to the directory of the file. Allowed values are local directory (such as /home/username/) or remote directory (such as <username@remote_host>:/home/username/directory).
       
      rsync_file_path_formatting
       
      (optional, boolean) Whether the file and rsync_path should be formatted by the given format. E.g. if the path is /path/to_file/{time[%Y]}, then the resulting path is /path/to/file/2023 for the year 2023. (Python's Format Specification Mini-Language <https://docs.python.org/3/library/string.html#formatspec> is used for this.). You may use a JSON specifying time-delta <https://docs.python.org/3/library/datetime.html#datetime.timedelta> parameters to shift the current time accordingly. For example use {\"days\": -1} for the yesterday's date; the path /path/to/file/{time[%Y-%m-%d]} will get translated to \"/path/to/file/2018-12-31\" for the 1st Jan of 2023. Defaults to false.
       
      extra_params
       
      (optional, array of strings) A list of extra parameters to pass to rsync.
       
      private_key
       
      (optional, string) Private key to use for rsync authentication.
       
      private_key_path
       
      (optional, string) Path to private key to use for rsync authentication. Use private_key or private_key_path, not both.
       
      strict_host_key_checking
       
      (optional, boolean) Whether the host key should be checked. Defaults to false.
       
      temp_directory
       
      (optional, string) The temporary directory for rsync to use for collected files. Defaults to /opt/intelmq/var/run/{BOT-ID} or /var/run/intelmq/{BOT-ID}.
      "},{"location":"user/bots/#shadowserver-reports-api","title":"Shadowserver Reports API","text":"
      Connects to the Shadowserver API, requests a list of all the reports for a specific country and processes the ones that are new.
       
      The Cache is required to memorize which files have already been processed (TTL needs to be high enough to cover the oldest files available!).
       
      The resulting reports contain the following special field:
       
         
      • extra.file_name: The name of the downloaded file, with fixed filename extension.
        •  
       
      Module: intelmq.bots.collectors.shadowserver.collector_reports_api
       
      Parameters (also expects feed parameters and cache parameters):
       
      apikey
       
      (required, string) Your Shadowserver API key.
       
      secret
       
      (required, string) Your Shadowserver API secret.
       
      reports
       
      (required, string/array of strings) An array of strings (or a list of comma-separated values) of the mailing lists you want to process.
       
      types
       
      (optional, string/array of strings) An array of strings (or a list of comma-separated values) with the names of report types you want to process. If you leave this empty, all the available reports will be downloaded and processed (i.e. 'scan', 'drones', 'intel', 'sandbox_connection', 'sinkhole_combined'). The possible report types are equivalent to the file names defined the the schema.  Please see the Supported Reports of the Shadowserver parser for details.
       
      Sample configuration
       
        shadowserver-collector:\n    description: Our bot responsible for getting reports from Shadowserver\n    enabled: true\n    group: Collector\n    module: intelmq.bots.collectors.shadowserver.collector_reports_api\n    name: Shadowserver_Collector\n    parameters:\n      destination_queues:\n        _default: [shadowserver-parser-queue]\n      file_format: csv\n      api_key: \"$API_KEY_received_from_the_shadowserver_foundation\"\n      secret: \"$SECRET_received_from_the_shadowserver_foundation\"\n    run_mode: continuous\n
      "},{"location":"user/bots/#shodan-stream","title":"Shodan Stream","text":"
      Queries the Shodan Streaming API.
       
      Requires the shodan library to be installed:
       
         
      •  
        https://github.com/achillean/shodan-python/
         
        •  
      •  
        https://pypi.org/project/shodan/
         
        •  
       
      Module: intelmq.bots.collectors.shodan.collector_stream
       
      Parameters (also expects feed parameters and HTTP parameters):
       
      Only the proxy is used (requires shodan-python > 1.8.1). Certificate is always verified.
       
      countries
       
      () A list of countries to query for. If it is a string, it will be spit by ,.
       
      alert
       
      () Alert ID from monitor.shodan.io.
       
      If the stream is interrupted, the connection will be aborted using the timeout parameter. No error will be logged if the number of consecutive connection fails does not reach the parameter error_max_retries. Instead of errors, an INFO message is logged. This is a measurement against too frequent ERROR logging messages. The consecutive connection fails are reset if a data line has been successfully transferred. If the consecutive connection fails reaches the parameter error_max_retries, an exception will be thrown and rate_limit applies, if not null.
      "},{"location":"user/bots/#tcp","title":"TCP","text":"
      TCP is the bot responsible to receive events on a TCP port (ex: from TCP Output of another IntelMQ instance). Might not be working on Python 3.4.6.
       
      Response
       
      TCP collector just sends an \"OK\" message after every received message, this should not pose a problem for an arbitrary input. If you intend to link two IntelMQ instance via TCP, have a look at the TCP output bot documentation.
       
      Module: intelmq.bots.collectors.tcp.collector
       
      Parameters (also expects feed parameters):
       
      ip
       
      (required, string) IP of the destination server.
       
      port
       
      (required, integer) Port of destination server.
      "},{"location":"user/bots/#blueliv-crimeserver","title":"Blueliv Crimeserver","text":"
      Collects report messages from Blueliv API.
       
      For more information visit https://github.com/Blueliv/api-python-sdk
       
      Module: intelmq.bots.collectors.blueliv.collector_crimeserver
       
      Requirements
       
      Install the required library:
       
      pip3 install -r intelmq/bots/collectors/blueliv/REQUIREMENTS.txt\n
       
      Parameters (also expects feed parameters):
       
      api_key
       
      (required, string) location of information resource, see https://map.blueliv.com/?redirect=get-started#signup
       
      api_url
       
      (optional, string) The optional API endpoint. Defaults to https://freeapi.blueliv.com.
      "},{"location":"user/bots/#calidog-certstream","title":"Calidog Certstream","text":"
      A Bot to collect data from the Certificate Transparency Log (CTL). This bot works based on certstream library (https://github.com/CaliDog/certstream-python)
       
      Module: intelmq.bots.collectors.calidog.collector_certstream
       
      Parameters (also expects feed parameters):
      "},{"location":"user/bots/#eset-eti","title":"ESET ETI","text":"
      Collects data from ESET ETI TAXII server.
       
      For more information visit https://www.eset.com/int/business/services/threat-intelligence/.
       
      Module: intelmq.bots.collectors.eset.collector
       
      Requirements
       
      Install the required cabby library:
       
      pip3 install -r intelmq/bots/collectors/eset/REQUIREMENTS.txt\n
       
      Parameters (also expects feed parameters):
       
      username
       
      (required, string) Your username.
       
      password
       
      (required, string) Your password.
       
      endpoint
       
      (optional, string) Defaults to eti.eset.com.
       
      time_delta
       
      (optional, integer) The time (in seconds) span to look back. Default to 3600.
       
      collection
       
      (required, string) The collection to fetch.
      "},{"location":"user/bots/#mcafee-opendxl","title":"McAfee openDXL","text":"
      Collects messages via McAfee openDXL.
       
      Module: intelmq.bots.collectors.opendxl.collector
       
      Parameters (also expects feed parameters):
       
      dxl_config_file
       
      (required, string) Path to the the configuration file containing required information to connect.
       
      dxl_topic
       
      (optional, string) Name of the DXL topic to subscribe to. Defaults to /mcafee/event/atd/file/report.
      "},{"location":"user/bots/#microsoft-azure","title":"Microsoft Azure","text":"
      Collects blobs from Microsoft Azure using their library.
       
      Iterates over all blobs in all containers in an Azure storage. The Cache is required to memorize which files have already been processed (TTL needs to be high enough to cover the oldest files available!).
       
      This bot significantly changed in a backwards-incompatible way in IntelMQ Version 2.2.0 to support current versions of the Microsoft Azure Python libraries. azure-storage-blob>=12.0.0 is required.
       
      Module: intelmq.bots.collectors.microsoft.collector_azure
       
      Parameters (also expects feed parameters and cache parameters):
       
      connection_string
       
      (required, string) Connection string as given by Microsoft.
       
      container_name
       
      (required, string) Name of the container to connect to.
      "},{"location":"user/bots/#microsoft-interflow","title":"Microsoft Interflow","text":"
      This bot collects files from Microsoft Interflow API.
       
      Iterates over all files available by this API. Make sure to limit the files to be downloaded with the parameters, otherwise you will get a lot of data! The cache is used to remember which files have already been downloaded. Make sure the TTL is high enough, higher than not_older_than.
       
      Module: intelmq.bots.collectors.microsoft.collector_interflow
       
      Parameters (also expects feed parameters):
       
      api_key
       
      (required, string) API generated in their portal.
       
      file_match
       
      (optional, string) Regular expression to match file names.
       
      not_older_than
       
      (optional, integer/datetime) an optional relative (minutes) or absolute time (UTC is assumed) expression to determine the oldest time of a file to be downloaded.
       
      redis_cache_* and especially redis_cache_ttl
       
      Settings for the cache where file names of downloaded files are saved. The cache's TTL must always be bigger than not_older_than.
       
      Additional functionalities
       
      Files are automatically ungzipped if the filename ends with .gz.
      "},{"location":"user/bots/#stomp","title":"STOMP","text":"
      Collects messages from a STOMP server.
       
      Module: intelmq.bots.collectors.stomp.collector
       
      Requirements
       
      Install the stomp.py library from PyPI:
       
      pip3 install -r intelmq/bots/collectors/stomp/REQUIREMENTS.txt\n
       
      Alternatively, you may want to install it using your OS's native packaging tools, e.g.:
       
      apt install python3-stomp\n
       
      Apart from that, depending on what STOMP server you connect to, you may need to obtain, from the organization or company owning the server, one or more of the following security/authentication-related resources:
       
         
      • CA certificate file;
        •  
      • either: client certificate and client certificate's key files,   or: username (STOMP login) and password (STOMP passcode).
        •  
       
      Also, you will need to know an appropriate STOMP destination (aka exchange point), e.g. /exchange/my.example.org/*.*.*.*.
       
      Parameters (also expects feed parameters):
       
      server
       
      (required, string) STOMP server's hostname or IP, e.g. \"n6stream.cert.pl\" (which is default)
       
      port
       
      (optional, integer) STOMP server's port number (default: 61614)
       
      exchange
       
      (required, string) STOMP destination to subscribe to, e.g. \"/exchange/my.org/*.*.*.*\"
       
      heartbeat
       
      (optional, integer) default: 6000
       
      ssl_ca_certificate
       
      (optional, string) Path to CA file, or empty string to load system's default CA certificates
       
      auth_by_ssl_client_certificate
       
      (optional, boolean) Default: true (note: false is needed for new n6 auth)
       
      ssl_client_certificate
       
      (optional, string) Path to client certificate to use for TLS connections.
       
      ssl_client_certificate_key
       
      (optional, string) Path to client private key to use for TLS connections.
       
      username
       
      (optional, string) Username to use.
       
      password
       
      (optional, string) Password to use.
      "},{"location":"user/bots/#twitter-remove","title":"Twitter (REMOVE?)","text":"
      Collects tweets.
       
      Collects tweets from target_timelines. Up to tweet_count tweets from each user and up to timelimit back in time. The tweet text is sent separately and if allowed, links to pastebin are followed and the text sent in a separate report
       
      Module: intelmq.bots.collectors.twitter.collector_twitter
       
      Parameters (also expects feed parameters):
       
      target_timelines
       
      () screen_names of twitter accounts to be followed
       
      tweet_count
       
      () number of tweets to be taken from each account
       
      timelimit
       
      () maximum age of the tweets collected in seconds
       
      follow_urls
       
      () list of screen_names for which URLs will be followed
       
      exclude_replies
       
      () exclude replies of the followed screen_names
       
      include_rts
       
      () whether to include retweets by given screen_name
       
      consumer_key
       
      () Twitter API login data
       
      consumer_secret
       
      () Twitter API login data
       
      access_token_key
       
      () Twitter API login data
       
      access_token_secret
       
      () Twitter API login data
      "},{"location":"user/bots/#parser-bots","title":"Parser Bots","text":"
      If not set differently during parsing, all parser bots copy the following fields from the report to an event:
       
         
      • feed.accuracy
        •  
      • feed.code
        •  
      • feed.documentation
        •  
      • feed.name
        •  
      • feed.provider
        •  
      • feed.url
        •  
      • rtir_id
        •  
      • time.observation
        •  
      "},{"location":"user/bots/#common-parameters","title":"Common parameters","text":""},{"location":"user/bots/#default_fields","title":"default_fields","text":"
      (optional, object) Map of statically added fields to each event (only applied if parsing the event doesn't set the value).
       
      example usage:
       
      defaults_fields:\n  classification.type: c2-server\n  protocol.transport: tcp\n
      "},{"location":"user/bots/#copy_collector_provided_fields","title":"copy_collector_provided_fields","text":"
      (optional, list) List of additional fields to be copy from the report (only applied if parsing the event doesn't set the value).
       
      Example usage:
       
      copy_collector_provided_fields:\n  - extra.file_name\n
      "},{"location":"user/bots/#abusech-feodo-tracker","title":"Abuse.ch Feodo Tracker","text":"
      Parses data from Abuse.ch Feodo Tracker (JSON format).
       
      Module: intelmq.bots.parsers.abusech.parser_feodotracker
       
      No additional parameters.
      "},{"location":"user/bots/#alienvault-api","title":"AlienVault API","text":"
      Parses data from AlienVault API.
       
      Module: intelmq.bots.parsers.alienvault.parser
       
      No additional parameters.
      "},{"location":"user/bots/#alienvault-otx","title":"AlienVault OTX","text":"
      Parses data from AlientVault Open Threat Exchange (OTX).
       
      Module: intelmq.bots.parsers.alienvault.parser_otx
       
      No additional parameters.
      "},{"location":"user/bots/#anubisnetworks-cyberfeed-stream","title":"AnubisNetworks Cyberfeed Stream","text":"
      Parses data from AnubisNetworks Cyberfeed Stream.
       
      The feed format changes over time. The parser supports at least data from 2016 and 2020.
       
      Events with the Malware \"TestSinkholingLoss\" are ignored, as they are for the feed provider's internal purpose only and should not be processed at all.
       
      Module: intelmq.bots.parsers.anubisnetworks.parser
       
      Parameters:
       
      use_malware_family_as_classification_identifier
       
      (optional, boolean) Use the malw.family field as classification.type. If false, check if the same as malw.variant. If it is the same, it is ignored. Otherwise saved as extra.malware.family. Defaults to true.
      "},{"location":"user/bots/#bambenek","title":"Bambenek","text":"
      Parses data from Bambenek DGA, Domain, and IP feeds.
       
      Module: intelmq.bots.parsers.bambenek.parser
       
      No additional parameters.
      "},{"location":"user/bots/#blocklistde","title":"Blocklist.de","text":"
      Parses data from Blocklist.de feeds.
       
      Module: intelmq.bots.parsers.blocklistde.parser
       
      No additional parameters.
      "},{"location":"user/bots/#blueliv-crimeserver_1","title":"Blueliv Crimeserver","text":"
      Parses data from Blueliv Crimeserver feed.
       
      Module: intelmq.bots.parsers.blueliv.parser_crimeserver
       
      No additional parameters.
      "},{"location":"user/bots/#calidog-certstream_1","title":"Calidog Certstream","text":"
      Parses data from Certificate Transparency Log.
       
      For each domain in the leaf_cert.all_domains object one event with the domain in source.fqdn (and source.ip as fallback) is produced. The seen-date is saved in time.source and the classification type is other.
       
      Module: intelmq.bots.parsers.calidog.parser_certstream
       
      No additional parameters.
      "},{"location":"user/bots/#cert-eu","title":"CERT-EU","text":"
      Parses data from CERT-EU feed (CSV).
       
      Module: intelmq.bots.parsers.certeu.parser_csv
       
      No additional parameters.
      "},{"location":"user/bots/#ci-army","title":"CI Army","text":"
      Parses data from CI Army feed.
       
      Module: intelmq.bots.parsers.ci_army.parser
       
      No additional parameters.
      "},{"location":"user/bots/#cleanmx","title":"CleanMX","text":"
      Parses data from CleanMX feed.
       
      Module: intelmq.bots.parsers.cleanmx.parser
       
      No additional parameters.
      "},{"location":"user/bots/#team-cymru-cap","title":"Team Cymru CAP","text":"
      Parses data from Team Cymru's CSIRT Assistance Program (CAP) feed.
       
      There are two different feeds available:
       
         
      • infected_$date.txt (\"old\")
        •  
      • $certname_$date.txt (\"new\")
        •  
       
      The new will replace the old at some point in time, currently you need to fetch both. The parser handles both formats.
       
      Old feed
       
      As little information on the format is available, the mappings might not be correct in all cases. Some reports are not implemented at all as there is no data available to check if the parsing is correct at all. If you do get errors like Report ... not implement or similar please open an issue and report the (anonymized) example data. Thanks.
       
      The information about the event could be better in many cases but as Cymru does not want to be associated with the report, we can't add comments to the events in the parser, because then the source would be easily identifiable for the recipient.
       
      Module: intelmq.bots.parsers.cymru.parser_cap_program
       
      No additional parameters.
      "},{"location":"user/bots/#team-cymru-full-bogons","title":"Team Cymru Full Bogons","text":"
      Parses data from full bogons feed.
       
      http://www.team-cymru.com/bogon-reference.html
       
      Module: intelmq.bots.parsers.cymru.parser_full_bogons
       
      No additional parameters.
      "},{"location":"user/bots/#cznic-haas","title":"CZ.NIC HaaS","text":"
      Parses data from CZ.NIC Honeypot as a service (HaaS) feed.
       
      Module: intelmq.bots.parsers.cznic.parser_haas
       
      No additional parameters.
      "},{"location":"user/bots/#cznic-proki","title":"CZ.NIC PROKI","text":"
      Parses data from CZ.NIC PROKI API.
       
      Module: intelmq.bots.parsers.cznic.parser_proki
       
      No additional parameters.
      "},{"location":"user/bots/#danger-rulez","title":"Danger Rulez","text":"
      Parses data from Danger Rulez SSH blocklist.
       
      Module: intelmq.bots.parsers.danger_rulez.parser
       
      No additional parameters.
      "},{"location":"user/bots/#dataplane","title":"Dataplane","text":"
      Parses data from Dataplane feed.
       
      Module: intelmq.bots.parsers.dataplane.parser
       
      No additional parameters.
      "},{"location":"user/bots/#dshield-asn","title":"DShield ASN","text":"
      Parses data from DShield ASN feed.
       
      Module: intelmq.bots.parsers.dshield.parser_asn
       
      No additional parameters.
      "},{"location":"user/bots/#dshield-block","title":"DShield Block","text":"
      Parses data from DShield Block feed.
       
      Module: intelmq.bots.parsers.dshield_parser_block
       
      No additional parameters.
      "},{"location":"user/bots/#eset","title":"ESET","text":"
      Parses data from ESET ETI TAXII server.
       
      Supported collections:
       
         
      • \"ei.urls (json)\"
        •  
      • \"ei.domains v2 (json)\"
        •  
       
      Module: intelmq.bots.parsers.eset.parser
       
      No additional parameters.
      "},{"location":"user/bots/#dyn-todo","title":"Dyn (TODO)","text":""},{"location":"user/bots/#fireeye_1","title":"FireEye","text":"
      Parses data from FireEye MAS appliance.
       
      Module: intelmq.bots.parsers.fireeye.parser
       
      No additional parameters.
      "},{"location":"user/bots/#fraunhofer-dga","title":"Fraunhofer DGA","text":"
      Parses data from Fraunhofer DGA feed.
       
      Module: intelmq.bots.parsers.fraunhofer.parser_dga
       
      No additional parameters.
      "},{"location":"user/bots/#generic-csv","title":"Generic CSV","text":"
      Parses CSV data.
       
      Lines starting with # are skipped. Headers won't be interpreted.
       
      Module: intelmq.bots.parsers.generic.parser_csv
       
      Parameters
       
      columns
       
      (required, string/array of strings) A list of strings or a string of comma-separated values with field names. The names must match the IntelMQ Data Format field names. Empty column specifications and columns named __IGNORE__ are ignored. E.g.
       
      columns:\n  - \"source.ip\"\n  - \"source.fqdn\"\n  - \"extra.http_host_header\"\n  - \"__IGNORE__\"\n
       
      is equivalent to:
       
      columns: \"source.ip,source.fqdn,extra.http_host_header,__IGNORE__\"\n
       
      The fourth column is not used in this example.
       
      It is possible to specify multiple columns using the | character. E.g.
       
      columns:\n  - \"source.url|source.fqdn|source.ip\"\n  - \"source.fqdn\"\n  - \"extra.http_host_header\"\n  - \"__IGNORE__\"\n
       
      First, the bot will try to parse the value as URL, if it fails, it will try to parse it as FQDN, if that fails, it will try to parse it as IP, if that fails, an error will be raised. Some use cases:
       
         
      • Mixed data set, e.g. URL/FQDN/IP/NETMASK:
        •  
       
      columns:\n  - \"source.url|source.fqdn|source.ip|source.network\"\n
       
         
      • Parse a value and ignore if it fails:
        •  
       
      columns:\n  - \"source.url|__IGNORE__\"\n
       
      column_regex_search
       
      (optional, object) A dictionary mapping field names (as given per the columns parameter) to regular expression. The field is evaluated using re.search. Eg. to get the ASN out of AS1234 use: {\"source.asn\": \"[0-9]*\"}. Make sure to properly escape any backslashes in your regular expression (see also this issue).
       
      compose_fields
       
      (optional, object) Compose fields from multiple columns, e.g. with data like this:
       
      # Host,Path\nexample.com,/foo/\nexample.net,/bar/\n
       
      Using this parameter:
       
      compose_fields:\n  source.url: \"http://{0}{1}\"\n
       
      You get:
       
      http://example.com/foo/\nhttp://example.net/bar/\n
       
      in the respective source.url fields. The value in the dictionary mapping is formatted whereas the columns are available with their index.
       
      default_url_protocol
       
      (optional, string) For URLs you can give a default protocol which will be prepended to the data. Defaults to null.
       
      delimiter
       
      (optional, string) Character used for columns separation. Defaults to , (comma).
       
      skip_header
       
      (optional, boolean/integer) Whether to skip the first N lines of the input (true equals to 1, false requalis to 0). Lines starting with # will be skipped additionally, make sure you do not skip more lines than needed! Defaults to false/0.
       
      time_format
       
      (optional, string) Allowed values: timestamp, windows_nt or epoch_millis. When null then fuzzy time parsing is used. Defaults to null.
       
      type
       
      (optional, string) Set the classification.type statically. Deprecated in favour of default_fields . Will be removed in IntelMQ 4.0.0.
       
      data_type
       
      (optional, object) Sets the data of specific type, currently only json is a supported value.
       
      Example:
       
      columns:\n  - source.ip\n  - source.url\n  - extra.tags\ndata_type:\n  extra.tags: json\n
       
      It will ensure that extra.tags is treated as JSON.
       
      filter_text
       
      (optional, string) Only process the lines containing or not containing specified text. It is expected to be used in conjunction with filter_type.
       
      filter_type
       
      (optional, string) Allowed values: whitelist or blacklist. When whitelist is used, only lines containing the text specified in filter_text option will be processed. When blacklist is used, only lines NOT containing the text will be processed.
       
      Example (processing ipset format files):
       
      filter_text: 'ipset add '\nfilter_type: whitelist\ncolumns:\n  - __IGNORE__\n  - __IGNORE__\n  - __IGNORE__\n  - source.ip\n
       
      type_translation
       
      (optional, object) If the source does have a field with information for classification.type, but it does not correspond to IntelMQ's types, you can map them to the correct ones. The type_translation field can hold a dictionary, or a string with a JSON dictionary which maps the feed's values to IntelMQ's.
       
      Example:
       
      type_translation:\n  malware_download: \"malware-distribution\"\n
       
      columns_required
       
      (optional, array of booleans) An array of true/false for each column. By default, it is true for every column.
      "},{"location":"user/bots/#github-feed","title":"Github Feed","text":"
      Parses data publicly available on GitHub (should receive from github_api collector).
       
      Module: intelmq.bots.parsers.github_feed.parser
       
      No additional parameters.
      "},{"location":"user/bots/#have-i-been-pwned-callback","title":"Have I Been Pwned Callback","text":"
      Parsers data from the callback of Have I Been Pwned Enterprise Subscription.
       
      Parses breaches and pastes and creates one event per e-mail address. The e-mail address is stored in source.account . classification.type is leak and classification.identifier is breach or paste.
       
      Module: intelmq.bots.parsers.hibp.parser_callback
       
      No additional parameters.
      "},{"location":"user/bots/#html-table","title":"HTML Table","text":"
      Parses tables in HTML documents.
       
      Module: intelmq.bots.parsers.html_table.parser
       
      Parameters:
       
      (required, string/array of strings) A list of strings or a string of comma-separated values with field names. The names must match the IntelMQ Data Format field names. Empty column specifications and columns named __IGNORE__ are ignored. E.g.
       
      columns:\n  - \"source.ip\"\n  - \"source.fqdn\"\n  - \"extra.http_host_header\"\n  - \"__IGNORE__\"\n
       
      is equivalent to:
       
      columns: \"source.ip,source.fqdn,extra.http_host_header,__IGNORE__\"\n
       
      The fourth column is not used in this example.
       
      It is possible to specify multiple columns using the | character. E.g.
       
      columns:\n  - \"source.url|source.fqdn|source.ip\"\n  - \"source.fqdn\"\n  - \"extra.http_host_header\"\n  - \"__IGNORE__\"\n
       
      First, the bot will try to parse the value as URL, if it fails, it will try to parse it as FQDN, if that fails, it will try to parse it as IP, if that fails, an error will be raised. Some use cases:
       
         
      • Mixed data set, e.g. URL/FQDN/IP/NETMASK:
        •  
       
      columns:\n  - \"source.url|source.fqdn|source.ip|source.network\"\n
       
         
      • Parse a value and ignore if it fails:
        •  
       
      columns:\n  - \"source.url|__IGNORE__\"\n
       
      ignore_values
       
      (optional, string/array of strings) A list of strings or a string of comma-separated values which are ignored when encountered.
       
      Example:
       
      ignore_values:\n  - \"\"\n  - \"unknown\"\n  - \"Not listed\"\n
       
      The following configuration will lead to assigning all values to malware.name and extra.SBL except unknown and Not listed respectively.
       
      columns:\n  - source.url\n  - malware.name\n  - extra.SBL\nignore_values:\n  - ''\n  - unknown\n  - Not listed\n
       
      Parameters columns and ignore_values must have same length!
       
      attribute_name
       
      (optional, string) Filtering table with table attributes. To be used in conjunction with attribute_value. E.g. class, id, style.
       
      attribute_value
       
      (optional, string) To filter all tables with attribute class='details' use
       
      attribute_name: \"class\"\nattribute_value: \"details\"\n
       
      table_index
       
      (optional, integer) Index of the table if multiple tables present. If attribute_name and attribute_value given, index according to tables remaining after filtering with table attribute. Defaults to 0.
       
      split_column
       
      (optional, ) Padded column to be split to get values, to be used in conjunction with split_separator and split_index, optional.
       
      split_separator
       
      (optional, string) Delimiter string for padded column.
       
      split_index
       
      (optional, integer) Index of unpadded string in returned list from splitting split_column with split_separator as delimiter string. Defaults to 0.
       
      Example:
       
      split_column: \"source.fqdn\"\nsplit_separator: \" \"\nsplit_index: 1\n
       
      With above configuration, column corresponding to source.fqdn with value D lingvaworld.ru will be assigned as source.fqdn: lingvaworld.ru.
       
      skip_table_head
       
      (optional, boolean) Skip the first row of the table. Defaults to true.
       
      default_url_protocol
       
      (optional, string) For URLs you can give a default protocol which will be pretended to the data. Defaults to http://.
       
      time_format
       
      (optional, string) Allowed values: timestamp, windows_nt or epoch_millis. When null then fuzzy time parsing is used. Defaults to null.
       
      html_parser
       
      (optional, string) The HTML parser to use. Allowed values: html.parser or lxml (see also https://www.crummy.com/software/BeautifulSoup/bs4/doc/). Defaults to html.parser.
      "},{"location":"user/bots/#json-todo","title":"JSON (TODO)","text":"
      TODO
       
      Module: intelmq.bots.parsers.json.parser
      "},{"location":"user/bots/#keyvalue-parser","title":"Key=Value Parser","text":"
      Parses text lines in key=value format, for example FortiGate firewall logs.
       
      Parsing limitations
       
      The input must not have (quoted) occurrences of the separator in the values. For example, this is not parsable (with space as separator):
       
      key=\"long value\" key2=\"other value\"\n
       
      In firewall logs like FortiGate, this does not occur. These logs usually look like:
       
      srcip=192.0.2.1 srcmac=\"00:00:5e:00:17:17\"\n
       
      Module: intelmq.bots.parsers.key_value.parser
       
      Parameters:
       
      pair_separator
       
      (optional, string) String separating key=value pairs. Defaults to space.
       
      kv_separator
       
      (optional, string) String separating the key and the value. Defaults to =.
       
      keys
       
      (optional, object) Mapping of original key names to IntelMQ Data Format.
       
      Example:
       
      keys:\n  srcip: source.ip\n  dstip: destination.ip\n
       
      The value mapped to time.source is parsed. If the value is numeric, it is interpreted. Otherwise, or if it fails, it is parsed fuzzy with dateutil. If the value cannot be parsed, a warning is logged per line.
       
      strip_quotes
       
      (optional, boolean) Whether to remove opening and closing quotes from values. Defaults to true.
      "},{"location":"user/bots/#malwarepatrol","title":"MalwarePatrol","text":"
      Parses data from MalwarePatrol feed.
       
      Module: intelmq.bots.parsers.malwarepatrol.parser_dansguardian
       
      No additional parameters.
      "},{"location":"user/bots/#malwareurl","title":"MalwareURL","text":"
      Parses data from MalwareURL feed.
       
      Module: intelmq.bots.parsers.malwareurl.parser
       
      No additional parameters.
      "},{"location":"user/bots/#mcafee-advanced-threat-defense-file","title":"McAfee Advanced Threat Defense File","text":"
      Parse IoCs from McAfee Advanced Threat Defense reports (hash, IP, URL).
       
      Module: intelmq.bots.parsers.mcafee.parser_atd
       
      Parameters:
       
      verdict_severity
       
      (optional, integer) Minimum report severity to parse. Defaults to 4.
      "},{"location":"user/bots/#microsoft-ctip","title":"Microsoft CTIP","text":"
      Parses data from the Microsoft CTIP feed.
       
      Can parse the JSON format provided by the Interflow interface (lists of dictionaries) as well as the format provided by the Azure interface (one dictionary per line). The provided data differs between the two formats/providers.
       
      The parser is capable of parsing both feeds:
       
         
      • ctip-c2
        •  
      • ctip-infected-summary The feeds only differ by a few fields, not in the format.
        •  
       
      The feeds contain a field called Payload which is nearly always a base64 encoded JSON structure. If decoding works, the contained fields are saved as extra.payload.*, otherwise the field is saved as extra.payload.text.
       
      Module: intelmq.bots.parsers.microsoft.parser_ctip
       
      Parameters:
       
      overwrite
       
      (optional, boolean) Overwrite an existing field feed.name with DataFeed of the source. Defaults to false.
      "},{"location":"user/bots/#misp","title":"MISP","text":"
      Parses MISP events.
       
      MISP events collected by the MISPCollectorBot are passed to this parser for processing. Supported MISP event categories and attribute types are defined in the SUPPORTED_MISP_CATEGORIES and MISP_TYPE_MAPPING class constants.
       
      Module: intelmq.bots.parsers.misp.parser
       
      No additional parameters.
      "},{"location":"user/bots/#n6","title":"N6","text":"
      Parses n6 data into IntelMQ format.
       
      Test messages are ignored, this is logged with debug logging level. Also contains a mapping for the classification ( results in taxonomy, type and identifier). The name field is normally used as malware.name, if that fails due to disallowed characters, these characters are removed and the original value is saved as event_description.text. This can happen for names like further iocs: text with invalid ' char.
       
      If a n6 message contains multiple IP addresses, multiple events are generated, resulting in events only differing in the address information.
       
      Module: intelmq.bots.parsers.n6.parser_n6stomp
       
      No additional parameters.
      "},{"location":"user/bots/#openphish-free","title":"OpenPhish Free","text":"
      Parses data from OpenPhish Free feed.
       
      Module: intelmq.bots.parsers.openphish.parser
       
      No additional parameters.
      "},{"location":"user/bots/#openphish-premium","title":"OpenPhish Premium","text":"
      Parses data from OpenPhish Premium feed (JSON).
       
      Module: intelmq.bots.parsers.openphish.parser_commercial
       
      No additional parameters.
      "},{"location":"user/bots/#phishtank","title":"Phishtank","text":"
      Parses data from Phishtank feed.
       
      Module: intelmq.bots.parsers.phishtank.parser
       
      No additional parameters.
      "},{"location":"user/bots/#shadowserver","title":"Shadowserver","text":"
      The Shadowserver parser operates on CSV formatted data.
       
      How this bot works?
       
      There are two possibilities for the bot to determine which report type the data belongs to in order to determine the correct mapping of the columns:
       
         
      1.  
        Automatic report type detection
         
        Since IntelMQ version 2.1 the parser can detect the feed based on metadata provided by the collector.
         
        When processing a report, this bot takes extra.file_name from the report and looks in config.py how the report should be parsed. If this lookup is not possible, and the feedname is not given as parameter, the feed cannot be parsed.
         
        The field extra.file_name has the following structure: %Y-%m-%d-${report_name}[-suffix].csv where the optional suffix can be something like country-geo. For example, some possible filenames are 2019-01-01-scan_http-country-geo.csv or 2019-01-01-scan_tftp.csv. The important part is the report_name, between the date and the suffix. Since version 2.1.2 the date in the filename is optional, so filenames like scan_tftp.csv are also detected.
         
        2.  
      2.  
        Fixed report type
         
        If the method above is not possible and for upgraded instances, the report type can be set with the feedname parameter. Report type is derived from the subject of Shadowserver e-mails. A list of possible values of the feedname parameter can be found in the table below in the column \"Report Type\".
         
        3.  
       
      Module:
       
      intelmq.bots.parsers.shadowserver.parser
       
      Parameters:
       
      feedname
       
      (optional, string) Name of the Shadowserver report. The value for each report type can be found in the schema feed_name field.
       
      For example using curl -s https://interchange.shadowserver.org/intelmq/v1/schema | jq .[].feed_name.
       
      overwrite
       
      (optional, boolean) If an existing feed.name should be overwritten.
       
      auto_update
       
      (optional, boolean) Enable automatic schema download.
       
      Supported reports:
       
      The report configuration is stored in a shadowserver-schema.json file downloaded from https://interchange.shadowserver.org/intelmq/v1/schema.
       
      The parser will attempt to download a schema update on startup when the auto_update option is enabled.
       
      Schema downloads can also be scheduled as a cron job for the intelmq user:
       
        02  01 *   *   *     intelmq.bots.parsers.shadowserver.parser --update-schema\n
       
      For air-gapped systems automation will be required to download and copy the file to VAR_STATE_PATH/shadowserver-schema.json.
       
      The parser will automatically reload the configuration when the file changes.
       
      Schema contract
       
      Once set in the schema, the classification.identifier, classification.taxonomy, and classification.type fields will remain static for a specific report.
       
      The schema revision history is maintained at https://github.com/The-Shadowserver-Foundation/report_schema/.
       
      Sample configuration
       
        shadowserver-parser:\n    bot_id: shadowserver-parser\n    name: Shadowserver Parser\n    enabled: true\n    group: Parser\n    groupname: parsers\n    module: intelmq.bots.parsers.shadowserver.parser\n    parameters:\n      destination_queues:\n        _default: [file-output-queue]\n      auto_update: true\n    run_mode: continuous\n
      "},{"location":"user/bots/#shodan","title":"Shodan","text":"
      Parses data from Shodan (search, stream etc).
       
      The parser is by far not complete as there are a lot of fields in a big nested structure. There is a minimal mode available which only parses the important/most useful fields and also saves everything in extra.shodan keeping the original structure. When not using the minimal mode if may be useful to ignore errors as many parsing errors can happen with the incomplete mapping.
       
      Module: intelmq.bots.parsers.shodan.parser
       
      Parameters:
       
      ignore_errors
       
      (optional, boolean) Defaults to true.
       
      minimal_mode
       
      (optional, boolean) Defaults to false.
      "},{"location":"user/bots/#spamhaus-drop","title":"Spamhaus DROP","text":"
      Parses data from Spamhaus DROP feed.
       
      Module: intelmq.bots.parsers.spamhaus.parser_drop
       
      No additional parameters.
      "},{"location":"user/bots/#spamhaus-cert","title":"Spamhaus CERT","text":"
      Parses data from Spamhaus CERT feed.
       
      Module: intelmq.bots.parsers.spamhaus.parser_cert
       
      No additional parameters.
      "},{"location":"user/bots/#surbl","title":"Surbl","text":"
      Parses data from surbl feed.
       
      Module: intelmq.bots.parsers.surbl.parser
       
      No additional parameters.
      "},{"location":"user/bots/#threatminer","title":"Threatminer","text":"
      Parses data from Threatminer feed.
       
      Module: intelmq.bots.parsers.threatminer.parser
       
      No additional parameters.
      "},{"location":"user/bots/#turris","title":"Turris","text":"
      Parses data from Turris Greylist feed.
       
      Module: intelmq.bots.parsers.turris.parser
       
      No additional parameters.
      "},{"location":"user/bots/#twitter","title":"Twitter","text":"
      Extracts URLs from text, fuzzy, aimed at parsing tweets.
       
      Module: intelmq.bots.parsers.twitter.parser
       
      Parameters:
       
      domain_whitelist
       
      (optional, array of strings) domains to be filtered out
       
      substitutions
       
      (optional, string) Semicolon delimited list of even length of pairs of substitutions (for example: .;.;,;. substitutes . for . and , for .).
       
      classification_type
       
      (optional, string) Statically set classification.type.
       
      default_scheme
       
      (optional, string) Default scheme for URLs if not given. See also the next section.
       
      Default scheme
       
      The dependency url-normalize changed it's behavior in version 1.4.0 from using http:// as default scheme to https://. Version 1.4.1 added the possibility to specify it. Thus you can only use the default_scheme parameter with a current version of this library >= 1.4.1, with 1.4.0 you will always get https:// as default scheme and for older versions < 1.4.0 http:// is used.
       
      This does not affect URLs which already include the scheme.
      "},{"location":"user/bots/#vxvault","title":"VxVault","text":"
      Parses data from VxVault feed.
       
      Module: intelmq.bots.parsers.vxvault.parser
       
      No additional parameters.
      "},{"location":"user/bots/#zoneh","title":"ZoneH","text":"
      Parses data from ZoneH.
       
      This bot is designed to consume defacement reports from zone-h.org. It expects fields normally present in CSV files distributed by email.
       
      Module: intelmq.bots.parsers.zoneh.parser
       
      No additional parameters.
      "},{"location":"user/bots/#expert-bots","title":"Expert Bots","text":"
      Expert bots are used for enriching, filtering and/or other data manipulation.
      "},{"location":"user/bots/#abusix","title":"Abusix","text":"
      This bot adds source.abuse_contact and destination.abuse_contact e-mail addresses. They are obtained via DNS TXT queries to Abusix servers.
       
      Requirements
       
      This bot can optionally use the python module querycontacts by Abusix itself: https://pypi.org/project/querycontacts/
       
      pip3 install querycontacts\n
       
      If the package is not installed, our own routines are used.
       
      Module: intelmq.bots.experts.abusix.expert
       
      Parameters (also expects cache parameters):
       
      No additional parameters.
      "},{"location":"user/bots/#aggregate","title":"Aggregate","text":"
      Aggregates events based upon given fields & timespan.
       
      Define specific fields to filter incoming events and aggregate them. Also set the timespan you want the events to get aggregated.
       
      The \"cleanup\" procedure, sends out the aggregated events or drops them based upon the given threshold value. It is called on every incoming message and on the bot's initialization. If you're potentially running on low traffic ( no incoming events within the given timestamp ) it is recommended to reload or restart the bot via cronjob each 30 minutes (adapt to your configured timespan). Otherwise you might loose information.
       
      I. e.:
       
      crontab -e\n\n0,30 * * * * intelmqctl reload my-aggregate-bot\n
       
      For reloading/restarting please check the intelmqctl documentation.
       
      Module: intelmq.bots.experts.aggregate.expert
       
      Parameters (also expects cache parameters):
       
      Warning
       
      redis_cache_ttl is not used at it would result in data loss.
       
      fields
       
      (required, string) Given fields which are used to aggregate like classification.type, classification.identifier.
       
      threshold
       
      (required, integer) If the aggregated event is lower than the given threshold after the timespan, the event will get dropped.
       
      timespan
       
      (required, string) Timespan to aggregate events during the given time. I. e. 1 hour
      "},{"location":"user/bots/#asn-lookup","title":"ASN Lookup","text":"
      This bot uses an offline database to add source.asn and destination.asn based on the respective IP address.
       
      Requirements
       
      Install pyasn module.
       
      pip3 install pyasn\n
       
      Module: intelmq.bots.experts.asn_lookup.expert
       
      Parameters:
       
      database
       
      (required, string) Path to the downloaded database.
       
      Database
       
      Use this command to create/update the database and reload the bot:
       
      intelmq.bots.experts.asn_lookup.expert --update-database\n
       
      The database is fetched from routeviews.org and licensed under the Creative Commons Attribution 4.0 International license (see the routeviews FAQ).
      "},{"location":"user/bots/#csv-converter","title":"CSV Converter","text":"
      Converts an event to CSV format, saved in the output field.
       
      To use the CSV-converted data in an output bot - for example in a file output, use the configuration parameter single_key of the output bot and set it to output.
       
      Module: intelmq.bots.experts.csv_converter.expert
       
      Parameters:
       
      delimiter
       
      (optional, string) Defaults to ,.
       
      fieldnames
       
      (required, string) Comma-separated list of field names, e.g. \"time.source,classification.type,source.ip\".
      "},{"location":"user/bots/#team-cymru-whois","title":"Team Cymru Whois","text":"
      This bot adds geolocation, ASN and BGP prefix based on IP address.
       
      Public documentation: https://www.team-cymru.com/IP-ASN-mapping.html#dns
       
      Module: intelmq.bots.experts.cymru_whois.expert
       
      Parameters (also expects cache parameters):
       
      overwrite
       
      (optional, boolean) Whether to overwrite existing fields. Defaults to true.
      "},{"location":"user/bots/#remove-affix","title":"Remove Affix","text":"
      Remove part of string from string fields, example: www. from source.fqdn.
       
      Module: intelmq.bots.experts.remove_affix.expert
       
      Parameters:
       
      remove_prefix
       
      (optional, boolean) True - cut from start, False - cut from end. Defaults to true.
       
      affix
       
      (required, string) example 'www.'
       
      field
       
      (required, string) Which field to modify. 'source.fqdn'
      "},{"location":"user/bots/#domain-suffix","title":"Domain Suffix","text":"
      This bots uses an offline database to add the public suffix to the event, derived by a domain. See or information on the public suffix list: https://publicsuffix.org/list/. Only rules for ICANN domains are processed. The list can (and should) contain Unicode data, punycode conversion is done during reading.
       
      Note that the public suffix is not the same as the top level domain (TLD). E.g. co.uk is a public suffix, but the TLD is uk. Privately registered suffixes (such as blogspot.co.at) which are part of the public suffix list too, are ignored.
       
      Rule processing
       
      A short summary how the rules are processed:
       
      The simple ones:
       
      com\nat\ngv.at\n
       
      example.com leads to com, example.gv.at leads to gv.at.
       
      Wildcards:
       
      *.example.com\n
       
      www.example.com leads to www.example.com.
       
      And additionally the exceptions, together with the above wildcard rule:
       
      !www.example.com\n
       
      www.example.com does now not lead to www.example.com, but to example.com.
       
      Module: intelmq.bots.experts.domain_suffix.expert
       
      Parameters:
       
      field
       
      (required, string) Allowed values: fqdn or reverse_dns.
       
      suffix_file
       
      (required, string) path to the suffix file
       
      Database
       
      Use this command to create/update the database and reload the bot:
       
      intelmq.bots.experts.domain_suffix.expert --update-database\n
      "},{"location":"user/bots/#domain-valid","title":"Domain Valid","text":"
      Checks if a domain is valid by performing multiple validity checks (see below).
       
      If the field given in domain_field does not exist in the event, the event is dropped. If the domain contains underscores (_), the event is dropped. If the domain is not valid according to the validators library, the event is dropped. If the domain's last part (the TLD) is not in the TLD-list configured by parameter tlds_domains_list, the field is dropped. Latest TLD list: https://data.iana.org/TLD/
       
      Module: intelmq.bots.experts.domain_valid.expert
       
      Parameters:
       
      domain_field
       
      (required, string) The name of the field to be validated.
       
      tlds_domains_list
       
      (required, string) Path to a local file with all valid TLDs. Defaults to /opt/intelmq/var/lib/bots/domain_valid/tlds-alpha-by-domain.txt
      "},{"location":"user/bots/#deduplicator","title":"Deduplicator","text":"
      Bot responsible for dropping duplicate events. Deduplication can be performed based on an arbitrary set of fields.
       
      Module: intelmq.bots.experts.deduplicator.expert
       
      Parameters (also expects cache parameters):
       
      bypass
       
      (optional, boolean) Whether to bypass the deduplicator or not. When set to true, messages will not be deduplicated. Defaults to false.
       
      filter_type
       
      (optional, string) Allowed values: blacklist or whitelist. The filter type will be used to define how Deduplicator bot will interpret the parameter filter_keys in order to decide whether an event has already been seen or not, i.e., duplicated event or a completely new event.
       
         
      • whitelist configuration: only the keys listed in filter_keys will be considered to verify if an event is   duplicated or not.
        •  
      • blacklist configuration: all keys except those in filter_keys will be considered to verify if an event is   duplicated or not.
        •  
       
      filter_keys
       
      (optional, string) string with multiple keys separated by comma. Please note that time.observation key will not be considered even if defined, because the system always ignore that key.
       
      When using a whitelist field pattern and a small number of fields (keys), it becomes more important, that these fields exist in the events themselves. If a field does not exist, but is part of the hashing/deduplication, this field will be ignored. If such events should not get deduplicated, you need to filter them out before the deduplication process, e.g. using a sieve expert. See also this discussion thread on the mailing-list.
       
      Configuration Example
       
      Example 1
       
      The bot with this configuration will detect duplication only based on source.ip and destination.ip keys.
       
      parameters:\n  redis_cache_db: 6\n  redis_cache_host: \"127.0.0.1\"\n  redis_cache_password: null\n  redis_cache_port: 6379\n  redis_cache_ttl: 86400\n  filter_type: \"whitelist\"\n  filter_keys: \"source.ip,destination.ip\"\n
       
      Example 2
       
      The bot with this configuration will detect duplication based on all keys, except source.ip and destination.ip keys.
       
      parameters:\n  redis_cache_db: 6\n  redis_cache_host: \"127.0.0.1\"\n  redis_cache_password: null\n  redis_cache_port: 6379\n  redis_cache_ttl: 86400\n  filter_type: \"blacklist\"\n  filter_keys: \"source.ip,destination.ip\"\n
       
      Flushing the cache
       
      To flush the deduplicator's cache, you can use the redis-cli tool. Enter the database used by the bot and submit the flushdb command:
       
      redis-cli -n 6\nflushdb\n
      "},{"location":"user/bots/#do-portal","title":"DO Portal","text":"
      The DO portal retrieves the contact information from a DO portal instance: http://github.com/certat/do-portal/
       
      Module: intelmq.bots.experts.do_portal.expert
       
      Parameters:
       
      mode
       
      (required, string) Allowed values: replace or append. How to handle new abuse contacts in case there are existing ones.
       
      portal_url
       
      (required, string) The URL to the portal, without the API-path. The used URL is $portal_url + '/api/1.0/ripe/contact?cidr=%s'.
       
      portal_api_key
       
      (required, string) The API key of the user to be used. Must have sufficient privileges.
      "},{"location":"user/bots/#field-reducer","title":"Field Reducer","text":"
      The field reducer bot is capable of removing fields from events.
       
      Module: intelmq.bots.experts.field_reducer.expert
       
      Parameters:
       
      type
       
      (required, string) Allowed values: whitelist or blacklist. When whitelist is set, tnly the fields in keys will passed along. When blacklist is set then the fields in keys will be removed from events.
       
      keys
       
      (required, array of strings) Can be an array of field names or a string with a comma-separated list of field names.
      "},{"location":"user/bots/#filter","title":"Filter","text":"
      The filter bot is capable of filtering specific events.
       
      A simple filter for messages (drop or pass) based on a exact string comparison or regular expression.
       
      Module: intelmq.bots.experts.filter.expert
       
      Parameters:
       
      Parameters for filtering with key/value attributes
       
      filter_key
       
      (required, string) - key from data format
       
      filter_value
       
      (required, string) - value for the key
       
      filter_action
       
      (required, string) - action when a message match to the criteria (possible actions: keep/drop)
       
      filter_regex
       
      (optional, boolean) - attribute determines if the filter_value shall be treated as regular expression or not.
       
      If this attribute is not empty (can be true, yes or whatever), the bot uses python's re.search function to evaluate the filter with regular expressions. If this attribute is empty or evaluates to false, an exact string comparison is performed. A check on string inequality can be achieved with the usage of Paths described below.
       
      Parameters for time based filtering
       
      not_before
       
      (optional, string) Events before this time will be dropped. Example: 1 week.
       
      not_after
       
      (optional, string) - Events after this time will be dropped.
       
      Both parameters accept string values describing absolute or relative time:
       
         
      • absolute
        •  
      • basically anything parseable by datetime parser, eg.
        •  
       
      2015-09-12T06:22:11+00:00\n
       
      time.source
       
      (optional, string) Taken from the event will be compared to this value to decide the filter behavior.
       
         
      • relative
        •  
      • accepted string formatted like this \" \", where epoch could be any of following strings (could   optionally end with trailing 's'): hour, day, week, month, year 
      • time.source taken from the event will be compared to the value (now - relative) to decide the filter behavior
        •  
        Examples of time filter definition
         
           
        • not_before: \"2015-09-12T06:22:11+00:00\" - events older than the specified time will be dropped
          •  
        • not_after: \"6 months\" - just events older than 6 months will be passed through the pipeline
          •  
         
        Possible paths
         
           
        • _default: default path, according to the configuration
          •  
        • action_other: Negation of the default path
          •  
        • filter_match: For all events the filter matched on
          •  
        • filter_no_match: For all events the filter does not match
          •  
         action match _default action_other filter_match filter_no_match keep \u2713 \u2713 \u2717 \u2713 \u2717 keep \u2717 \u2717 \u2713 \u2717 \u2713 drop \u2713 \u2717 \u2713 \u2713 \u2717 drop \u2717 \u2713 \u2717 \u2717 \u2713 
        In DEBUG logging level, one can see that the message is sent to both matching paths, also if one of the paths is not configured. Of course the message is only delivered to the configured paths.
        "},{"location":"user/bots/#format-field","title":"Format Field","text":"
        String method operations on column values.
         
        Module: intelmq.bots.experts.format_field.expert
         
        Parameters:
         
        Parameters for stripping chars
         
        strip_columns (optional, string/array of strings) A list of strings or a string of comma-separated values with field names. The names must match the IntelMQ Data Format field names.
         
        For example:
         
        columns:\n  - malware.name\n  - extra.tags\n
         
        is equivalent to:
         
        columns: \"malware.name,extra.tags\"\n
         
        strip_chars
         
        (optional, string) Set of characters to remove as leading/trailing characters. Defaults to space.
         
        Parameters for replacing chars
         
        replace_column
         
        () key from data format
         
        old_value
         
        () the string to search for
         
        new_value
         
        () the string to replace the old value with
         
        replace_count () number specifying how many occurrences of the old value you want to replace(default: [1])
         
        Parameters for splitting string to list of string
         
        split_column
         
        () key from data format
         
        split_separator
         
        () specifies the separator to use when splitting the string(default: ,)
         
        Order of operation: strip -> replace -> split. These three methods can be combined such as first strip and then split.
        "},{"location":"user/bots/#generic-db-lookup","title":"Generic DB Lookup","text":"
        This bot is capable for enriching intelmq events by lookups to a database. Currently only PostgreSQL and SQLite are supported.
         
        If more than one result is returned, a ValueError is raised.
         
        Module: intelmq.bots.experts.generic_db_lookup.expert
         
        Parameters:
         
        Connection
         
        engine
         
        (required, string) Allowed values: postgresql or sqlite.
         
        database
         
        (optional, string) Database name or the SQLite filename. Defaults to intelmq.
         
        table
         
        (optional, string) Name of the table. Defaults to contacts.
         
        PostgreSQL specific parameters
         
        host
         
        (optional, string) Hostname of the PostgreSQL server. Defaults to localhost.
         
        port
         
        (optional, integer) Port of the PostgreSQL server. Defaults to 5432.
         
        user
         
        (optional, string) Username for accessing PostgreSQL. Defaults to intelmq.
         
        password
         
        (optional, string) Password for accessing PostgreSQL. Defaults to ?.
         
        sslmode
         
        (optional, string) Type of TLS mode to use. Defaults to require.
         
        Lookup
         
        match_fields
         
        (optional, object) The value is a key-value mapping an arbitrary number IntelMQ field names to table column names. The values are compared with = only. Defaults to source.asn: \"asn\".
         
        Replace fields
         
        overwrite
         
        (optional, boolean) Whether to overwrite existing fields. Defaults to false.
         
        replace_fields
         
        (optional, object) Key-value mapping an arbitrary number of table column names to IntelMQ field names. Defaults to {\"contact\": \"source.abuse_contact\"}.
        "},{"location":"user/bots/#gethostbyname","title":"Gethostbyname","text":"
        This bot resolves to IP address (source.ip and destination.ip). Can possibly use also the source.url and destination.url for extracting FQDN.
         
        This bot resolves the DNS name (source.fqdn and destination.fqdn) using the gethostbyname syscall to an IP address (source.ip and destination.ip). The following gaierror resolution errors are ignored and treated as if the hostname cannot be resolved:
         
           
        • -2/EAI_NONAME: NAME or SERVICE is unknown
          •  
        • -4/EAI_FAIL: Non-recoverable failure in name res.
          •  
        • -5/EAI_NODATA: No address associated with NAME.
          •  
        • -8/EAI_SERVICE: SERVICE not supported for `ai_socktype'.
          •  
        • -11/EAI_SYSTEM: System error returned in `errno'.
          •  
         
        Other errors result in an exception if not ignored by the parameter gaierrors_to_ignore. All gaierrors can be found here: http://www.castaglia.org/proftpd/doc/devel-guide/src/lib/glibc-gai_strerror.c.html
         
        Module: intelmq.bots.experts.gethostbyname.expert
         
        Parameters:
         
        fallback_to_url
         
        (optional, boolean) When true and no source.fqdn present, use source.url instead for producing source.ip.
         
        gaierrors_to_ignore
         
        (optional, array of integers) Gaierror codes to ignore, e.g. -3 for EAI_AGAIN (Temporary failure in name resolution). Only accepts the integer values, not the names.
         
        overwrite
         
        (optional, boolean) Whether to overwrite existing source.ip and/or source.destination fields. Defaults to false.
        "},{"location":"user/bots/#http-status","title":"HTTP Status","text":"
        The bot fetches the HTTP status for a given URL and saves it in the event.
         
        Module: intelmq.bots.experts.http.expert_status
         
        Parameters:
         
        field
         
        (required, string) The name of the field containing the URL to be checked.
         
        success_status_codes
         
        (optional, array of integers) An array of success status codes. If this parameter is omitted or the list is empty, successful status codes are the ones between 200 and 400.
         
        overwrite
         
        (optional, boolean) Whether to overwrite existing status field. Defaults to false.
        "},{"location":"user/bots/#http-content","title":"HTTP Content","text":"
        Fetches an HTTP resource and checks if it contains a specific string.
         
        The bot fetches an HTTP resource and checks if it contains a specific string.
         
        Module: intelmq.bots.experts.http.expert_content
         
        Parameters:
         
        field
         
        (optional, string) The name of the field containing the URL to be checked. Defaults to source.url.
         
        needle
         
        (optional, string) The string that the content available on URL is checked for.
         
        overwrite
         
        (optional, boolean) Whether to overwrite existing status field. Defaults to false.
        "},{"location":"user/bots/#idea-converter","title":"IDEA Converter","text":"
        Converts the event to IDEA format and saves it as JSON in the field output. All other fields are not modified.
         
        Documentation about IDEA: https://idea.cesnet.cz/en/index
         
        Module: intelmq.bots.experts.idea.expert
         
        Parameters:
         
        test_mode
         
        (optional, boolean) Adds Test category to mark all outgoing IDEA events as informal (meant to simplify setting up and debugging new IDEA producers). Defaults to true.
        "},{"location":"user/bots/#jinja2-template","title":"Jinja2 Template","text":"
        This bot lets you modify the content of your IntelMQ message fields using Jinja2 templates.
         
        Documentation about Jinja2 templating language: https://jinja.palletsprojects.com/
         
        Module: intelmq.bots.experts.jinja.expert
         
        Parameters:
         
        fields
         
        (required, object) a dict containing as key the name of the field where the result of the Jinja2 template should be written to and as value either a Jinja2 template or a filepath to a Jinja2 template file (starting with file:///). Because the experts decides if it is a filepath based on the value starting with file:/// it is not possible to simply write values starting with file:/// to fields. The object containing the existing message will be passed to the Jinja2 template with the name msg.
         
        fields:\n  output: The provider is {{ msg['feed.provider'] }}!\n  feed.url: \"{{ msg['feed.url'] | upper }}\"\n  extra.somejinjaoutput: file:///etc/intelmq/somejinjatemplate.j2\n
        "},{"location":"user/bots/#lookyloo","title":"Lookyloo","text":"
        Lookyloo is a website screenshotting and analysis tool. For more information and installation instructions visit https://www.lookyloo.eu/
         
        The bot sends a request for source.url to the configured Lookyloo instance and saves the retrieved website screenshot link in the field screenshot_url. Lookyloo only queues the website for screenshotting, therefore the screenshot may not be directly ready after the bot requested it. The pylookyloo library is required for this bot. The http_user_agent parameter is passed on, but not other HTTP-related parameter like proxies.
         
        Events without source.url are ignored.
         
        Module: intelmq.bots.experts.lookyloo.expert
         
        Parameters:
         
        instance_url
         
        (required, string) LookyLoo instance to connect to.
        "},{"location":"user/bots/#maxmind-geoip","title":"MaxMind GeoIP","text":"
        This bot uses an offline database for adding geolocation information based on the IP address (source.ip and destination.ip).
         
        Requirements
         
        The bot requires the MaxMind's geoip2 Python library, version 2.2.0 has been tested.
         
        To download the database a free license key is required. More information can be found at https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/.
         
        Module: intelmq.bots.experts.maxmind_geoip.expert
         
        Parameters:
         
        database
         
        (required, string) Path to the local database file.
         
        overwrite
         
        (optional, boolean) Whether to overwrite existing fields. Defaults to true.
         
        use_registered
         
        (optional, boolean) MaxMind has two country ISO codes: One for the physical location of the address and one for the registered location. See also https://github.com/certtools/intelmq/pull/1344 for a short explanation. Defaults to false (backwards-compatibility).
         
        license_key
         
        (required, string) MaxMind license key is necessary for downloading the GeoLite2 database.
         
        Database
         
        Use this command to create/update the database and reload the bot:
         
        intelmq.bots.experts.maxmind_geoip.expert --update-database\n
        "},{"location":"user/bots/#misp_1","title":"MISP","text":"
        Queries a MISP instance for the source.ip and adds the MISP Attribute UUID and MISP Event ID of the newest attribute found.
         
        Module: intelmq.bots.experts.misp.expert
         
        Parameters:
         
        misp_key
         
        (required, string) MISP Authkey.
         
        misp_url
         
        (required, string) URL of MISP server (with trailing '/')
         
        http_verify_cert
         
        (optional, boolean) Verify the TLS certificate of the server. Default to true.
        "},{"location":"user/bots/#mcafee-active-response-lookup","title":"McAfee Active Response Lookup","text":"
        Queries DXL bus for hashes, IP addresses or FQDNs.
         
        Module: intelmq.bots.experts.mcafee.expert_mar
         
        Parameters:
         
        dxl_config_file
         
        (required, string) Location of the file containing required information to connect to DXL bus.
         
        lookup_type
         
        (required, string) Allowed values:
         
           
        • Hash - Looks up malware.hash.md5, malware.hash.sha1 and malware.hash.sha256.
          •  
        • DestSocket - Looks up destination.ip and destination.port.
          •  
        • DestIP - Looks up destination.ip.
          •  
        • DestFQDN - Looks up in destination.fqdn.
          •  
        "},{"location":"user/bots/#modify","title":"Modify","text":"
        This bots allows you to change arbitrary field values of events using a configuration file.
         
        Module: intelmq.bots.experts.modify.expert
         
        Parameters:
         
        configuration_path
         
        (required, string) Location of the configuration file.
         
        case_sensitive
         
        (optional, boolean) Defaults to true.
         
        maximum_matches
         
        (optional, boolean) Maximum number of matches. Processing stops after the limit is reached. Defaults to null (no limit).
         
        overwrite
         
        (optional, boolean) Overwrite any existing fields by matching rules. Defaults to false.
         
        Configuration File
         
        The modify expert bot allows you to change arbitrary field values of events just using a configuration file. Thus it is possible to adapt certain values or adding new ones only by changing JSON-files without touching the code of many other bots.
         
        The configuration is called modify.conf and looks like this:
         
        [\n  {\n    \"rulename\": \"Standard Protocols http\",\n    \"if\": {\n      \"source.port\": \"^(80|443)$\"\n    },\n    \"then\": {\n      \"protocol.application\": \"http\"\n    }\n  },\n  {\n    \"rulename\": \"Spamhaus Cert conficker\",\n    \"if\": {\n      \"malware.name\": \"^conficker(ab)?$\"\n    },\n    \"then\": {\n      \"classification.identifier\": \"conficker\"\n    }\n  },\n  {\n    \"rulename\": \"bitdefender\",\n    \"if\": {\n      \"malware.name\": \"bitdefender-(.*)$\"\n    },\n    \"then\": {\n      \"malware.name\": \"{matches[malware.name][1]}\"\n    }\n  },\n  {\n    \"rulename\": \"urlzone\",\n    \"if\": {\n      \"malware.name\": \"^urlzone2?$\"\n    },\n    \"then\": {\n      \"classification.identifier\": \"urlzone\"\n    }\n  },\n  {\n    \"rulename\": \"default\",\n    \"if\": {\n      \"feed.name\": \"^Spamhaus Cert$\"\n    },\n    \"then\": {\n      \"classification.identifier\": \"{msg[malware.name]}\"\n    }\n  }\n]\n
         
        In our example above we have five groups labeled Standard Protocols http, Spamhaus Cert conficker, bitdefender, urlzone and default. All sections will be considered, in the given order (from top to bottom).
         
        Each rule consists of conditions and actions. Conditions and actions are dictionaries holding the field names of events and regular expressions to match values (selection) or set values (action). All matching rules will be applied in the given order. The actions are only performed if all selections apply.
         
        If the value for a condition is an empty string, the bot checks if the field does not exist. This is useful to apply default values for empty fields.
         
        Actions
         
        You can set the value of the field to a string literal or number.
         
        In addition you can use the standard Python string format syntax to access the values from the processed event as msg and the match groups of the conditions as matches, see the bitdefender example above. Group 0 ([0]) contains the full matching string. See also the documentation on re.Match.group.
         
        Note that matches will also contain the match groups from the default conditions if there were any.
         
        Examples
         
        We have an event with feed.name = Spamhaus Cert and malware.name = confickerab. The expert loops over all sections in the file and eventually enters section Spamhaus Cert. First, the default condition is checked, it matches! OK, going on. Otherwise the expert would have selected a different section that has not yet been considered. Now, go through the rules, until we hit the rule conficker. We combine the conditions of this rule with the default conditions, and both rules match! So we can apply the action: classification.identifier is set to conficker, the trivial name.
         
        Assume we have an event with feed.name = Spamhaus Cert and malware.name = feodo. The default condition matches, but no others. So the default action is applied. The value for classification.identifier will be set to feodo by {msg[malware.name]}.
         
        Types
         
        If the rule is a string, a regular expression search is performed, also for numeric values (str() is called on them). If the rule is numeric for numeric values, a simple comparison is done. If other types are mixed, a warning will be thrown.
         
        For boolean values, the comparison value needs to be true or false as in JSON they are written all-lowercase.
        "},{"location":"user/bots/#national-cert-contact-lookup-by-certat","title":"National CERT Contact Lookup by CERT.AT","text":"
        https://contacts.cert.at offers an IP address to national CERT contact (and cc) mapping.
         
        Module: intelmq.bots.experts.national_cert_contact_certat.expert
         
        Parameters:
         
        filter
         
        (optional, boolean) Whether to act as a filter for AT.
         
        overwrite_cc
         
        (optional, boolean) Set to true if you want to overwrite any potentially existing cc fields in the event. Defaults to false.
        "},{"location":"user/bots/#rdap","title":"RDAP","text":"
        This bot queries RDAP servers for additional information about a domain.
         
        Module: intelmq.bots.experts.rdap.expert
         
        Parameters:
         
        rdap_order
         
        (optional, array of strings) Search order of contacts with these roles. Defaults to [\"abuse\", \"technical\"].
         
        rdap_bootstrapped_servers
         
        (optional, object) Customized RDAP servers. Do not forget the trailing slash. For example:
         
        {\n  \"at\": {\n    \"url\": \"rdap.server.at/v1/\",\n    \"auth\": {\n      \"type\": \"jwt\",\n      \"token\": \"ey...\"\n    }\n  },\n  \"de\": \"rdap.service:1337/v1/\"\n}\n
        "},{"location":"user/bots/#recordedfuture-ip-risk","title":"RecordedFuture IP Risk","text":"
        This bot tags events with the score found in RecordedFuture large IP risklist.
         
        Record risk score associated to source and destination IP if they are present. Assigns 0 to IP addresses not in the RF list.
         
        For both source.ip and destination.ip the corresponding risk score is fetched from a local database created from RecordedFuture's API. The score is recorded in extra.rf_iprisk.source and extra.rf_iprisk.destination. If a lookup for an IP fails a score of 0 is recorded.
         
        See https://www.recordedfuture.com/products/api/ and speak with your RecordedFuture representative for more information.
         
        The list is obtained from recorded future API and needs a valid API TOKEN The large list contains all IP's with a risk score of 25 or more. If IP's are not present in the database a risk score of 0 is given.
         
        Module: intelmq.bots.experts.recordedfuture_iprisk.expert
         
        Parameters:
         
        database
         
        (required, string) Path to the local database file.
         
        api_token
         
        (required, string) This needs to contain valid API token to download the latest database data.
         
        overwrite
         
        (optional, boolean) Whether to overwrite existing fields. Defaults to false.
         
        Database
         
        Use this command to create/update the database and reload the bot:
         
        intelmq.bots.experts.recordedfuture_iprisk.expert --update-database\n
        "},{"location":"user/bots/#reverse-dns","title":"Reverse DNS","text":"
        For both source.ip and destination.ip the PTR record is fetched and the first valid result is used for source.reverse_dns or destination.reverse_dns.
         
        Module: intelmq.bots.experts.reverse_dns.expert
         
        Parameters (also expects cache parameters):
         
        cache_ttl_invalid_response
         
        (required, integer) The TTL for cached invalid responses.
         
        overwrite
         
        (optional, boolean) Whether to overwrite existing fields. Defaults to false.
        "},{"location":"user/bots/#rfc1918","title":"RFC1918","text":"
        Several RFCs define ASNs, IP Addresses and Hostnames (and TLDs) reserved for documentation. Events or fields of events can be dropped if they match the criteria of either being reserved for documentation (e.g. AS 64496, Domain example.com) or belonging to a local area network (e.g. 192.168.0.0/24). These checks can applied to URLs, IP Addresses, FQDNs and ASNs.
         
        It is configurable if the whole event should be dropped (\"policies\") or just the field removed, as well as which fields should be checked.
         
        Sources:
         
           
        • 1918
          •  
        • 2606
          •  
        • 3849
          •  
        • 4291
          •  
        • 5737
          •  
        • https://en.wikipedia.org/wiki/IPv4
          •  
        • https://en.wikipedia.org/wiki/Autonomous_system_(Internet)
          •  
         
        Module: intelmq.bots.experts.rfc1918.expert
         
        Parameters:
         
        fields
         
        (required, string) Comma-separated list of fields. Allowed values:
         
           
        • destination.asn & source.asn
          •  
        • destination.fqdn & source.fqdn
          •  
        • destination.ip & source.ip
          •  
        • destination.url & source.url
          •  
         
        policy
         
        (required, string) Comma-separated list of policies. Allowed values:
         
           
        • drop - the entire events is dropped
          •  
        • del - the affected field is removed
          •  
         
        With the example parameter values given above, this means that:
         
           
        • If a destination.ip value is part of a reserved network block, the field will be removed (policy del).
          •  
        • If a source.asn value is in the range of reserved AS numbers, the event will be removed altogether (policy drop).
          •  
        • If a source.url value contains a host with either an IP address part of a reserved network block, or a reserved   domain name (or with a reserved TLD), the event will be dropped (policy drop).
          •  
        "},{"location":"user/bots/#ripe","title":"RIPE","text":"
        Online RIPE Abuse Contact and Geolocation Finder for IP addresses and Autonomous Systems.
         
        Module: intelmq.bots.experts.ripe.expert
         
        Parameters (also expects cache parameters):
         
        mode
         
        (optional, string) Allowed values: append or replace. Defaults to append.
         
        query_ripe_db_asn
         
        (optional, boolean) Query for IPs at http://rest.db.ripe.net/abuse-contact/%s.json. Defaults to true.
         
        query_ripe_db_ip
         
        (optional, boolean) Query for ASNs at http://rest.db.ripe.net/abuse-contact/as%s.json. Defaults to true.
         
        query_ripe_stat_asn
         
        (optional, boolean) Query for ASNs at https://stat.ripe.net/data/abuse-contact-finder/data.json?resource=%s. Defaults to true.
         
        query_ripe_stat_ip
         
        (optional, boolean) Query for IPs at https://stat.ripe.net/data/abuse-contact-finder/data.json?resource=%s. Defaults to true.
         
        query_ripe_stat_geolocation
         
        (optional, boolean) Query for IPs at https://stat.ripe.net/data/maxmind-geo-lite/data.json?resource=%s. Defaults to true.
        "},{"location":"user/bots/#securitytxt","title":"SecurityTXT","text":"
        SecurityTXT is an initiative to standardize how websites publish their abuse contact information. It is standardized in RFC 9116 \"A File Format to Aid in Security Vulnerability Disclosure\". Refer to the linked document RFC for more information on security.txt. This bot looks for security.txt files on a URL or IP, retrieves the primary contact information out of it and adds this to the event.
         
        Requirements
         
        To use this bot, you need to install the required dependencies:
         
        pip3 install -r intelmq/bots/experts/securitytxt/REQUIREMENTS.txt\n
         
        Module: intelmq.bots.experts.securitytxt.expert
         
        Parameters
         
        url_field
         
        The field in the event that contains the URL/IP on which to look for the the security.txt file. Default: source.reverse_dns
         
        contact_field
         
        The field in the event in which to put the found contact details. Default: source.abuse_contact
         
        only_email_address (bool)
         
        Contact details can be web URLs or email addresses. When this value is set to True, it only selects email addresses as contact information. Default: true
         
        overwrite (bool)
         
        Boolean indicating whether to override existing data in contact_field. Default: true
         
        check_expired (bool)
         
        Boolean indicating whether to check if the security.txt has expired according to its own expiry date. Default: false
         
        check_canonical (bool)
         
        Boolean indicating whether to check if the url is contained in the list of canonical urls. Default: false
        "},{"location":"user/bots/#sieve","title":"Sieve","text":"
        This bot is used to filter and/or modify events based on a set of rules. The rules are specified in an external configuration file and with a syntax similar to the Sieve language used for mail filtering.
         
        Each rule defines a set of matching conditions on received events. Events can be matched based on keys and values in the event. Conditions can be combined using parenthesis and the boolean operators && and ||. If the processed event matches a rule's conditions, the corresponding actions are performed. Actions can specify whether the event should be kept or dropped in the pipeline (filtering actions) or if keys and values should be changed (modification actions).
         
        Requirements
         
        To use this bot, you need to install the required dependencies:
         
        pip3 install -r intelmq/bots/experts/sieve/REQUIREMENTS.txt\n
         
        Module: intelmq.bots.experts.sieve.expert
         
        Parameters:
         
        file
         
        (required, string) Path to sieve file. Syntax can be validated with intelmq_sieve_expert_validator.
         
        Examples
         
        The following excerpts illustrate some of the basic features of the sieve file format:
         
        if :exists source.fqdn {\n keep // aborts processing of subsequent rules and forwards the event.\n}\n\n\nif :notexists source.abuse_contact || source.abuse_contact =~ '.*@example.com' {\n drop // aborts processing of subsequent rules and drops the event.\n}\n\nif source.ip << '192.0.0.0/24' {\n add! comment = 'bogon' // sets the field comment to this value and overwrites existing values\n path 'other-path' // the message is sent to the given path\n}\n\nif classification.type :in ['phishing', 'malware-distribution'] && source.fqdn =~ '.*.(ch|li)$' {\n add! comment = 'domainabuse'\n keep\n} elif classification.type == 'scanner' {\n add! comment = 'ignore'\n drop\n} else {\n remove comment\n}\n
         
        Reference
         
        Sieve File Structure
         
        The sieve file contains an arbitrary number of rules of the form:
         
        if EXPRESSION {\n ACTIONS\n} elif EXPRESSION {\n ACTIONS\n} else {\n ACTIONS\n}\n
         
        Nested if-statements and mixed if statements and rules in the same scope are possible.
         
        Expressions
         
        Each rule specifies on or more expressions to match an event based on its keys and values. Event keys are specified as strings without quotes. String values must be enclosed in single quotes. Numeric values can be specified as integers or floats and are unquoted. IP addresses and network ranges (IPv4 and IPv6) are specified with quotes. List values for use with list/set operators are specified as string, float, int, bool and string literals separated by commas and enclosed in square brackets. Expression statements can be combined and chained using parentheses and the boolean operators && and ||. The following operators may be used to match events:
         
           
        • :exists and :notexists match if a given key exists, for example:
          •  
         
        if :exists source.fqdn { ... }\n
         
           
        • == and != match for equality of strings, numbers, and booleans, for example:
          •  
         
        if feed.name != 'acme-security' || feed.accuracy == 100 || extra.false_positive == false { ... }\n
         
           
        •  
          :contains matches on substrings (str.find).
           
          •  
        •  
          =~ matches strings based on the given regular expression. !~ is the inverse regular expression match.
           
          •  
        •  
          For :contains, =~ and !~, the value is converted to string before matching. If the value is a dict, convert the value to JSON.
           
          •  
        •  
          Numerical comparisons are evaluated with <, <=, >, >=.
           
          •  
        •  
          << matches if an IP address is contained in the specified network range:
           
          •  
         
        if source.ip << '10.0.0.0/8' { ... }\n
         
           
        • String values to match against can also be specified as lists of strings, which have separate operators. For example:
          •  
         
        if source.ip :in ['8.8.8.8', '8.8.4.4'] { ... }\n
         
        In this case, the event will match if it contains a key source.ip with either value 8.8.8.8 or 8.8.4.4.
         
        There are also :containsany to match at least one of a list of substrings, and :regexin to match at least one of a list of regular expressions, similar to the :contains and =~ operators.
         
           
        • Lists of numeric values support :in to check for inclusion in a list of numbers:
          •  
         
        if source.port :in [80, 443] { ... }\n
         
           
        • :equals tests for equality between lists, including order. Example for checking a hostname-port pair:
          •  
         
        if extra.host_tuple :equals ['dns.google', 53] { ... }\n
         
           
        • :setequals tests for set-based equality (ignoring duplicates and value order) between a list of given values.   Example for checking for the first nameserver of two domains, regardless of the order they are given in the list:
          •  
         
        if extra.hostnames :setequals ['ns1.example.com', 'ns1.example.mx'] { ... }\n
         
           
        • :overlaps tests if there is at least one element in common between the list specified by a key and a list of values.   Example for checking if at least one of the ICS, database or vulnerable tags is given:
          •  
         
        if extra.tags :overlaps ['ics', 'database', 'vulnerable'] { ... }\n
         
           
        • :subsetof tests if the list of values from the given key only contains values from a set of values specified as the   argument. Example for checking for a host that has only ns1.example.com and/or ns2.* as its apparent hostname:
          •  
         
        if extra.hostnames :subsetof ['ns1.example.com', 'ns2.example.com'] { ... }\n
         
           
        • :supersetof tests if the list of values from the given key is a superset of the values specified as the argument.   Example for matching hosts with at least the IoT and vulnerable tags:
          •  
         
        if extra.tags :supersetof ['iot', 'vulnerable'] { ... }\n
         
           
        • :before tests if the date value occurred before given time ago. The time might be absolute (basically anything parseable by pendulum parser, eg. \u201c2015-09-12T06:22:11+00:00\u201d) or relative (accepted string formatted like this \u201c \u201d, where epoch could be any of following strings (could optionally end with trailing \u2018s\u2019): hour, day, week, month, year) 
          if time.observation :before '1 week' { ... }\n
           
             
          • :after  tests if the date value occurred after given time ago; see :before
            •  
           
          if time.observation :after '2015-09-12' { ... }  # happened after midnight the 12th Sep\n
           
             
          • Boolean values can be matched with == or != followed by true or false. Example:
            •  
           
          if extra.has_known_vulns == true { ... }\n
           
             
          • The combination of multiple expressions can be done using parenthesis and boolean operators:
            •  
           
          if (source.ip == '127.0.0.1') && (comment == 'add field' || classification.taxonomy == 'vulnerable') { ... }\n
           
             
          • Any single expression or a parenthesised group of expressions can be negated using !:
            •  
           
          if ! source.ip :contains '127.0.0.' || ! ( source.ip == '172.16.0.5' && source.port == 25 ) { ... }\n
           
          !!! note Since 3.0.0, list-based operators are used on list values, such as foo :in [1, 2, 3] instead of foo == [1, 2, 3] and foo :regexin ['.mx', '.zz'] rather than foo =~ ['.mx', '.zz'], and similarly for :containsany vs :contains. Besides that, :notcontains has been removed, with e.g foo :notcontains ['.mx', '.zz'] now being represented using negation as ! foo :contains ['.mx', '.zz'].
           
          Actions
           
          If part of a rule matches the given conditions, the actions enclosed in { and } are applied. By default, all events that are matched or not matched by rules in the sieve file will be forwarded to the next bot in the pipeline, unless the drop action is applied.
           
             
          • add adds a key value pair to the event. It can be a string, number, or boolean. This action only applies if the key   is not yet defined in the event. If the key is already defined, the action is ignored. Example:
            •  
           
          add comment = 'hello, world'\n
           
          Some basic mathematical expressions are possible, but currently support only relative time specifications objects are supported. For example:
           
          add time.observation += '1 hour'\nadd time.observation -= '10 hours'\n
           
             
          •  
            add! same as above, but will force overwrite the key in the event.
             
            •  
          •  
            update modifies an existing value for a key. Only applies if the key is already defined. If the key is not defined   in the event, this action is ignored. This supports mathematical expressions like above. Example:
             
            •  
           
          update feed.accuracy = 50\n
           
          Some basic mathematical expressions are possible, but currently support only relative time specifications objects are supported. For example:
           
          update time.observation += '1 hour'\nupdate time.observation -= '10 hours'\n
           
             
          • remove removes a key/value from the event. Action is ignored if the key is not defined in the event. Example:
            •  
           
          remove extra.comments\n
           
             
          •  
            keep sends the message to the next bot in the pipeline (same as the default behaviour), and stops sieve rules   processing.
             
            •  
          •  
            path sets the path (named queue) the message should be sent to (implicitly or with the command keep. The named   queue needs to configured in the pipeline, see the User Guide for more information.
             
            •  
           
          path 'named-queue'\n
           
          You can as well set multiple destination paths with the same syntax as for value lists:
           
          path ['one', 'two']\n
           
          This will result in two identical message, one sent to the path one and the other sent to the path two.
           
          If the path is not configured, the error looks like:
           
          File \"/path/to/intelmq/intelmq/lib/pipeline.py\", line 353, in send for destination_queue in self.destination_queues path]: KeyError: 'one'\n
           
             
          • drop marks the event to be dropped. The event will not be forwarded to the next bot in the pipeline. The sieve file   processing is interrupted upon reaching this action. No other actions may be specified besides the drop action   within { and }.
            •  
           
          Comments
           
          Comments may be used in the sieve file: all characters after // and until the end of the line will be ignored.
          "},{"location":"user/bots/#splunk-saved-search-lookup","title":"Splunk Saved Search Lookup","text":"
          Runs a saved search in Splunk using fields in an event, adding fields from the search result into the event.
           
          Splunk documentation on saved searches: https://docs.splunk.com/Documentation/Splunk/latest/Report/Createandeditreports
           
          The saved search should take parameters according to the search_parameters configuration and deliver results according to result_fields. The examples above match a saved search of this format:
           
          index=\"dhcp\" ipv4address=\"$ip$\" | ... | fields _time username ether\n
           
          The time window used is the one saved with the search.
           
          Waits for Splunk to return an answer for each message, so slow searches will delay the entire botnet. If you anticipate a load of more than one search every few seconds, consider running multiple load-balanced copies of this bot.
           
          Module: intelmq.bots.experts.splunk_saved_search.expert
           
          Parameters (also expects HTTP parameters):
           
          auth_token
           
          (required, string) Splunk API authentication token.
           
          url
           
          (required, string) base URL of the Splunk REST API.
           
          retry_interval
           
          (optional, integer) Number of seconds to wait between polling for search results to be available. Defaults to 5.
           
          saved_search
           
          (required, string) Name of Splunk saved search to run.
           
          search_parameters
           
          (optional, object) Mapping of IntelMQ event fields containing the data to search for to parameters of the Splunk saved search. Defaults to {}. Example:
           
          search_parameters:\n  source.ip: ip\n
           
          result_fields
           
          (optional, object) Mapping of Splunk saved search result fields to IntelMQ event fields to store the results in. Defaults to {}. Example:
           
          result_fields:\n  username: source.account\n
           
          not_found
           
          (optional, array of strings) How to handle empty search results. Allowed values:
           
             
          • warn - log a warning message
            •  
          • send - send the event on unmodified
            •  
          • drop - drop the message
            •  
          • send - and drop are mutually exclusive
            •  
           
          All specified actions are performed. Defaults to [ \"warn\", \"send\" ].
           
          multiple_result_handling
           
          (optional, array of strings) How to handle more than one search result. Allowed values:
           
             
          • limit - limit the search so that duplicates are impossible
            •  
          • warn - log a warning message
            •  
          • use_first - use the first search result
            •  
          • ignore - do not modify the event
            •  
          • send - send the event on
            •  
          • drop - drop the message
            •  
          • limit cannot be combined with any other value
            •  
          • send and drop are mutually exclusive
            •  
          • ignore and use_first are mutually exclusive
            •  
           
          All specified actions are performed. Defaults to [\"warn\", \"use_first\", \"send\" ].
           
          overwrite
           
          (optional, boolean/null) Whether search results overwrite values already in the message or not. If null, attempting to add a field that already exists throws an exception. Defaults to null.
          "},{"location":"user/bots/#taxonomy","title":"Taxonomy","text":"
          This bot adds the classification.taxonomy field according to the RSIT taxonomy.
           
          Please note that there is a slight mismatch of IntelMQ's taxonomy to the upstream taxonomy. See also this issue.
           
          Information on the \"Reference Security Incident Taxonomy\" can be found here: https://github.com/enisaeu/Reference-Security-Incident-Taxonomy-Task-Force
           
          For brevity, \"type\" means classification.type and \"taxonomy\" means classification.taxonomy.
           
             
          • If taxonomy is missing, and type is given, the according taxonomy is set.
            •  
          • If neither taxonomy, not type is given, taxonomy is set to \"other\" and type to \"unknown\".
            •  
          • If taxonomy is given, but type is not, type is set to \"unknown\".
            •  
           
          Module: intelmq.bots.experts.taxonomy.expert
           
          No additional parameters.
          "},{"location":"user/bots/#threshold","title":"Threshold","text":"
          Check if the number of similar messages during a specified time interval exceeds a set value.
           
          Limitations
           
          This bot has certain limitations and is not a true threshold filter (yet). It works like this:
           
             
          1. Every incoming message is hashed according to the filter_* parameters.
            2.  
          2. The hash is looked up in the cache and the count is incremented by 1, and the TTL of the key is (re-)set to the    timeout.
            3.  
          3. If the new count matches the threshold exactly, the message is forwarded. Otherwise it is dropped.
            4.  
           
          Note
           
          Even if a message is sent, any further identical messages are dropped, if the time difference to the last message is less than the timeout! The counter is not reset if the threshold is reached.
           
          Module: intelmq.bots.experts.threshold.expert
           
          Parameters (also expects cache parameters):
           
          filter_keys
           
          (required, string/array of strings) Array or comma-separated list of field names to consider or ignore when determining which messages are similar.
           
          filter_type
           
          (required, string) Allowed values: whitelist or blacklist. When whitelist is used, only lines containing the text specified in filter_text option will be processed. When blacklist is used, only lines NOT containing the text will be processed.
           
          threshold
           
          (required, integer) Number of messages required before propagating one. In forwarded messages, the threshold is saved in the message as extra.count.
           
          add_keys
           
          (optional, object) List of keys and their respective values to add to the propagated messages. Example:
           
          add_keys:\n  classification.type: \"spam\"\n  comment: \"Started more than 10 SMTP connections\"\n
          "},{"location":"user/bots/#tor-exit-node","title":"Tor Exit Node","text":"
          This bot uses an offline database to determine whether the host is a Tor exit node.
           
          Module: intelmq.bots.experts.tor_nodes.expert
           
          Parameters:
           
          database
           
          (required, string) Path to the database file.
           
          Database
           
          Use this command to create/update the database and reload the bot:
           
          intelmq.bots.experts.tor_nodes.expert --update-database\n
          "},{"location":"user/bots/#trusted-introducer-lookup","title":"Trusted Introducer Lookup","text":"
          Lookups data from Trusted Introducer public teams list.
           
          Module: intelmq.bots.experts.trusted_introducer_lookup.expert
           
          Parameters:
           
          order
           
          (required, string) Allowed values: domain and asn. You can set multiple values, so first match wins.
           
             
          • When domain is set, it will lookup the source.fqdn field. It will go from high-order to low-order, i.e.   1337.super.example.com -> super.example.com -> example.com -> .com
            •  
          • If asn is set, it will lookup source.asn.
            •  
           
          After a match, the abuse contact will be fetched from the trusted introducer teams list and will be stored in the event as source.abuse_contact. If there is no match, the event will not be enriched and will be sent to the next configured step.
          "},{"location":"user/bots/#tuency","title":"Tuency","text":"
          Queries the IntelMQ API of a Tuency Contact Database instance.
           
          Tuency is a contact management database addressing the needs of CERTs. Users of tuency can configure contact addresses and delivery settings for IP objects (addresses, netblocks), Autonomous Systems, and (sub-)domains. This expert queries the information for source.ip and source.fqdn using the following other fields:
           
             
          • classification.taxonomy
            •  
          • classification.type
            •  
          • feed.provider
            •  
          • feed.name
            •  
           
          These fields therefore need to exist, otherwise the message is skipped.
           
          The API parameter \"feed_status\" is currently set to \"production\" constantly, until IntelMQ supports this field.
           
          The API answer is processed as following. For the notification interval:
           
             
          • If suppress is true, then extra.notify is set to false.
            •  
          • Otherwise:
            •  
          • If the interval is immediate, then extra.ttl is set to 0.
            •  
          • Otherwise the interval is converted into seconds and saved in   extra.ttl.
            •  
           
          For the contact lookup: For both fields ip and domain, the destinations objects are iterated and its email fields concatenated to a comma-separated list in source.abuse_contact.
           
          The IntelMQ fields used by this bot may change in the next IntelMQ release, as soon as better suited fields are available.
           
          Module: intelmq.bots.experts.tuency.expert
           
          Parameters:
           
          url
           
          (required, string) Tuency instance URL. Without the API path.
           
          authentication_token
           
          (required, string) The Bearer authentication token. Without the Bearer prefix.
           
          overwrite
           
          (optional, boolean) Whether the existing data in source.abuse_contact should be overwritten. Defaults to true.
          "},{"location":"user/bots/#truncate-by-delimiter","title":"Truncate By Delimiter","text":"
          Cut string if length is bigger than maximum length.
           
          Module: intelmq.bots.experts.truncate_by_delimiter.expert
           
          Parameters:
           
          delimiter
           
          (required, string) The delimiter to be used for truncating. Defaults to ..
           
          max_length
           
          (required, integer) The maximum string length.
           
          field
           
          (required, string) The field to be truncated, e.g. source.fqdn. The given field is truncated step-by-step using the delimiter from the beginning, until the field is shorter than max_length.
           
          Example: Cut through a long domain with a dot. The string is truncated until the domain does not exceed the configured maximum length.
           
             
          • Input domain (e.g. source.fqdn): www.subdomain.web.secondsubomain.test.domain.com
            •  
          • delimiter: .
            •  
          • max_length: 20
            •  
          • Resulting value test.domain.com (length: 15 characters)
            •  
          "},{"location":"user/bots/#url","title":"URL","text":"
          This bot extracts additional information from source.url and destination.url fields. It can fill the following fields:
           
             
          • source.fqdn
            •  
          • source.ip
            •  
          • source.port
            •  
          • source.urlpath
            •  
          • source.account
            •  
          • destination.fqdn
            •  
          • destination.ip
            •  
          • destination.port
            •  
          • destination.urlpath
            •  
          • destination.account
            •  
          • protocol.application
            •  
          • protocol.transport
            •  
           
          Module: intelmq.bots.experts.url.expert
           
          Parameters:
           
          overwrite
           
          (optional, boolean) Whether to overwrite existing fields. Defaults to false.
           
          skip_fields
           
          (optional, array of string) An array of field names that shouldn't be extracted from the URL.
          "},{"location":"user/bots/#url2fqdn","title":"Url2FQDN","text":"
          This bot is deprecated and will be removed in version 4.0. Use URL Expert bot instead.
           
          This bot extracts the Host from the source.url and destination.url fields and writes it to source.fqdn or destination.fqdn if it is a hostname, or source.ip or destination.ip if it is an IP address.
           
          Module: intelmq.bots.experts.url2fqdn.expert
           
          Parameters:
           
          overwrite
           
          (optional, boolean) Whether to overwrite existing fields. Defaults to false.
          "},{"location":"user/bots/#uwhoisd","title":"uWhoisd","text":"
          uWhoisd is a universal Whois server that supports caching and stores whois entries for historical purposes.
           
          The bot sends a request for source.url, source.fqdn, source.ip or source.asn to the configured uWhoisd instance and saves the retrieved whois entry:
           
             
          • If both source.url and source.fqdn are present, it will only do a request for source.fqdn, as the hostname of source.url should be the same as source.fqdn. The whois entry will be saved in extra.whois.fqdn.
            •  
          • If source.ip is present, the whois entry will be saved in extra.whois.ip.
            •  
          • If source.asn is present, he whois entry will be saved in extra.whois.asn.
            •  
           
          Events without source.url, source.fqdn, source.ip, or source.asn, are ignored.
           
          Note
           
          Requesting a whois entry for a fully qualified domain name (FQDN) only works if the request only contains the domain. uWhoisd will automatically strip the subdomain part if it is present in the request.
           
          Example: https://www.theguardian.co.uk
           
             
          • TLD: co.uk (uWhoisd uses the Mozilla public suffix list as a reference)
            •  
          • Domain: theguardian.co.uk
            •  
          • Subdomain: www
            •  
           
          The whois request will be for theguardian.co.uk
           
          Module: intelmq.bots.experts.uwhoisd.expert
           
          Parameters:
           
          server
           
          (optional, string) Hostname of the uWhoisd server. Defaults to localhost.
           
          port
           
          (optional, integer) Port of the uWhoisd server. Defaults to 4243.
          "},{"location":"user/bots/#wait","title":"Wait","text":"
          Waits for a some time or until a queue size is lower than a given number.
           
          Only one of the two modes is possible. If a queue name is given, the queue mode is active. If the sleep_time is a number, sleep mode is active. Otherwise the dummy mode is active, the events are just passed without an additional delay.
           
          Note that SIGHUPs and reloads interrupt the sleeping.
           
          Module: intelmq.bots.experts.wait.expert
           
          Parameters:
           
          queue_db
           
          (optional, integer) Database number of the database. Defaults to 2.
           
          queue_host
           
          (optional, string) Hostname of the database. Defaults to localhost.
           
          queue_name
           
          (optional, string) Name of the queue to be watched. This is not the name of a bot but the queue's name. Defaults to null.
           
          queue_password
           
          (optional, string) Password for the database. Defaults to null.
           
          queue_polling_interval
           
          (required, float) Interval to poll the list length in seconds. Defaults to ?.
           
          queue_port
           
          (optional, integer) Port of the database. Defaults to 6379.
           
          queue_size
           
          (optional, integer) Maximum size of the queue. Defaults to 0.
           
          sleep_time
           
          (optional, integer) Time to sleep before sending the event. Defaults to null.
          "},{"location":"user/bots/#output-bots","title":"Output Bots","text":""},{"location":"user/bots/#amqp-topic","title":"AMQP Topic","text":"
          Sends the event to a specified topic of an AMQP server
           
          Sends data to an AMQP Server See https://www.rabbitmq.com/tutorials/amqp-concepts.html for more details on amqp topic exchange.
           
          Requires the pika python library.
           
          Module: intelmq.bots.outputs.amqptopic.output
           
          Parameters:
           
          connection_attempts
           
          (optional, integer) The number of connection attempts to defined server. Defaults to 3.
           
          connection_heartbeat
           
          (optional, integer) Heartbeat to server (in seconds). Defaults to 3600.
           
          connection_host
           
          (optional, string) Hostname of the AMQP server. Defaults to 127.0.0.1.
           
          connection_port
           
          (optional, integer) Port of the AMQP server. Defaults to 5672.
           
          connection_vhost
           
          (optional, string) Virtual host to connect, on an http(s) connection would be http://IP/<your virtual host>.
           
          content_type
           
          (optional, string) Content type to deliver to AMQP server. Currently only supports application/json.
           
          delivery_mode
           
          (optional, integer) Allowed values:
           
             
          • 1 - Non-persistent delivery.
            •  
          • 2 - Persistent delivery. Messages are delivered to 'durable' queues and will be saved to disk.
            •  
           
          exchange_durable
           
          (optional, boolean) When set to true, the exchange will survive broker restart, otherwise will be a transient exchange.
           
          exchange_name
           
          (optional, string) The name of the exchange to use.
           
          exchange_type
           
          (optional, string) Type of the exchange, e.g. topic, fanout etc.
           
          keep_raw_field
           
          (optional, boolean) Whether to keep the raw field or not. Defaults to false.
           
          password
           
          (optional, boolean) Password for authentication on your AMQP server. Leave empty if authentication is not required.
           
          require_confirmation
           
          (optional, boolean) If set to True, an exception will be raised if a confirmation error is received.
           
          routing_key
           
          (required, string) The routing key for your amqptopic.
           
          single_key
           
          (optional, boolean) Only send the field instead of the full event (expecting a field name as string). Defaults to false.
           
          username
           
          (required, string) Username for authentication on your AMQP server.
           
          use_ssl
           
          (optional, boolean) Use ssl for the connection, make sure to also set the correct port, usually 5671. Defaults to false.
           
          message_hierarchical_output
           
          (optional, boolean) Convert the message to hierarchical JSON. Defaults to false.
           
          message_with_type
           
          (optional, boolean) Whether to include the type in the sent message. Defaults to false.
           
          message_jsondict_as_string
           
          (optional, boolean) Whether to convert JSON fields (extra) to string. Defaults to false.
           
          Examples of usage
           
             
          • Useful to send events to a RabbitMQ exchange topic to be further processed in other platforms.
            •  
           
          Confirmation
           
          If routing key or exchange name are invalid or non existent, the message is accepted by the server but we receive no confirmation. If parameter require_confirmation is True and no confirmation is received, an error is raised.
           
          Common errors
           
          Unroutable messages / Undefined destination queue
           
          The destination exchange and queue need to exist beforehand, with your preferred settings (e.g. durable, lazy queue. If the error message says that the message is \"unroutable\", the queue doesn't exist.
          "},{"location":"user/bots/#blackhole","title":"Blackhole","text":"
          This bot discards all incoming messages.
           
          Module: intelmq.bots.outputs.blackhole.output
           
          No additional parameters.
          "},{"location":"user/bots/#bro-file","title":"Bro File","text":"
          This bot outputs to BRO (zeek) file.
           
          File example:
           
          #fields indicator indicator_type meta.desc meta.cif_confidence meta.source xxx.xxx.xxx.xxx Intel::ADDR phishing 100 MISP XXX www.testdomain.com Intel::DOMAIN apt 85 CERT\n
           
          Module: intelmq.bots.outputs.bro_file.output
           
          No additional parameters.
          "},{"location":"user/bots/#cifv3-api","title":"CIFv3 API","text":"
          This bot outputs to a CIFv3 API instance and adds new indicator if not there already.
           
          By default, CIFv3 does an upsert check and will only insert entirely new indicators. Otherwise, upsert matches will have their count increased by 1. By default, the CIF3 output bot will batch indicators up to 500 at a time prior to doing a single bulk send. If the output bot doesn't receive a full 500 indicators within 5 seconds of the first received indicator, it will send what it has so far.
           
          CIFv3 should be able to process indicators as fast as IntelMQ can send them.
           
          Module: intelmq.bots.outputs.cif3.output
           
          Parameters:
           
          add_feed_provider_as_tag
           
          (required, boolean) Use false when in doubt.
           
          cif3_additional_tags
           
          (required, array of strings) An array of tags to set on submitted indicator(s).
           
          cif3_feed_confidence
           
          (required, float) Used when mapping a feed's confidence fails or if static confidence parameter is true.
           
          cif3_static_confidence
           
          (required, boolean) Whether to always use cif3_feed_confidence value as confidence rather than dynamically interpret feed value (use false when in doubt).
           
          cif3_token
           
          (required, string) Token key for accessing CIFv3 API.
           
          cif3_url
           
          (required, string) URL of the CIFv3 instance.
           
          fireball
           
          (required, integer) Used to batch events before submitting to a CIFv3 instance, use 0 to disable batch and send each event as received. Defaults to 500.
           
          http_verify_cert
           
          (optional, boolean) Verify the TLS certificate of the server. Defaults to true.
          "},{"location":"user/bots/#elasticsearch","title":"Elasticsearch","text":"
          This bot outputs to Elasticsearch.
           
          Module: intelmq.bots.outputs.elasticsearch.output
           
             
          • lookup: yes
            •  
          • public: yes
            •  
          • cache: no
            •  
          • description: Output Bot that sends events to Elasticsearch
            •  
           
          Only ElasticSearch version 7 supported.
           
          It is also possible to feed data into ElasticSearch using ELK-Stack via Redis and Logstash, see ELK-Stack {.interpreted-text role=\"doc\"} for more information. This methods supports various different versions of ElasticSearch.
           
          Parameters:
           
          elastic_host
           
          (optional, string) Name/IP for the Elasticsearch server. Defaults to 127.0.0.1.
           
          elastic_port
           
          (optional, int) Port for the Elasticsearch server. Defaults to 9200.
           
          elastic_index
           
          (optional, string) Index for the Elasticsearch output. Defaults to intelmq.
           
          rotate_index
           
          (optional, string) Allowed values: never, daily, weekly, monthly or yearly. If set, will index events using the date information associated with the event. Defaults to never.
           
          Using 'intelmq' as the elastic_index, the following are examples of the generated index names:
           
          'never' --> intelmq\n'daily' --> intelmq-2018-02-02\n'weekly' --> intelmq-2018-42\n'monthly' --> intelmq-2018-02\n'yearly' --> intelmq-2018\n
           
          http_username
           
          (optional, string) HTTP basic authentication username.
           
          http_password
           
          (optional, string) HTTP basic authentication password.
           
          use_ssl
           
          (optional, boolean) Whether to use SSL/TLS when connecting to Elasticsearch. Defaults to false.
           
          http_verify_cert
           
          (optional, boolean) Whether to require verification of the server's certificate. Defaults to false.
           
          ssl_ca_certificate
           
          (optional, string) Path to trusted CA certificate.
           
          ssl_show_warnings
           
          (optional, boolean) Whether to show warnings if the server's certificate cannot be verified. Defaults to true.
           
          replacement_char
           
          (optional, string) If set, dots ('.') in field names will be replaced with this character prior to indexing. This is for backward compatibility with ES 2.X. Defaults to null. Recommended for Elasticsearch 2.X: _
           
          flatten_fields
           
          (optional, array of strings) In ES, some query and aggregations work better if the fields are flat and not JSON. Here you can provide a list of fields to convert. Defaults to ['extra'].
           
          Can be a list of strings (fieldnames) or a string with field names separated by a comma (,). eg extra,field2 or ['extra', 'field2'].
           
          See contrib/elasticsearch/elasticmapper for a utility for creating Elasticsearch mappings and templates.
           
          If using rotate_index, the resulting index name will be of the form elastic_index-event date. To query all intelmq indices at once, use an alias (https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-aliases.html), or a multi-index query.
           
          The data in ES can be retrieved with the HTTP-Interface:
           
           curl -XGET 'http://localhost:9200/intelmq/events/_search?pretty=True'\n
          "},{"location":"user/bots/#file_1","title":"File","text":"
          This bot outputs messages (reports or events) to a file.
           
          Multihreading is disabled for this bot, as this would lead to corrupted files.
           
          Module: intelmq.bots.outputs.file.output
           
          Parameters:
           
          encoding_errors_mode
           
          (optional, string) See for more details and options: https://docs.python.org/3/library/functions.html#open For example with backslashreplace all characters which cannot be properly encoded will be written escaped with backslashes. Defaults to strict.
           
          file
           
          (optional, string) Path to the output file. Missing directories will be created if possible with the mode 755. Defaults to /opt/intelmq/var/lib/bots/file-output/events.txt.
           
          format_filename
           
          (optional, boolean) Whether the file name should be formatted. Defaults to false.
           
          Uses Python formatted strings. See: https://docs.python.org/3/library/string.html#formatstrings
           
          Example:
           
             
          • The filename .../{event[source.abuse_contact]}.txt will be (for example) .../abuse@example.com.txt.
            •  
          • .../{event[time.source]:%Y-%m-%d} results in the date of the event used as filename.
            •  
           
          If the field used in the format string is not defined, None will be used as fallback.
           
          hierarchical_output
           
          (optional, boolean) Whether the resulting dictionary should be hierarchical (field names split by a dot). Defaults to false.
           
          single_key
           
          (optional, string) Output only a single specified key. In case of raw key the data is base64 decoded. Defaults to null (output the whole message).
          "},{"location":"user/bots/#files","title":"Files","text":"
          This bot outputs each message to a separate file.
           
          Module: intelmq.bots.outputs.files.output
           
          Parameters:
           
          dir
           
          (optional, string) Path to the output directory. Defaults to /opt/intelmq/var/lib/bots/files-output/incoming.
           
          tmp
           
          (optional, string) Temporary directory to use (must reside on the same filesystem as dir). Defaults to /opt/intelmq/var/lib/bots/files-output/tmp.
           
          suffix
           
          (optional, strings) Extension of created files. Defaults to .json.
           
          hierarchical_output
           
          (optional, boolean) Whether the resulting dictionary should be hierarchical (field names split by a dot). Defaults to false.
           
          single_key
           
          (optional, string) Output only a single specified key. In case of raw key the data is base64 decoded. Defaults to null (output the whole message).
          "},{"location":"user/bots/#mcafee-enterprise-security-manager","title":"McAfee Enterprise Security Manager","text":"
          This bot outputs messages to McAfee Enterprise Security Manager watchlist.
           
          Module: intelmq.bots.outputs.mcafee.output_esm_ip
           
          Parameters:
           
             
          • Feed parameters (see above)
            •  
           
          esm_ip
           
          (optional, string) Hostname of the ESM server. Defaults to 1.2.3.4.
           
          esm_user
           
          (optional, string) Username of user entitled to write to watchlist. Defaults to NGCP.
           
          esm_pw
           
          (required, string) Password of user entitled to write to watchlist.
           
          esm_watchlist
           
          (required, string) Name of the watchlist to write to.
           
          field
           
          (optional, string) Name of the IntelMQ field to be written to ESM. Defaults to source.ip.
          "},{"location":"user/bots/#misp-feed","title":"MISP Feed","text":"
          Create a directory layout in the MISP Feed format.
           
          The PyMISP library >= 2.4.119.1 is required, see REQUIREMENTS.txt.
           
          Module: intelmq.bots.outputs.misp.output_feed
           
          Parameters:
           
             
          • Feed parameters (see above)
            •  
           
          misp_org_name
           
          () Org name which creates the event, string
           
          misp_org_uuid
           
          () Org UUID which creates the event, string
           
          output_dir
           
          () Output directory path, e.g. [/opt/intelmq/var/lib/bots/mispfeed-output]. Will be created if it does not exist and possible.
           
          interval_event
           
          () The output bot creates one event per each interval, all data in this time frame is part of this event. Default \"1 hour\", string.
           
          Usage in MISP
           
          Configure the destination directory of this feed as feed in MISP, either as local location, or served via a web server. See the MISP documentation on Feeds for more information
          "},{"location":"user/bots/#misp-api","title":"MISP API","text":"
          Module: intelmq.bots.outputs.misp.output_api
           
          Connect to a MISP instance and add event as MISPObject if not there already.
           
          The PyMISP library >= 2.4.120 is required, see REQUIREMENTS.txt.
           
          Parameters:
           
             
          • Feed parameters (see above)
            •  
           
          add_feed_provider_as_tag
           
          () boolean (use [true] when in doubt)
           
          add_feed_name_as_tag
           
          () boolean (use [true] when in doubt)
           
          misp_additional_correlation_fields
           
          () list of fields for which the correlation flags will be enabled (in addition to those which are in significant_fields)
           
          misp_additional_tags
           
          () list of tags to set not be searched for when looking for duplicates
           
          misp_key
           
          () string, API key for accessing MISP
           
          misp_publish
           
          () boolean, if a new MISP event should be set to \"publish\".
           
          Expert setting as MISP may really make it \"public\"! (Use [false] when in doubt.)
           
          misp_tag_for_bot
           
          () string, used to mark MISP events
           
          misp_to_ids_fields
           
          () list of fields for which the [to_ids] flags will be set
           
          misp_url
           
          () string, URL of the MISP server
           
          significant_fields
           
          () list of intelmq field names
           
          The significant_fields values will be searched for in all MISP attribute values and if all values are found in the same MISP event, no new MISP event will be created. Instead if the existing MISP events have the same feed.provider and match closely, their timestamp will be updated.
           
          If a new MISP event is inserted the significant_fields and the misp_additional_correlation_fields will be the attributes where correlation is enabled.
           
          Make sure to build the IntelMQ Botnet in a way the rate of incoming events is what MISP can handle, as IntelMQ can process many more events faster than MISP (which is by design as MISP is for manual handling). Also remove the fields of the IntelMQ events with an expert bot that you do not want to be inserted into MISP.
           
          (More details can be found in the docstring of output_api.py.
          "},{"location":"user/bots/#mongodb","title":"MongoDB","text":"
          MongoDB is the bot responsible to send events to a MongoDB database
           
          Saves events in a MongoDB either as hierarchical structure or flat with full key names. time.observation and time.source are saved as datetime objects, not as ISO formatted string.
           
          Module: intelmq.bots.outputs.mongodb.output
           
          Requirements
           
          pip3 install pymongo>=2.7.1\n
           
          The bot has been tested with pymongo versions 2.7.1, 3.4 and 3.10.1 (server versions 2.6.10 and 3.6.8).
           
          Parameters:
           
          host
           
          (optional, string) Hostname of the MongoDB server. Defaults to localhost.
           
          port
           
          (optional, integer) Port of the MongoDB server. Defaults to 27017.
           
          database
           
          (required, string) Name of the MongoDB database to use.
           
          db_user
           
          (optional, string) User that should be used if authentication is required.
           
          db_pass
           
          (optional, string) Password.
           
          collection
           
          (required, string) Name of the MongoDB collection to use.
           
          hierarchical_output
           
          (optional, boolean) MongoDB does not allow saving keys with dots, we split the dictionary in sub-dictionaries. Defaults to true.
           
          replacement_char
           
          (optional, string) Replacement character for replacing the dots in key names if hierarchical output is not used. Defaults to _.
          "},{"location":"user/bots/#redis","title":"Redis","text":"
          This bot outputs events to a remote Redis server/queue.
           
          Examples of usage
           
             
          • Can be used to send events to be processed in another system. E.g.: send events to Logstash.
            •  
          • In a multi tenant installation can be used to send events to external/remote IntelMQ instance. Any expert bot queue   can receive the events.
            •  
          • In a complex configuration can be used to create logical sets in IntelMQ-Manager.
            •  
           
          Module: intelmq.bots.outputs.redis.output
           
          Parameters:
           
          redis_server_ip
           
          (optional, string) Hostname of the Redis server. Defaults to 127.0.0.1.
           
          redis_server_port
           
          (optional, integer) Port of the Redis server. Defaults to 6379.
           
          redis_db
           
          (optional, integer) Redis database number. Defaults to 2.
           
          redis_password
           
          (optional, string) Redis server password. Defaults to null.
           
          redis_queue
           
          (required, string) Redis queue name (such as remote-server-queue).
           
          redis_timeout
           
          (optional, integer) Connection timeout, in milliseconds. Defaults to 5000.
           
          hierarchical_output
           
          (optional, boolean) Whether the resulting dictionary should be hierarchical (field names split by a dot). Defaults to false.
           
          with_type
           
          (optional, boolean) Whether to include __type field. Defaults to true.
          "},{"location":"user/bots/#request-tracker_1","title":"Request Tracker","text":"
          Output Bot that creates Request Tracker tickets from events.
           
          Module: intelmq.bots.outputs.rt.output
           
          Description
           
          The bot creates tickets in Request Tracker and uses event fields for the ticket body text. The bot follows the workflow of the RTIR:
           
             
          • create ticket in Incidents queue (or any other queue)
            •  
          • all event fields are included in the ticket body,
            •  
          • event attributes are assigned to tickets' CFs according to the attribute mapping,
            •  
          • ticket taxonomy can be assigned according to the CF mapping. If you use taxonomy different   from ENISA RSIT, consider using some   extra attribute field and do value mapping with modify or sieve bot,
            •  
          • create linked ticket in Investigations queue, if these conditions are met
            •  
          • if first ticket destination was Incidents queue,
            •  
          • if there is source.abuse_contact is specified,
            •  
          • if description text is specified in the field appointed by configuration,
            •  
          • RT/RTIR supposed to do relevant notifications by script working on condition \"On Create\",
            •  
          • configuration option investigation_fields specifies which event fields has to be included in the investigation,
            •  
          • Resolve Incident ticket, according to configuration (Investigation ticket status should depend on RT script   configuration),
            •  
           
          Take extra caution not to flood your ticketing system with enormous amount of tickets. Add extra filtering for that to pass only critical events to the RT, and/or deduplicating events.
           
          Parameters:
           
          rt_uri
           
          ()
           
          rt_user
           
          ()
           
          rt_password
           
          ()
           
          verify_cert
           
          () RT API endpoint connection details, string.
           
          queue
           
          () ticket destination queue. If set to 'Incidents', 'Investigations' ticket will be created if create_investigation is set to true, string.
           
          CF_mapping
           
          (optional, object) Mapping event fields to ticket CFs. Defaults to:
           
          classification.taxonomy: Classification\nclassification.type: Incident Type\nevent_description.text: Description\nextra.incident.importance: Importance\nextra.incident.severity: Incident Severity\nextra.organization.name: Customer\nsource.ip: IP\n
           
          final_status
           
          (optional, string) The final status for the created ticket. Defaults to resolved. The linked Investigation ticket will be resolved automatically by RTIR scripts.
           
          create_investigation
           
          (optional, boolean) Whether an Investigation ticket should be created (in case of RTIR workflow). Defaults to false.
           
          investigation_fields
           
          (optional, string) Comma-separated string of attributes to include in an Investigation ticket. Defaults to time.source,source.ip,source.port,source.fqdn,source.url,classification.taxonomy,classification.type,classification.identifier,event_description.url,event_description.text,malware.name,protocol.application,protocol.transport.
           
          description_attr
           
          (optional, string) Event field to be used as a text message being sent to the recipient. If it is not specified or not found in the event, the Investigation ticket is not going to be created. Defaults to event_decription.text.
          "},{"location":"user/bots/#rest-api","title":"REST API","text":"
          REST API is the bot responsible to send events to a REST API listener through POST.
           
          Module: intelmq.bots.outputs.restapi.output
           
          Parameters:
           
          host
           
          (required, host) Destination URL of the POST request.
           
          auth_type
           
          (required, string) Allowed values: http_basic_auth or http_header. Type of authentication to use.
           
          auth_token
           
          (required, string) Username or HTTP header key.
           
          auth_token_name
           
          (required, string) Password or HTTP header value.
           
          hierarchical_output
           
          (optional, boolean) Whether the resulting dictionary should be hierarchical (field names split by a dot). Defaults to false.
           
          use_json
           
          (optional, boolean) Whether to use JSON. Defaults to true.
          "},{"location":"user/bots/#rpz-file","title":"RPZ File","text":"
          This bot outputs events into DNS RPZ blocklist file used for \"DNS firewall\".
           
          The prime motivation for creating this feature was to protect users from badness on the Internet related to known-malicious global identifiers such as host names, domain names, IP addresses, or nameservers. More information: https://dnsrpz.info
           
          Example: 
          $TTL 3600 @ SOA rpz.yourdomain.eu. hostmaster.rpz.yourdomain.eu. 2105260601 60 60 432000 60 NS localhost. ; ;\nyourdomain.eu. CERT.XX Response Policy Zones (RPZ) ; Last updated: 2021-05-26 06:01:41 (UTC) ; ; Terms Of\nUse: https://rpz.yourdomain.eu ; For questions please contact rpz [at] yourdomain.eu ; *.maliciousdomain.com CNAME\nrpz.yourdomain.eu. *.secondmaliciousdomain.com CNAME rpz.yourdomain.eu.\n
           
          Module: intelmq.bots.outputs.rpz_file.output
           
          Parameters:
           
          cname
           
          (optional, string) example rpz.yourdomain.eu
           
          organization_name
           
          (optional, string) Your organisation name
           
          rpz_domain
           
          (optional, string) Information website about RPZ
           
          hostmaster_rpz_domain
           
          () Technical website
           
          rpz_email
           
          () Contact email
           
          ttl
           
          () Time to live
           
          ncachttl
           
          () DNS negative cache
           
          serial
           
          () Time stamp or another numbering
           
          refresh
           
          () Refresh time
           
          retry
           
          () Retry time
           
          expire
           
          () Expiration time
           
          test_domain
           
          () For test domain, it's added in first rpz file (after header)
          "},{"location":"user/bots/#smtp-batch","title":"SMTP Batch","text":"
          Aggregate events by e-mail addresses in the source.abuse_contact field and batch send them at once as a zipped CSV file attachment in a GPG signed message.
           
          When the bot is run normally by IntelMQ, it just aggregates the events for later use into a custom Redis database. If run through CLI (by a cron or manually), it shows e-mail messages that are ready to be sent and let you send them to the tester's e-mail OR to abuse contact e-mails. E-mails are sent in a zipped CSV file, delimited by a comma, while keeping strings in double quotes. Note: The field \"raw\" gets base64 decoded if possible. Bytes \\n and \\r are replaced with \"\\n\" and \"\\r\" strings in order to guarantee best CSV files readability both in Microsoft Office and LibreOffice. (A multiline string may be stored in \"raw\" which completely confused Microsoft Excel.)
           
          Launch it like this: 
          </usr/local/bin executable> <bot-id> --cli [--tester tester's email]\n
           Example: 
          intelmq.bots.outputs.smtp_batch.output smtp-batch-output --cli --tester your-email@example.com\n
           
          CLI flags: 
          -h, --help            show this help message and exit\n--cli                 initiate CLI interface\n--tester TESTING_TO   tester's e-mail\n--ignore-older-than-days IGNORE_OLDER_THAN_DAYS\n                      1..n skip all events with time.observation older than 1..n day; 0 disabled (allow all)\n--gpg-key GPG_KEY     fingerprint of gpg key to be used\n--limit-results LIMIT_RESULTS\n                      Just send first N mails.\n--send                Sends now, without dialog.\n
           
          You can schedule the batch sending easily with a cron script, I.E. put this into crontab -e of the intelmq user:
           
          # Send the e-mails every day at 6 AM\n0 6 * * *  /usr/local/bin/intelmq.bots.outputs.smtp_batch.output smtp-batch-output-cz cli --ignore-older-than-days 4 --send &> /tmp/intelmq-send.log\n
           
          Module: intelmq.bots.outputs.smtp_batch.output
           
          Parameters:
           
          alternative_mails
           
          (optional, string) Path to CSV in the form original@email.com,alternative@email.com. Needed when some of the recipients ask you to forward their e-mails to another address. Delimit multiple recipients by the semicolon. The field is internally parsed by Envelope so pretty anything is allowed:
           
          original@email.com,alternative@email.com\noriginal2@email.com,person1@email.com;person2@email.com\noriginal3@email.com, Mary <person1@example.com>; John <person2@example.com>\n
           
          attachment_name
           
          (optional, string)  Attachment file name for the outgoing messages. May contain date formatting like this %Y-%m-%d. Example: \"events_%Y-%m-%d\" will appear as \"events_2022-12-01.zip\". Defaults to \"intelmq_%Y-%m-%d\".
           
          bcc
           
          (optional, array of strings) An array of e-mails to be put in the Bcc field for every mail.
           
          email_from
           
          (required, string) Sender's e-mail of the outgoing messages.
           
          gpg_key
           
          (optional, string) The Key or the fingerprint of a GPG key stored in ~/.gnupg keyring folder.
           
          gpg_pass
           
          (optional, string) Password for the GPG key if needed.
           
          mail_template
           
          (required, string) Path to the file containing the body of the mail for the outgoing messages.
           
          ignore_older_than_days
           
          (optional, integer) Skips events with time.observation older than now-N. (If your queue gets stuck for a reason, you do not want to send old and probably already solved events.) Defaults to 0 (allow all).
           
          limit_results
           
          (optional, integer) Intended as a debugging option, allows loading just first N e-mails from the queue.
           
          redis_cache_db
           
          (required, integer)  Redis database used for event aggregation. As the databases < 10 are reserved for the IntelMQ core, recommended is a bigger number.
           
          redis_cache_host
           
          (required, string) Hostname of the Redis database.
           
          redis_cache_port
           
          (required, string)  Port of the Redis database.
           
          redis_cache_ttl
           
          (required, integer) TTL in seconds used for caching. Recommended 1728000 for 20 days.
           
          smtp_server
           
          (required, string/array/object) SMTP server information and credentials. See SMTP parameter of the envelope module.
           
          Examples: 
          smtp_server: \"mailer\"\nsmtp_server: {\"host\": \"mailer\", \"port\": 587, \"user\": \"john\", \"password\": \"123\"}\nsmtp_server: [\"mailer\", 587, \"john\", \"password\"]\n
           
          subject
           
          (required, string) Subject for the outgoing messages. May contain date formatting like this %Y-%m-%d. Example: \"IntelMQ weekly warning (%d.%m.%Y)\".
           
          testing_to
           
          (optional, string) Tester's e-mail.
          "},{"location":"user/bots/#smtp","title":"SMTP","text":"
          Sends a MIME Multipart message containing the text and the event as CSV for every single event.
           
          Module: intelmq.bots.outputs.smtp.output
           
          Parameters:
           
          fieldnames
           
          (optional, string/array of strings) Array of field names (or comma-separated list) to be included in the email. If empty, no attachment is sent - this can be useful if the actual data is already in the body (parameter text) or the subject.
           
          mail_from
           
          (optional, string) Sender's e-email address. Defaults to cert@localhost.
           
          mail_to
           
          (required, string) Comma-separated string of recipient email addresses. Supports formatting.
           
          smtp_host
           
          (optional, string) Hostname of the SMTP server. Defaults to localhost.
           
          smtp_password
           
          (optional, string) Password for authentication to your SMTP server. Defaults to null.
           
          smtp_port
           
          (optional, integer) Port of the SMTP server. Defaults to 25.
           
          smtp_username
           
          (optional, string) Username for authentication to your SMTP server. Defaults to null.
           
          fail_on_errors
           
          (optional, boolean) Whether any error should cause the bot to fail (raise an exception) or otherwise rollback. If false, the bot eventually waits and re-try (e.g. re-connect) etc. to solve the issue. If true, the bot raises an exception and - depending on the IntelMQ error handling configuration - stops. Defaults to false.
           
          ssl
           
          (optional, boolean) Defaults to false.
           
          starttls
           
          (optional, boolean) Defaults to true.
           
          subject
           
          (optional, string) Subject of the e-mail message. Supports formatting. Defaults to Incident in your AS {ev[source.asn]}.
           
          text
           
          (optional, string) Body of the e-mail message. Supports formatting. Defaults to 
          Dear network owner,\n\nWe have been informed that the following device might have security problems.\n\nYour localhost CERT\n
           
          For several strings you can use values from the string using the standard Python string format syntax. Access the event's values with {ev[source.ip]} and similar. Any not existing fields will result in None. For example, to set the recipient(s) to the value given in the event's source.abuse_contact field, use this as mail_to parameter: {ev[source.abuse_contact]}
           
          Authentication is optional. If both username and password are given, these mechanism are tried: CRAM-MD5, PLAIN, and LOGIN.
           
          Client certificates are not supported. If http_verify_cert is true, TLS certificates are checked.
          "},{"location":"user/bots/#sql","title":"SQL","text":"
          SQL is the bot responsible to send events to a PostgreSQL, SQLite, or MSSQL Database.
           
          Note
           
          When activating autocommit, transactions are not used. See: http://initd.org/psycopg/docs/connection.html#connection.autocommit
           
          Module: intelmq.bots.outputs.sql.output
           
          Parameters:
           
          The parameters marked with 'PostgreSQL' will be sent to libpq via psycopg2. Check the libpq parameter documentation for the versions you are using.
           
          autocommit
           
          (optional, boolean) Psycopg's autocommit mode. Defaults to true.
           
          engine
           
          (required, string) Allowed values are postgresql, sqlite, or mssql.
           
          database
           
          (optional, string) Database name or SQLite database file. Defaults to intelmq-events.
           
          host
           
          (optional, string) Hostname of the database server. Defaults to localhost.
           
          jsondict_as_string
           
          (optional, boolean) Whether to save JSON fields as JSON string. Defaults to true.
           
          message_jsondict_as_string
           
          (optional, boolean) Whether to save JSON fields as JSON string. Defaults to true.
           
          port
           
          (optional, integer) Port of the database server. Defaults to 5432.
           
          user
           
          (optional, string) Username for connecting to the database system. Defaults to intelmq.
           
          password
           
          (optional, string) Password for connecting to the database system. Defaults to null.
           
          sslmode
           
          (optional, string) Database sslmode, Allowed values: disable, allow, prefer, require, verify-ca or verify-full. See: https://www.postgresql.org/docs/current/static/images/libpq-connect.html#libpq-connect-sslmode. Defaults to require.
           
          table
           
          (optional, string) Name of the database table to use. Defaults to events.
           
          fields
           
          (optional, array) Array of event fields to output to the database. Defaults to null (use all fields).
           
          reconnect_delay
           
          (optional, integer) Number of seconds to wait before reconnecting in case of an error. Defaults to 0.
           
          fail_on_errors
           
          (optional, boolean) Whether an error should cause the bot to fail (raise an exception) or otherwise rollback. If false, the bot eventually waits and re-try (e.g. re-connect) etc. to solve the issue. If true, the bot raises an exception and - depending on the IntelMQ error handling configuration - stops. Defaults to false.
          "},{"location":"user/bots/#stomp_1","title":"STOMP","text":"
          This bot pushes data to any STOMP stream. STOMP stands for Streaming Text Oriented Messaging Protocol. See: https://en.wikipedia.org/wiki/Streaming_Text_Oriented_Messaging_Protocol
           
          Module: intelmq.bots.outputs.stomp.output
           
          Requirements
           
          Install the stomp.py library from PyPI:
           
          pip3 install -r intelmq/bots/outputs/stomp/REQUIREMENTS.txt\n
           
          Alternatively, you may want to install it using your OS's native packaging tools, e.g.:
           
          apt install python3-stomp\n
           
          Apart from that, depending on what STOMP server you connect to, you may need to obtain, from the organization or company owning the server, one or more of the following security/authentication-related resources:
           
             
          • CA certificate file;
            •  
          • either: client certificate and client certificate's key files,   or: username (STOMP login) and password (STOMP passcode).
            •  
           
          Also, you will need to know an appropriate STOMP destination (aka exchange point), e.g. /exchange/_push.
           
          Parameters:
           
          server
           
          (optional, string) STOMP server's hostname or IP, e.g. \"n6stream.cert.pl\" or \"127.0.0.1\" (which is default)
           
          port
           
          (optional, integer) STOMP server's port number (default: 61614)
           
          exchange
           
          (optional, string) STOMP destination to push at, e.g. \"/exchange/_push\" (which is default)
           
          heartbeat
           
          (optional, integer) Defaults to 60000.
           
          ssl_ca_certificate
           
          (optional, string) path to CA file, or empty string to load system's default CA certificates
           
          auth_by_ssl_client_certificate
           
          (optional, boolean) default: true (note: false is needed for new n6 auth)
           
          ssl_client_certificate
           
          (optional, string) Path to client certificate to use for TLS connections.
           
          ssl_client_certificate_key
           
          (optional, string) Path to client private key to use for TLS connections.
           
          username
           
          (optional, string) STOMP login (e.g., n6 user login), used only if auth_by_ssl_client_certificate is false
           
          password
           
          (optional, string) STOMP passcode (e.g., n6 user API key), used only if auth_by_ssl_client_certificate is false
           
          message_hierarchical_output
           
          (optional, boolean) Defaults to false.
           
          message_jsondict_as_string
           
          (optional, boolean) Defaults to false.
           
          message_with_type
           
          (optional, boolean) Defaults to false.
           
          single_key
           
          (optional, string) Output only a single specified key. In case of raw key the data is base64 decoded. Defaults to null (output the whole message).
          "},{"location":"user/bots/#tcp_1","title":"TCP","text":"
          TCP is the bot responsible to send events to a TCP port (Splunk, another IntelMQ, etc..).
           
          Multihreading is disabled for this bot.
           
          Sending to an IntelMQ TCP collector
           
          If you intend to link two IntelMQ instance via TCP, set the parameter counterpart_is_intelmq to true. The bot then awaits an \"Ok\" message to be received after each message is sent. The TCP collector just sends \"Ok\" after every message it gets.
           
          Module: intelmq.bots.outputs.tcp.output
           
          Parameters:
           
          counterpart_is_intelmq
           
          (optional, boolean) Whether the receiver is an IntelMQ TCP collector bot. Defaults to true.
           
          ip
           
          (required, string) Hostname of the destination server.
           
          hierarchical_output
           
          (optional, boolean) True for a nested JSON, false for a flat JSON (when sending to a TCP collector).
           
          port
           
          (required, integer) Port of destination server.
           
          separator
           
          (optional, string) Separator of messages, e.g. \"n\", optional. When sending to a TCP collector, parameter shouldn't be present. In that case, the output waits every message is acknowledged by \"Ok\" message the TCP collector bot implements.
          "},{"location":"user/bots/#templated-smtp","title":"Templated SMTP","text":"
          Sends a MIME Multipart message built from an event and static text using Jinja2 templates.
           
          See the Jinja2 documentation at https://jinja.palletsprojects.com/.
           
          Authentication is attempted only if both username and password are specified.
           
          Templates are in Jinja2 format with the event provided in the variable event. E.g.:
           
          mail_to: \"{{ event['source.abuse_contact'] }}\"\n
           
          As an extension to the Jinja2 environment, the function from_json is available for parsing JSON strings into Python structures. This is useful if you want to handle complicated structures in the output field of an event. In that case, you would start your template with a line like:
           
          {%- set output = from_json(event['output']) %}\n
           
          and can then use output as a regular Python object in the rest of the template.
           
          Attachments are templated strings, especially useful for sending structured data. E.g. to send a JSON document including malware.name and all other fields starting with source.:
           
          attachments:\n  - content-type: application/json\n    text: |\n      {\n        \"malware\": \"{{ event['malware.name'] }}\",\n        {%- set comma = joiner(\", \") %}\n        {%- for key in event %}\n           {%- if key.startswith('source.') %}\n        {{ comma() }}\"{{ key }}\": \"{{ event[key] }}\"\n           {%- endif %}\n        {%- endfor %}\n      }\n    name: report.json\n
           
          You are responsible for making sure that the text produced by the template is valid according to the content-type.
           
          If you are migrating from the SMTP output bot that produced CSV format attachments, use the following configuration to produce a matching format:
           
          attachments:\n  - content-type: text/csv\n    text: |\n      {%- set fields = [\"classification.taxonomy\", \"classification.type\", \"classification.identifier\", \"source.ip\",\"source.asn\", \"source.port\"] %}\n      {%- set sep = joiner(\";\") %}\n      {%- for field in fields %}{{ sep() }}{{ field }}{%- endfor %}\n      {% set sep = joiner(\";\") %}\n      {%- for field in fields %}{{ sep() }}{{ event[field] }}{%- endfor %}\n    name: event.csv\n
           
          Module: intelmq.bots.outputs.templated_smtp.output
           
          Requirements
           
          Install the required jinja2 library:
           
          pip3 install -r intelmq/bots/collectors/templated_smtp/REQUIREMENTS.txt\n
           
          Parameters:
           
          attachments
           
          (required, array of objects) Each object must have content-type, text (attachment text) and name (filename of the attachment) fields.
           
          - content-type: simple string/jinja template\n  text: simple string/jinja template\n  name: simple string/jinja template\n
           
          body
           
          (optional, string) Simple string or Jinja template. The default body template prints every field in the event except raw, in undefined order, one field per line, as \"field: value\".
           
          mail_from
           
          (optional, string) Simple string or Jinja template. Sender's address.
           
          mail_to
           
          (required, string) Simple string or Jinja template. Comma-separated array of recipient addresses.
           
          smtp_host
           
          (optional, string) Hostname of the SMTP server. Defaults to localhost.
           
          smtp_password
           
          (optional, string) Password (if any) for authenticated SMTP. Defaults to null.
           
          smtp_port
           
          (optional, integer) TCP port to connect to. Defaults to 25.
           
          smtp_username
           
          (optional, string)  Username (if any) for authenticated SMTP. Defaults to null.
           
          tls
           
          (optional, boolean) Whether to use use SMTPS. When true, also set smtp_port to the SMTPS port. Defaults to false.
           
          starttls
           
          (optional, boolean) Whether to use opportunistic STARTTLS over SMTP. Defaults to true.
           
          subject
           
          (optional, string) Simple string or Jinja template. E-mail subject line. Defaults to \"IntelMQ event\".
           
          verify_cert
           
          (optional, boolean) Whether to verify the server certificate in STARTTLS or SMTPS. Defaults to true.
          "},{"location":"user/bots/#touch","title":"Touch","text":"
          Touches a file for every event received. Does not output the event!
           
          Module: intelmq.bots.outputs.touch.output
           
          Parameters:
           
          path
           
          (optional, string) Path to the file to touch.
          "},{"location":"user/bots/#udp","title":"UDP","text":"
          Output Bot that sends events to a remote UDP server.
           
          Multihreading is disabled for this bot.
           
          Module: intelmq.bots.outputs.udp.output
           
          Parameters:
           
          format
           
          (optional, string) Allowed values: json or delimited. The JSON format outputs the event 'as-is'. Delimited will deconstruct the event and print each field:value separated by the field delimit. See examples below.
           
          field_delimiter
           
          (optional, string) If the format is delimited then this parameter is used as a delimiter between fields. Defaults to |.
           
          header
           
          (required, string) Header text to be sent in the UDP datagram.
           
          keep_raw_field
           
          (optional, boolean) Whether to keep raw field. Defaults to false.
           
          udp_host
           
          (optional, string) Hostname of the destination server.
           
          udp_port
           
          (required, integer) Port of the destination server.
           
          Examples of usage
           
          Consider the following event:
           
          {\n  \"raw\": \"MjAxNi8wNC8yNV8xMTozOSxzY2hpenppbm8ub21hcmF0aG9uLmNvbS9na0NDSnVUSE0vRFBlQ1pFay9XdFZOSERLbC1tWFllRk5Iai8sODUuMjUuMTYwLjExNCxzdGF0aWMtaXAtODUtMjUtMTYwLTExNC5pbmFkZHIuaXAtcG9vbC5jb20uLEFuZ2xlciBFSywtLDg5NzI=\",\n  \"source.asn\": 8972,\n  \"source.ip\": \"85.25.160.114\",\n  \"source.url\": \"http://schizzino.omarathon.com/gkCCJuTHM/DPeCZEk/WtVNHDKl-mXYeFNHj/\",\n  \"source.reverse_dns\": \"static-ip-85-25-160-114.inaddr.ip-pool.com\",\n  \"classification.type\": \"malware-distribution\",\n  \"event_description.text\": \"Angler EK\",\n  \"feed.url\": \"http://www.malwaredomainlist.com/updatescsv.php\",\n  \"feed.name\": \"Malware Domain List\",\n  \"feed.accuracy\": 100,\n  \"time.observation\": \"2016-04-29T10:59:34+00:00\",\n  \"time.source\": \"2016-04-25T11:39:00+00:00\"\n}\n
           
          With the following parameters:
           
          format: json\nheader: header example\nkeep_raw_field: true\nip: 127.0.0.1\nport: 514\n
           
          Resulting line in syslog:
           
          Apr 29 11:01:29 header example {\"raw\": \"MjAxNi8wNC8yNV8xMTozOSxzY2hpenppbm8ub21hcmF0aG9uLmNvbS9na0NDSnVUSE0vRFBlQ1pFay9XdFZOSERLbC1tWFllRk5Iai8sODUuMjUuMTYwLjExNCxzdGF0aWMtaXAtODUtMjUtMTYwLTExNC5pbmFkZHIuaXAtcG9vbC5jb20uLEFuZ2xlciBFSywtLDg5NzI=\", \"source\": {\"asn\": 8972, \"ip\": \"85.25.160.114\", \"url\": \"http://schizzino.omarathon.com/gkCCJuTHM/DPeCZEk/WtVNHDKl-mXYeFNHj/\", \"reverse_dns\": \"static-ip-85-25-160-114.inaddr.ip-pool.com\"}, \"classification\": {\"type\": \"malware-distribution\"}, \"event_description\": {\"text\": \"Angler EK\"}, \"feed\": {\"url\": \"http://www.malwaredomainlist.com/updatescsv.php\", \"name\": \"Malware Domain List\", \"accuracy\": 100.0}, \"time\": {\"observation\": \"2016-04-29T10:59:34+00:00\", \"source\": \"2016-04-25T11:39:00+00:00\"}}\n
           
          With the following Parameters:
           
          field_delimiter: |\nformat: delimited\nheader: IntelMQ-event\nkeep_raw_field: false\nip: 127.0.0.1\nport: 514\n
           
          Resulting line in syslog:
           
          Apr 29 11:17:47 localhost IntelMQ-event|source.ip: 85.25.160.114|time.source:2016-04-25T11:39:00+00:00|feed.url:http://www.malwaredomainlist.com/updatescsv.php|time.observation:2016-04-29T11:17:44+00:00|source.reverse_dns:static-ip-85-25-160-114.inaddr.ip-pool.com|feed.name:Malware Domain List|event_description.text:Angler EK|source.url:http://schizzino.omarathon.com/gkCCJuTHM/DPeCZEk/WtVNHDKl-mXYeFNHj/|source.asn:8972|classification.type:malware-distribution|feed.accuracy:100.0\n
          "},{"location":"user/event/","title":"Event","text":""},{"location":"user/event/#event","title":"Event","text":"
          An event represents individual piece of data processed by IntelMQ. It uses JSON format.
           
          Example Event:
           
          {\n    \"source.geolocation.cc\": \"JO\",\n    \"malware.name\": \"qakbot\",\n    \"source.ip\": \"82.212.115.188\",\n    \"source.asn\": 47887,\n    \"classification.type\": \"c2-server\",\n    \"extra.status\": \"offline\",\n    \"source.port\": 443,\n    \"classification.taxonomy\": \"malicious-code\",\n    \"source.geolocation.latitude\": 31.9522,\n    \"feed.accuracy\": 100,\n    \"extra.last_online\": \"2023-02-16\",\n    \"time.observation\": \"2023-02-16T09:55:12+00:00\",\n    \"source.geolocation.city\": \"amman\",\n    \"source.network\": \"82.212.115.0/24\",\n    \"time.source\": \"2023-02-15T14:19:09+00:00\",\n    \"source.as_name\": \"NEU-AS\",\n    \"source.geolocation.longitude\": 35.939,\n    \"feed.name\": \"abusech-feodo-c2-tracker\"\n  }\n
          "},{"location":"user/event/#minimum-requirements","title":"Minimum Requirements","text":"
          Below, we have enumerated the minimum recommended requirements for an actionable abuse event. These keys should be present for the abuse report to make sense for the end recipient. Please note that if you choose to anonymize your sources, you can substitute feed.name with feed.code. At least one of the fields ip, fqdn, url or account should be present. All the rest of the keys are optional. This list of required fields is not enforced by IntelMQ.
           Field Terminology feed.name Should classification.type Should classification.taxonomy Should time.source Should time.observation Should source.ip Should* source.fqdn Should* source.url Should* source.account Should* 
          * at least one of them
          "},{"location":"user/event/#classification","title":"Classification","text":"
          IntelMQ classifies events using three labels: classification.taxonomy, classification.type and classification.identifier. This tuple of three values can be used for deduplication of events and describes what happened.
           
          The taxonomy can be automatically added by the taxonomy expert bot based on the given type. The following classification scheme loosely follows the Reference Security Incident Taxonomy (RSIT):
           Classification Taxonomy Classification Type Description abusive-content harmful-speech Discreditation or discrimination of somebody, cyber stalking, racism or threats against one or more individuals. abusive-content spam Or 'Unsolicited Bulk Email', this means that the recipient has not granted verifiable permission for the message to be sent and that the message is sent as part of a larger collection of messages, all having a functionally comparable content. abusive-content violence Child pornography, glorification of violence, etc. availability ddos Distributed Denial of Service attack, e.g. SYN-Flood or UDP-based reflection/amplification attacks. availability dos Denial of Service attack, e.g. sending specially crafted requests to a web application which causes the application to crash or slow down. availability misconfiguration Software misconfiguration resulting in service availability issues, e.g. DNS server with outdated DNSSEC Root Zone KSK. availability outage Outage caused e.g. by air condition failure or natural disaster. availability sabotage Physical sabotage, e.g cutting wires or malicious arson. fraud copyright Offering or Installing copies of unlicensed commercial software or other copyright protected materials (Warez). fraud masquerade Type of attack in which one entity illegitimately impersonates the identity of another in order to benefit from it. fraud phishing Masquerading as another entity in order to persuade the user to reveal private credentials. fraud unauthorized-use-of-resources Using resources for unauthorized purposes including profit-making ventures, e.g. the use of e-mail to participate in illegal profit chain letters or pyramid schemes. information-content-security data-leak Leaked confidential information like credentials or personal data. information-content-security data-loss Loss of data, e.g. caused by harddisk failure or physical theft. information-content-security unauthorised-information-access Unauthorized access to information, e.g. by abusing stolen login credentials for a system or application, intercepting traffic or gaining access to physical documents. information-content-security unauthorised-information-modification Unauthorised modification of information, e.g. by an attacker abusing stolen login credentials for a system or application or a ransomware encrypting data. information-gathering scanner Attacks that send requests to a system to discover weaknesses. This also includes testing processes to gather information on hosts, services and accounts. Examples: fingerd, DNS querying, ICMP, SMTP (EXPN, RCPT, ...), port scanning. information-gathering sniffing Observing and recording of network traffic (wiretapping). information-gathering social-engineering Gathering information from a human being in a non-technical way (e.g. lies, tricks, bribes, or threats). This IOC refers to a resource, which has been observed to perform brute-force attacks over a given application protocol. intrusion-attempts brute-force Multiple login attempts (Guessing/cracking of passwords, brute force). intrusion-attempts exploit An attack using an unknown exploit. intrusion-attempts ids-alert IOCs based on a sensor network. This is a generic IOC denomination, should it be difficult to reliably denote the exact type of activity involved for example due to an anecdotal nature of the rule that triggered the alert. intrusions application-compromise Compromise of an application by exploiting (un)known software vulnerabilities, e.g. SQL injection. intrusions burglary Physical intrusion, e.g. into corporate building or data center. intrusions privileged-account-compromise Compromise of a system where the attacker gained administrative privileges. intrusions system-compromise Compromise of a system, e.g. unauthorised logins or commands. This includes compromising attempts on honeypot systems. intrusions unprivileged-account-compromise Compromise of a system using an unprivileged (user/service) account. malicious-code c2-server This is a command and control server in charge of a given number of botnet drones. malicious-code infected-system This is a compromised machine, which has been observed to make a connection to a command and control server. malicious-code malware-configuration This is a resource which updates botnet drones with a new configuration. malicious-code malware-distribution URI used for malware distribution, e.g. a download URL included in fake invoice malware spam. other blacklist Some sources provide blacklists, which clearly refer to abusive behavior, such as spamming, but fail to denote the exact reason why a given identity has been blacklisted. The reason may be that the justification is anecdotal or missing entirely. This type should only be used if the typing fits the definition of a blacklist, but an event specific denomination is not possible for one reason or another. Not in RSIT. other dga-domain DGA Domains are seen various families of malware that are used to periodically generate a large number of domain names that can be used as rendezvous points with their command and control servers. Not in RSIT. other other All incidents which don't fit in one of the given categories should be put into this class. other malware An IoC referring to a malware (sample) itself. Not in RSIT. other proxy This refers to the use of proxies from inside your network. Not in RSIT. test test Meant for testing. Not in RSIT. other tor This IOC refers to incidents related to TOR network infrastructure. Not in RSIT. other undetermined The categorisation of the incident is unknown/undetermined. vulnerable ddos-amplifier Publicly accessible services that can be abused for conducting DDoS reflection/amplification attacks, e.g. DNS open-resolvers or NTP servers with monlist enabled. vulnerable information-disclosure Publicly accessible services potentially disclosing sensitive information, e.g. SNMP or Redis. vulnerable potentially-unwanted-accessible Potentially unwanted publicly accessible services, e.g. Telnet, RDP or VNC. vulnerable vulnerable-system A system which is vulnerable to certain attacks. Example: misconfigured client proxy settings (example: WPAD), outdated operating system  version, etc. vulnerable weak-crypto Publicly accessible services offering weak crypto, e.g. web servers susceptible to POODLE/FREAK attacks."},{"location":"user/event/#meaning-of-source-and-destination-identities","title":"Meaning of source and destination identities","text":"
          Meaning of source and destination identities for each classification.type can be different. Usually the main information is in the source.* fields.
           
          The classification.identifier is often a normalized malware name, grouping many variants or the affected network protocol.
           
          Examples of the meaning of the source and destination fields for various classification.type and possible identifiers are shown here.
           Classification Type Source Destination Possible Identifiers blacklist blacklisted device brute-force attacker target c2-server (sinkholed) c&c server zeus, palevo, feodo ddos attacker target dga-domain infected device dropzone server hosting stolen data exploit hosting server ids-alert triggering device infected-system infected device contacted c&c server malware infected device zeus, palevo, feodo malware-configuration infected device malware-distribution server hosting malware phishing phishing website proxy server allowing policy/security bypass scanner scanning device scanned device http, modbus, wordpress spam infected device targeted server system-compromise server vulnerable-system vulnerable device heartbleed, openresolver, snmp, wpad 
          Examples:
           
             
          •  
            If an event describes IP address that connects to a zeus command and control server, it's about the infected device. Therefore the classification.taxonomy is malicious-code, classification.type is infected-system and the classification.identifier is zeus.
             
            •  
          •  
            If an event describes IP address where a command and control server is running, the event's classification.type is c2server. The malware.name can have the full name, eg. zeus_p2p.
             
            •  
          "},{"location":"user/event/#additional-information","title":"Additional Information","text":"
          Information that do not fit into any of the event fields should be placed in the extra namespace.Therefore the keys must be prefixed extra. string. There are no other rules on key names and values for additional information.
          "},{"location":"user/event/#fields-reference","title":"Fields Reference","text":"
          Here you can find detailed information about all the possible fields used in an event.
          "},{"location":"user/event/#classificationidentifier","title":"classification.identifier","text":"
          Type: String
           
          The lowercase identifier defines the actual software or service (e.g. heartbleed or ntp_version) or standardized malware name (e.g. zeus). Note that you MAY overwrite this field during processing for your individual setup. This field is not standardized across IntelMQ setups/users.
          "},{"location":"user/event/#classificationtaxonomy","title":"classification.taxonomy","text":"
          Type: ClassificationTaxonomy
           
          We recognize the need for the CSIRT teams to apply a static (incident) taxonomy to abuse data. With this goal in mind the type IOC will serve as a basis for this activity. Each value of the dynamic type mapping translates to a an element in the static taxonomy. The European CSIRT teams for example have decided to apply the eCSIRT.net incident classification. The value of the taxonomy key is thus a derivative of the dynamic type above. For more information about check ENISA taxonomies <http://www.enisa.europa.eu/activities/cert/support/incident-management/browsable/incident-handling-process/incident-taxonomy/existing-taxonomies>_.
          "},{"location":"user/event/#classificationtype","title":"classification.type","text":"
          Type: ClassificationType
           
          The abuse type IOC is one of the most crucial pieces of information for any given abuse event. The main idea of dynamic typing is to keep our ontology flexible, since we need to evolve with the evolving threatscape of abuse data. In contrast with the static taxonomy below, the dynamic typing is used to perform business decisions in the abuse handling pipeline. Furthermore, the value data set should be kept as minimal as possible to avoid type explosion, which in turn dilutes the business value of the dynamic typing. In general, we normally have two types of abuse type IOC: ones referring to a compromised resource or ones referring to pieces of the criminal infrastructure, such as a command and control servers for example.
          "},{"location":"user/event/#comment","title":"comment","text":"
          Type: String
           
          Free text commentary about the abuse event inserted by an analyst.
          "},{"location":"user/event/#destinationabuse_contact","title":"destination.abuse_contact","text":"
          Type: LowercaseString
           
          Abuse contact for destination address. A comma separated list.
          "},{"location":"user/event/#destinationaccount","title":"destination.account","text":"
          Type: String
           
          An account name or email address, which has been identified to relate to the destination of an abuse event.
          "},{"location":"user/event/#destinationallocated","title":"destination.allocated","text":"
          Type: DateTime
           
          Allocation date corresponding to BGP prefix.
          "},{"location":"user/event/#destinationas_name","title":"destination.as_name","text":"
          Type: String
           
          The autonomous system name to which the connection headed.
          "},{"location":"user/event/#destinationasn","title":"destination.asn","text":"
          Type: ASN
           
          The autonomous system number to which the connection headed.
          "},{"location":"user/event/#destinationdomain_suffix","title":"destination.domain_suffix","text":"
          Type: FQDN
           
          The suffix of the domain from the public suffix list.
          "},{"location":"user/event/#destinationfqdn","title":"destination.fqdn","text":"
          Type: FQDN
           
          A DNS name related to the host from which the connection originated. DNS allows even binary data in DNS, so we have to allow everything. A final point is stripped, string is converted to lower case characters.
          "},{"location":"user/event/#destinationgeolocationcc","title":"destination.geolocation.cc","text":"
          Type: UppercaseString
           
          Country-Code according to ISO3166-1 alpha-2 for the destination IP.
          "},{"location":"user/event/#destinationgeolocationcity","title":"destination.geolocation.city","text":"
          Type: String
           
          Some geolocation services refer to city-level geolocation.
          "},{"location":"user/event/#destinationgeolocationcountry","title":"destination.geolocation.country","text":"
          Type: String
           
          The country name derived from the ISO3166 country code (assigned to cc field).
          "},{"location":"user/event/#destinationgeolocationlatitude","title":"destination.geolocation.latitude","text":"
          Type: Float
           
          Latitude coordinates derived from a geolocation service, such as MaxMind geoip db.
          "},{"location":"user/event/#destinationgeolocationlongitude","title":"destination.geolocation.longitude","text":"
          Type: Float
           
          Longitude coordinates derived from a geolocation service, such as MaxMind geoip db.
          "},{"location":"user/event/#destinationgeolocationregion","title":"destination.geolocation.region","text":"
          Type: String
           
          Some geolocation services refer to region-level geolocation.
          "},{"location":"user/event/#destinationgeolocationstate","title":"destination.geolocation.state","text":"
          Type: String
           
          Some geolocation services refer to state-level geolocation.
          "},{"location":"user/event/#destinationip","title":"destination.ip","text":"
          Type: IPAddress
           
          The IP which is the target of the observed connections.
          "},{"location":"user/event/#destinationlocal_hostname","title":"destination.local_hostname","text":"
          Type: String
           
          Some sources report an internal hostname within a NAT related to the name configured for a compromised system
          "},{"location":"user/event/#destinationlocal_ip","title":"destination.local_ip","text":"
          Type: IPAddress
           
          Some sources report an internal (NATed) IP address related a compromised system. N.B. RFC1918 IPs are OK here.
          "},{"location":"user/event/#destinationnetwork","title":"destination.network","text":"
          Type: IPNetwork
           
          CIDR for an autonomous system. Also known as BGP prefix. If multiple values are possible, select the most specific.
          "},{"location":"user/event/#destinationport","title":"destination.port","text":"
          Type: Integer
           
          The port to which the connection headed.
          "},{"location":"user/event/#destinationregistry","title":"destination.registry","text":"
          Type: Registry
           
          The IP registry a given ip address is allocated by.
          "},{"location":"user/event/#destinationreverse_dns","title":"destination.reverse_dns","text":"
          Type: FQDN
           
          Reverse DNS name acquired through a reverse DNS query on an IP address. N.B. Record types other than PTR records may also appear in the reverse DNS tree. Furthermore, unfortunately, there is no rule prohibiting people from writing anything in a PTR record. Even JavaScript will work. A final point is stripped, string is converted to lower case characters.
          "},{"location":"user/event/#destinationtor_node","title":"destination.tor_node","text":"
          Type: Boolean
           
          If the destination IP was a known tor node.
          "},{"location":"user/event/#destinationurl","title":"destination.url","text":"
          Type: URL
           
          A URL denotes on IOC, which refers to a malicious resource, whose interpretation is defined by the abuse type. A URL with the abuse type phishing refers to a phishing resource.
          "},{"location":"user/event/#destinationurlpath","title":"destination.urlpath","text":"
          Type: String
           
          The path portion of an HTTP or related network request.
          "},{"location":"user/event/#event_descriptiontarget","title":"event_description.target","text":"
          Type: String
           
          Some sources denominate the target (organization) of a an attack.
          "},{"location":"user/event/#event_descriptiontext","title":"event_description.text","text":"
          Type: String
           
          A free-form textual description of an abuse event.
          "},{"location":"user/event/#event_descriptionurl","title":"event_description.url","text":"
          Type: URL
           
          A description URL is a link to a further description of the the abuse event in question.
          "},{"location":"user/event/#event_hash","title":"event_hash","text":"
          Type: UppercaseString
           
          Computed event hash with specific keys and values that identify a unique event. At present, the hash should default to using the SHA1 function. Please note that for an event hash to be able to match more than one event (deduplication) the receiver of an event should calculate it based on a minimal set of keys and values present in the event. Using for example the observation time in the calculation will most likely render the checksum useless for deduplication purposes.
          "},{"location":"user/event/#extra","title":"extra","text":"
          Type: JSONDict
           
          All anecdotal information, which cannot be parsed into the data harmonization elements. E.g. os.name, os.version, etc.  Note: this is only intended for mapping any fields which can not map naturally into the data harmonization. It is not intended for extending the data harmonization with your own fields.
          "},{"location":"user/event/#feedaccuracy","title":"feed.accuracy","text":"
          Type: Accuracy
           
          A float between 0 and 100 that represents how accurate the data in the feed is
          "},{"location":"user/event/#feedcode","title":"feed.code","text":"
          Type: String
           
          Code name for the feed, e.g. DFGS, HSDAG etc.
          "},{"location":"user/event/#feeddocumentation","title":"feed.documentation","text":"
          Type: String
           
          A URL or hint where to find the documentation of this feed.
          "},{"location":"user/event/#feedname","title":"feed.name","text":"
          Type: String
           
          Name for the feed, usually found in collector bot configuration.
          "},{"location":"user/event/#feedprovider","title":"feed.provider","text":"
          Type: String
           
          Name for the provider of the feed, usually found in collector bot configuration.
          "},{"location":"user/event/#feedurl","title":"feed.url","text":"
          Type: URL
           
          The URL of a given abuse feed, where applicable
          "},{"location":"user/event/#malwarehashmd5","title":"malware.hash.md5","text":"
          Type: String
           
          A string depicting an MD5 checksum for a file, be it a malware sample for example.
          "},{"location":"user/event/#malwarehashsha1","title":"malware.hash.sha1","text":"
          Type: String
           
          A string depicting a SHA1 checksum for a file, be it a malware sample for example.
          "},{"location":"user/event/#malwarehashsha256","title":"malware.hash.sha256","text":"
          Type: String
           
          A string depicting a SHA256 checksum for a file, be it a malware sample for example.
          "},{"location":"user/event/#malwarename","title":"malware.name","text":"
          Type: LowercaseString
           
          The malware name in lower case.
          "},{"location":"user/event/#malwareversion","title":"malware.version","text":"
          Type: String
           
          A version string for an identified artifact generation, e.g. a crime-ware kit.
          "},{"location":"user/event/#mispattribute_uuid","title":"misp.attribute_uuid","text":"
          Type: LowercaseString
           
          MISP - Malware Information Sharing Platform & Threat Sharing UUID of an attribute.
          "},{"location":"user/event/#mispevent_uuid","title":"misp.event_uuid","text":"
          Type: LowercaseString
           
          MISP - Malware Information Sharing Platform & Threat Sharing UUID.
          "},{"location":"user/event/#output","title":"output","text":"
          Type: JSON
           
          Event data converted into foreign format, intended to be exported by output plugin.
          "},{"location":"user/event/#protocolapplication","title":"protocol.application","text":"
          Type: LowercaseString
           
          e.g. vnc, ssh, sip, irc, http or smtp.
          "},{"location":"user/event/#protocoltransport","title":"protocol.transport","text":"
          Type: LowercaseString
           
          e.g. tcp, udp, icmp.
          "},{"location":"user/event/#raw","title":"raw","text":"
          Type: Base64
           
          The original line of the event from encoded in base64.
          "},{"location":"user/event/#rtir_id","title":"rtir_id","text":"
          Type: Integer
           
          Request Tracker Incident Response ticket id.
          "},{"location":"user/event/#screenshot_url","title":"screenshot_url","text":"
          Type: URL
           
          Some source may report URLs related to a an image generated of a resource without any metadata. Or an URL pointing to resource, which has been rendered into a webshot, e.g. a PNG image and the relevant metadata related to its retrieval/generation.
          "},{"location":"user/event/#sourceabuse_contact","title":"source.abuse_contact","text":"
          Type: LowercaseString
           
          Abuse contact for source address. A comma separated list.
          "},{"location":"user/event/#sourceaccount","title":"source.account","text":"
          Type: String
           
          An account name or email address, which has been identified to relate to the source of an abuse event.
          "},{"location":"user/event/#sourceallocated","title":"source.allocated","text":"
          Type: DateTime
           
          Allocation date corresponding to BGP prefix.
          "},{"location":"user/event/#sourceas_name","title":"source.as_name","text":"
          Type: String
           
          The autonomous system name from which the connection originated.
          "},{"location":"user/event/#sourceasn","title":"source.asn","text":"
          Type: ASN
           
          The autonomous system number from which originated the connection.
          "},{"location":"user/event/#sourcedomain_suffix","title":"source.domain_suffix","text":"
          Type: FQDN
           
          The suffix of the domain from the public suffix list.
          "},{"location":"user/event/#sourcefqdn","title":"source.fqdn","text":"
          Type: FQDN
           
          A DNS name related to the host from which the connection originated. DNS allows even binary data in DNS, so we have to allow everything. A final point is stripped, string is converted to lower case characters.
          "},{"location":"user/event/#sourcegeolocationcc","title":"source.geolocation.cc","text":"
          Type: UppercaseString
           
          Country-Code according to ISO3166-1 alpha-2 for the source IP.
          "},{"location":"user/event/#sourcegeolocationcity","title":"source.geolocation.city","text":"
          Type: String
           
          Some geolocation services refer to city-level geolocation.
          "},{"location":"user/event/#sourcegeolocationcountry","title":"source.geolocation.country","text":"
          Type: String
           
          The country name derived from the ISO3166 country code (assigned to cc field).
          "},{"location":"user/event/#sourcegeolocationcymru_cc","title":"source.geolocation.cymru_cc","text":"
          Type: UppercaseString
           
          The country code denoted for the ip by the Team Cymru asn to ip mapping service.
          "},{"location":"user/event/#sourcegeolocationgeoip_cc","title":"source.geolocation.geoip_cc","text":"
          Type: UppercaseString
           
          MaxMind Country Code (ISO3166-1 alpha-2).
          "},{"location":"user/event/#sourcegeolocationlatitude","title":"source.geolocation.latitude","text":"
          Type: Float
           
          Latitude coordinates derived from a geolocation service, such as MaxMind geoip db.
          "},{"location":"user/event/#sourcegeolocationlongitude","title":"source.geolocation.longitude","text":"
          Type: Float
           
          Longitude coordinates derived from a geolocation service, such as MaxMind geoip db.
          "},{"location":"user/event/#sourcegeolocationregion","title":"source.geolocation.region","text":"
          Type: String
           
          Some geolocation services refer to region-level geolocation.
          "},{"location":"user/event/#sourcegeolocationstate","title":"source.geolocation.state","text":"
          Type: String
           
          Some geolocation services refer to state-level geolocation.
          "},{"location":"user/event/#sourceip","title":"source.ip","text":"
          Type: IPAddress
           
          The ip observed to initiate the connection
          "},{"location":"user/event/#sourcelocal_hostname","title":"source.local_hostname","text":"
          Type: String
           
          Some sources report a internal hostname within a NAT related to the name configured for a compromised system
          "},{"location":"user/event/#sourcelocal_ip","title":"source.local_ip","text":"
          Type: IPAddress
           
          Some sources report a internal (NATed) IP address related a compromised system. N.B. RFC1918 IPs are OK here.
          "},{"location":"user/event/#sourcenetwork","title":"source.network","text":"
          Type: IPNetwork
           
          CIDR for an autonomous system. Also known as BGP prefix. If multiple values are possible, select the most specific.
          "},{"location":"user/event/#sourceport","title":"source.port","text":"
          Type: Integer
           
          The port from which the connection originated.
          "},{"location":"user/event/#sourceregistry","title":"source.registry","text":"
          Type: Registry
           
          The IP registry a given ip address is allocated by.
          "},{"location":"user/event/#sourcereverse_dns","title":"source.reverse_dns","text":"
          Type: FQDN
           
          Reverse DNS name acquired through a reverse DNS query on an IP address. N.B. Record types other than PTR records may also appear in the reverse DNS tree. Furthermore, unfortunately, there is no rule prohibiting people from writing anything in a PTR record. Even JavaScript will work. A final point is stripped, string is converted to lower case characters.
          "},{"location":"user/event/#sourcetor_node","title":"source.tor_node","text":"
          Type: Boolean
           
          If the source IP was a known tor node.
          "},{"location":"user/event/#sourceurl","title":"source.url","text":"
          Type: URL
           
          A URL denotes an IOC, which refers to a malicious resource, whose interpretation is defined by the abuse type. A URL with the abuse type phishing refers to a phishing resource.
          "},{"location":"user/event/#sourceurlpath","title":"source.urlpath","text":"
          Type: String
           
          The path portion of an HTTP or related network request.
          "},{"location":"user/event/#status","title":"status","text":"
          Type: String
           
          Status of the malicious resource (phishing, dropzone, etc), e.g. online, offline.
          "},{"location":"user/event/#timeobservation","title":"time.observation","text":"
          Type: DateTime
           
          The time the collector of the local instance processed (observed) the event.
          "},{"location":"user/event/#timesource","title":"time.source","text":"
          Type: DateTime
           
          The time of occurrence of the event as reported the feed (source).
          "},{"location":"user/event/#tlp","title":"tlp","text":"
          Type: TLP
           
          Traffic Light Protocol level of the event.
          "},{"location":"user/feeds/","title":"Feeds","text":""},{"location":"user/feeds/#feeds","title":"Feeds","text":"
          The available feeds are grouped by the provider of the feeds. For each feed the collector and parser that can be used is documented as well as any feed-specific parameters. To add feeds to this file add them to intelmq/etc/feeds.yaml and then rebuild the documentation.
          "},{"location":"user/feeds/#abusech","title":"Abuse.ch","text":""},{"location":"user/feeds/#feodo-tracker","title":"Feodo Tracker","text":"
          List of botnet Command & Control servers (C&Cs) tracked by Feodo Tracker, associated with Dridex and Emotet (aka Heodo).
           
          Public: yes
           
          Revision: 2022-11-15
           
          Documentation: https://feodotracker.abuse.ch/
           
          Additional Information: The data in the column Last Online is used for time.source if available, with 00:00 as time. Otherwise first_seen is used as time.source.
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://feodotracker.abuse.ch/downloads/ipblocklist.json\n  name: Feodo Tracker\n  provider: Abuse.ch\n  rate_limit: 3600\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.abusech.parser_feodotracker\n
          "},{"location":"user/feeds/#urlhaus","title":"URLhaus","text":"
          URLhaus is a project from abuse.ch with the goal of sharing malicious URLs that are being used for malware distribution. URLhaus offers a country, ASN (AS number) and Top Level Domain (TLD) feed for network operators / Internet Service Providers (ISPs), Computer Emergency Response Teams (CERTs) and domain registries.
           
          Public: yes
           
          Revision: 2020-07-07
           
          Documentation: https://urlhaus.abuse.ch/feeds/
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://urlhaus.abuse.ch/feeds/tld/<TLD>/, https://urlhaus.abuse.ch/feeds/country/<CC>/, or https://urlhaus.abuse.ch/feeds/asn/<ASN>/\n  name: URLhaus\n  provider: Abuse.ch\n  rate_limit: 86400\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.generic.parser_csv\nparameters:\n  columns: [\"time.source\", \"source.url\", \"status\", \"classification.type|__IGNORE__\", \"source.fqdn|__IGNORE__\", \"source.ip\", \"source.asn\", \"source.geolocation.cc\"]\n  default_url_protocol: http://\n  delimiter: ,\n  skip_header: False\n  type_translation: [{\"malware_download\": \"malware-distribution\"}]\n
          "},{"location":"user/feeds/#alienvault","title":"AlienVault","text":""},{"location":"user/feeds/#otx","title":"OTX","text":"
          AlienVault OTX Collector is the bot responsible to get the report through the API. Report could vary according to subscriptions.
           
          Public: no
           
          Revision: 2018-01-20
           
          Documentation: https://otx.alienvault.com/
           
          Collector configuration
           
          module: intelmq.bots.collectors.alienvault_otx.collector\nparameters:\n  api_key: {{ your API key }}\n  name: OTX\n  provider: AlienVault\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.alienvault.parser_otx\n
          "},{"location":"user/feeds/#reputation-list","title":"Reputation List","text":"
          List of malicious IPs.
           
          Public: yes
           
          Revision: 2018-01-20
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://reputation.alienvault.com/reputation.data\n  name: Reputation List\n  provider: AlienVault\n  rate_limit: 3600\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.alienvault.parser\n
          "},{"location":"user/feeds/#anubisnetworks","title":"AnubisNetworks","text":""},{"location":"user/feeds/#cyberfeed-stream","title":"Cyberfeed Stream","text":"
          Fetches and parsers the Cyberfeed data stream.
           
          Public: no
           
          Revision: 2020-06-15
           
          Documentation: https://www.anubisnetworks.com/ https://www.bitsight.com/
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http_stream\nparameters:\n  http_url: https://prod.cyberfeed.net/stream?key={{ your API key }}\n  name: Cyberfeed Stream\n  provider: AnubisNetworks\n  strip_lines: true\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.anubisnetworks.parser\nparameters:\n  use_malware_familiy_as_classification_identifier: True\n
          "},{"location":"user/feeds/#bambenek","title":"Bambenek","text":""},{"location":"user/feeds/#c2-domains","title":"C2 Domains","text":"
          Master Feed of known, active and non-sinkholed C&Cs domain names. Requires access credentials.
           
          Public: no
           
          Revision: 2018-01-20
           
          Documentation: https://osint.bambenekconsulting.com/feeds/
           
          Additional Information: License: https://osint.bambenekconsulting.com/license.txt
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_password: __PASSWORD__\n  http_url: https://faf.bambenekconsulting.com/feeds/c2-dommasterlist.txt\n  http_username: __USERNAME__\n  name: C2 Domains\n  provider: Bambenek\n  rate_limit: 3600\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.bambenek.parser\n
          "},{"location":"user/feeds/#c2-ips","title":"C2 IPs","text":"
          Master Feed of known, active and non-sinkholed C&Cs IP addresses. Requires access credentials.
           
          Public: no
           
          Revision: 2018-01-20
           
          Documentation: https://osint.bambenekconsulting.com/feeds/
           
          Additional Information: License: https://osint.bambenekconsulting.com/license.txt
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_password: __PASSWORD__\n  http_url: https://faf.bambenekconsulting.com/feeds/c2-ipmasterlist.txt\n  http_username: __USERNAME__\n  name: C2 IPs\n  provider: Bambenek\n  rate_limit: 3600\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.bambenek.parser\n
          "},{"location":"user/feeds/#dga-domains","title":"DGA Domains","text":"
          Domain feed of known DGA domains from -2 to +3 days
           
          Public: yes
           
          Revision: 2018-01-20
           
          Documentation: https://osint.bambenekconsulting.com/feeds/
           
          Additional Information: License: https://osint.bambenekconsulting.com/license.txt
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://faf.bambenekconsulting.com/feeds/dga-feed.txt\n  name: DGA Domains\n  provider: Bambenek\n  rate_limit: 3600\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.bambenek.parser\n
          "},{"location":"user/feeds/#benkow","title":"Benkow","text":""},{"location":"user/feeds/#malware-panels-tracker","title":"Malware Panels Tracker","text":"
          Benkow Panels tracker is a list of fresh panel from various malware. The feed is available on the webpage: http://benkow.cc/passwords.php
           
          Public: yes
           
          Revision: 2022-11-16
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: http://benkow.cc/export.php\n  name: Malware Panels Tracker\n  provider: Benkow\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.generic.parser_csv\nparameters:\n  columns: [\"__IGNORE__\", \"malware.name\", \"source.url\", \"source.fqdn|source.ip\", \"time.source\"]\n  columns_required: [false, true, true, false, true]\n  defaults_fields: {'classification.type': 'c2-server'}\n  delimiter: ;\n  skip_header: True\n
          "},{"location":"user/feeds/#blocklistde","title":"Blocklist.de","text":""},{"location":"user/feeds/#apache","title":"Apache","text":"
          Blocklist.DE Apache Collector is the bot responsible to get the report from source of information. All IP addresses which have been reported within the last 48 hours as having run attacks on the service Apache, Apache-DDOS, RFI-Attacks.
           
          Public: yes
           
          Revision: 2018-01-20
           
          Documentation: http://www.blocklist.de/en/export.html
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://lists.blocklist.de/lists/apache.txt\n  name: Apache\n  provider: Blocklist.de\n  rate_limit: 86400\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.blocklistde.parser\n
          "},{"location":"user/feeds/#bots","title":"Bots","text":"
          Blocklist.DE Bots Collector is the bot responsible to get the report from source of information. All IP addresses which have been reported within the last 48 hours as having run attacks attacks on the RFI-Attacks, REG-Bots, IRC-Bots or BadBots (BadBots = he has posted a Spam-Comment on a open Forum or Wiki).
           
          Public: yes
           
          Revision: 2018-01-20
           
          Documentation: http://www.blocklist.de/en/export.html
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://lists.blocklist.de/lists/bots.txt\n  name: Bots\n  provider: Blocklist.de\n  rate_limit: 86400\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.blocklistde.parser\n
          "},{"location":"user/feeds/#brute-force-logins","title":"Brute-force Logins","text":"
          Blocklist.DE Brute-force Login Collector is the bot responsible to get the report from source of information. All IPs which attacks Joomlas, Wordpress and other Web-Logins with Brute-Force Logins.
           
          Public: yes
           
          Revision: 2018-01-20
           
          Documentation: http://www.blocklist.de/en/export.html
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://lists.blocklist.de/lists/bruteforcelogin.txt\n  name: Brute-force Logins\n  provider: Blocklist.de\n  rate_limit: 86400\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.blocklistde.parser\n
          "},{"location":"user/feeds/#ftp","title":"FTP","text":"
          Blocklist.DE FTP Collector is the bot responsible to get the report from source of information. All IP addresses which have been reported within the last 48 hours for attacks on the Service FTP.
           
          Public: yes
           
          Revision: 2018-01-20
           
          Documentation: http://www.blocklist.de/en/export.html
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://lists.blocklist.de/lists/ftp.txt\n  name: FTP\n  provider: Blocklist.de\n  rate_limit: 86400\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.blocklistde.parser\n
          "},{"location":"user/feeds/#imap","title":"IMAP","text":"
          Blocklist.DE IMAP Collector is the bot responsible to get the report from source of information. All IP addresses which have been reported within the last 48 hours for attacks on the service like IMAP, SASL, POP3, etc.
           
          Public: yes
           
          Revision: 2018-01-20
           
          Documentation: http://www.blocklist.de/en/export.html
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://lists.blocklist.de/lists/imap.txt\n  name: IMAP\n  provider: Blocklist.de\n  rate_limit: 86400\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.blocklistde.parser\n
          "},{"location":"user/feeds/#irc-bots","title":"IRC Bots","text":"
          No description provided by feed provider.
           
          Public: yes
           
          Revision: 2018-01-20
           
          Documentation: http://www.blocklist.de/en/export.html
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://lists.blocklist.de/lists/ircbot.txt\n  name: IRC Bots\n  provider: Blocklist.de\n  rate_limit: 86400\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.blocklistde.parser\n
          "},{"location":"user/feeds/#mail","title":"Mail","text":"
          Blocklist.DE Mail Collector is the bot responsible to get the report from source of information. All IP addresses which have been reported within the last 48 hours as having run attacks on the service Mail, Postfix.
           
          Public: yes
           
          Revision: 2018-01-20
           
          Documentation: http://www.blocklist.de/en/export.html
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://lists.blocklist.de/lists/mail.txt\n  name: Mail\n  provider: Blocklist.de\n  rate_limit: 86400\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.blocklistde.parser\n
          "},{"location":"user/feeds/#sip","title":"SIP","text":"
          Blocklist.DE SIP Collector is the bot responsible to get the report from source of information. All IP addresses that tried to login in a SIP-, VOIP- or Asterisk-Server and are included in the IPs-List from http://www.infiltrated.net/ (Twitter).
           
          Public: yes
           
          Revision: 2018-01-20
           
          Documentation: http://www.blocklist.de/en/export.html
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://lists.blocklist.de/lists/sip.txt\n  name: SIP\n  provider: Blocklist.de\n  rate_limit: 86400\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.blocklistde.parser\n
          "},{"location":"user/feeds/#ssh","title":"SSH","text":"
          Blocklist.DE SSH Collector is the bot responsible to get the report from source of information. All IP addresses which have been reported within the last 48 hours as having run attacks on the service SSH.
           
          Public: yes
           
          Revision: 2018-01-20
           
          Documentation: http://www.blocklist.de/en/export.html
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://lists.blocklist.de/lists/ssh.txt\n  name: SSH\n  provider: Blocklist.de\n  rate_limit: 86400\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.blocklistde.parser\n
          "},{"location":"user/feeds/#strong-ips","title":"Strong IPs","text":"
          Blocklist.DE Strong IPs Collector is the bot responsible to get the report from source of information. All IPs which are older then 2 month and have more then 5.000 attacks.
           
          Public: yes
           
          Revision: 2018-01-20
           
          Documentation: http://www.blocklist.de/en/export.html
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://lists.blocklist.de/lists/strongips.txt\n  name: Strong IPs\n  provider: Blocklist.de\n  rate_limit: 86400\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.blocklistde.parser\n
          "},{"location":"user/feeds/#blueliv","title":"Blueliv","text":""},{"location":"user/feeds/#crimeserver","title":"CrimeServer","text":"
          Blueliv Crimeserver Collector is the bot responsible to get the report through the API.
           
          Public: no
           
          Revision: 2018-01-20
           
          Documentation: https://www.blueliv.com/
           
          Additional Information: The service uses a different API for free users and paying subscribers. In 'CrimeServer' feed the difference lies in the data points present in the feed. The non-free API available from Blueliv contains, for this specific feed, following extra fields not present in the free API; \"_id\" - Internal unique ID \"subType\" - Subtype of the Crime Server \"countryName\" - Country name where the Crime Server is located, in English \"city\" - City where the Crime Server is located \"domain\" - Domain of the Crime Server \"host\" - Host of the Crime Server \"createdAt\" - Date when the Crime Server was added to Blueliv CrimeServer database \"asnCidr\" - Range of IPs that belong to an ISP (registered via Autonomous System Number (ASN)) \"asnId\" - Identifier of an ISP registered via ASN \"asnDesc\" Description of the ISP registered via ASN
           
          Collector configuration
           
          module: intelmq.bots.collectors.blueliv.collector_crimeserver\nparameters:\n  api_key: __APIKEY__\n  name: CrimeServer\n  provider: Blueliv\n  rate_limit: 3600\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.blueliv.parser_crimeserver\n
          "},{"location":"user/feeds/#cert-bund","title":"CERT-Bund","text":""},{"location":"user/feeds/#cb-report-malware-infections-via-imap","title":"CB-Report Malware infections via IMAP","text":"
          CERT-Bund sends reports for the malware-infected hosts.
           
          Public: no
           
          Revision: 2020-08-20
           
          Additional Information: Traffic from malware related hosts contacting command-and-control servers is caught and sent to national CERT teams. There are two e-mail feeds with identical CSV structure -- one reports on general malware infections, the other on the Avalanche botnet.
           
          Collector configuration
           
          module: intelmq.bots.collectors.mail.collector_mail_attach\nparameters:\n  attach_regex: events.csv\n  extract_files: False\n  folder: INBOX\n  mail_host: __HOST__\n  mail_password: __PASSWORD__\n  mail_ssl: True\n  mail_user: __USERNAME__\n  name: CB-Report Malware infections via IMAP\n  provider: CERT-Bund\n  rate_limit: 86400\n  subject_regex: ^\\\\[CB-Report#.* Malware infections (\\\\(Avalanche\\\\) )?in country\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.generic.parser_csv\nparameters:\n  columns: [\"source.asn\", \"source.ip\", \"time.source\", \"classification.type\", \"malware.name\", \"source.port\", \"destination.ip\", \"destination.port\", \"destination.fqdn\", \"protocol.transport\"]\n  default_url_protocol: http://\n  defaults_fields: {'classification.type': 'infected-system'}\n  delimiter: ,\n  skip_header: True\n  time_format: from_format|%Y-%m-%d %H:%M:%S\n
          "},{"location":"user/feeds/#certpl","title":"CERT.PL","text":""},{"location":"user/feeds/#n6-stomp-stream","title":"N6 Stomp Stream","text":"
          N6 Collector - CERT.pl's n6 Stream API feed (via STOMP interface). Note that 'rate_limit' does not apply to this bot, as it is waiting for messages on a stream.
           
          Public: no
           
          Revision: 2023-10-08
           
          Documentation: https://n6.readthedocs.io/usage/streamapi/
           
          Additional Information: Contact CERT.pl to get access to the feed. Note that the configuration parameter values suggested here are suitable for the new n6 Stream API variant (with authentication based on 'username' and 'password'); for this variant, typically you can leave the 'ssl_ca_certificate' parameter's value empty - then the system's default CA certificates will be used; however, if that does not work, you need to set 'ssl_ca_certificate' to the path to a file containing CA certificates eligible to verify \".cert.pl\" server certificates (to be found among the publicly available CA certs distributed with modern web browsers/OSes). Also, note that the 'server' parameter's value (for the new API variant) suggested here, \"n6stream-new.cert.pl\", is a temporary domain; ultimately, it will be changed back to \"stream.cert.pl\". When it comes to the old API variant (turned off in November 2023!), you need to have the 'server' parameter set to the name \"n6stream.cert.pl\", 'auth_by_ssl_client_certificate' set to true, 'ssl_ca_certificate' set to the path to a file containing the n6's legacy self-signed CA certificate (which is stored in file \"intelmq/bots/collectors/stomp/ca.pem\"), and the parameters 'ssl_client_certificate' and 'ssl_client_certificate_key' set to the paths to your-n6*-client-specific certificate and key files (note that the 'username' and 'password' parameters are then irrelevant and can be omitted).
           
          Collector configuration
           
          module: intelmq.bots.collectors.stomp.collector\nparameters:\n  auth_by_ssl_client_certificate: False\n  exchange: {insert your STOMP *destination* to subscribe to, as given by CERT.pl, e.g. /exchange/my.example.org/*.*.*.*}\n  name: N6 Stomp Stream\n  password: {insert your *n6* API key}\n  port: 61614\n  provider: CERT.PL\n  server: n6stream-new.cert.pl\n  ssl_ca_certificate: \n  username: {insert your *n6* login, e.g. someuser@my.example.org}\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.n6.parser_n6stomp\n
          "},{"location":"user/feeds/#cins-army","title":"CINS Army","text":""},{"location":"user/feeds/#cins-army-list","title":"CINS Army List","text":"
          The CINS Army (CIArmy.com) list is a subset of the CINS Active Threat Intelligence ruleset, and consists of IP addresses that meet one of two basic criteria: 1) The IP's recent Rogue Packet score factor is very poor, or 2) The IP has tripped a designated number of 'trusted' alerts across a given number of our Sentinels deployed around the world.
           
          Public: yes
           
          Revision: 2018-01-20
           
          Documentation: https://cinsscore.com/#list
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: http://cinsscore.com/list/ci-badguys.txt\n  name: CINS Army List\n  provider: CINS Army\n  rate_limit: 3600\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.ci_army.parser\n
          "},{"location":"user/feeds/#cznic","title":"CZ.NIC","text":""},{"location":"user/feeds/#haas","title":"HaaS","text":"
          SSH attackers against HaaS (Honeypot as a Service) provided by CZ.NIC, z.s.p.o. The dump is published once a day.
           
          Public: yes
           
          Revision: 2020-07-22
           
          Documentation: https://haas.nic.cz/
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  extract_files: True\n  http_url: https://haas.nic.cz/stats/export/{time[%Y/%m/%Y-%m-%d]}.json.gz\n  http_url_formatting: {'days': -1}\n  rate_limit: 86400\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.cznic.parser_haas\n
          "},{"location":"user/feeds/#proki","title":"Proki","text":"
          Aggregation of various sources on malicious IP addresses (malware spreaders or C&C servers).
           
          Public: no
           
          Revision: 2020-08-17
           
          Documentation: https://csirt.cz/en/proki/
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://proki.csirt.cz/api/1/__APIKEY__/data/day/{time[%Y/%m/%d]}\n  http_url_formatting: {'days': -1}\n  name: Proki\n  provider: CZ.NIC\n  rate_limit: 86400\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.cznic.parser_proki\n
          "},{"location":"user/feeds/#calidog","title":"Calidog","text":""},{"location":"user/feeds/#certstream","title":"CertStream","text":"
          HTTP Websocket Stream from certstream.calidog.io providing data from Certificate Transparency Logs.
           
          Public: yes
           
          Revision: 2018-06-15
           
          Documentation: https://medium.com/cali-dog-security/introducing-certstream-3fc13bb98067
           
          Additional Information: Be aware that this feed provides a lot of data and may overload your system quickly.
           
          Collector configuration
           
          module: intelmq.bots.collectors.calidog.collector_certstream\nparameters:\n  name: CertStream\n  provider: Calidog\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.calidog.parser_certstream\n
          "},{"location":"user/feeds/#cleanmx","title":"CleanMX","text":""},{"location":"user/feeds/#phishing","title":"Phishing","text":"
          In order to download the CleanMX feed you need to use a custom user agent and register that user agent.
           
          Public: no
           
          Revision: 2018-01-20
           
          Documentation: http://clean-mx.de/
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_timeout_sec: 120\n  http_url: http://support.clean-mx.de/clean-mx/xmlphishing?response=alive&domain=\n  http_user_agent: {{ your user agent }}\n  name: Phishing\n  provider: CleanMX\n  rate_limit: 86400\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.cleanmx.parser\n
          "},{"location":"user/feeds/#virus","title":"Virus","text":"
          In order to download the CleanMX feed you need to use a custom user agent and register that user agent.
           
          Public: no
           
          Revision: 2018-01-20
           
          Documentation: http://clean-mx.de/
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_timeout_sec: 120\n  http_url: http://support.clean-mx.de/clean-mx/xmlviruses?response=alive&domain=\n  http_user_agent: {{ your user agent }}\n  name: Virus\n  provider: CleanMX\n  rate_limit: 86400\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.cleanmx.parser\n
          "},{"location":"user/feeds/#cybercrime-tracker","title":"CyberCrime Tracker","text":""},{"location":"user/feeds/#latest","title":"Latest","text":"
          C2 servers
           
          Public: yes
           
          Revision: 2019-03-19
           
          Documentation: https://cybercrime-tracker.net/index.php
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://cybercrime-tracker.net/index.php\n  name: Latest\n  provider: CyberCrime Tracker\n  rate_limit: 86400\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.html_table.parser\nparameters:\n  columns: [\"time.source\", \"source.url\", \"source.ip\", \"malware.name\", \"__IGNORE__\"]\n  default_url_protocol: http://\n  defaults_fields: {'classification.type': 'c2-server'}\n  skip_table_head: True\n
          "},{"location":"user/feeds/#dshield","title":"DShield","text":""},{"location":"user/feeds/#as-details","title":"AS Details","text":"
          No description provided by feed provider.
           
          Public: yes
           
          Revision: 2018-01-20
           
          Documentation: https://www.dshield.org/reports.html
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://dshield.org/asdetailsascii.html?as={{ AS Number }}\n  name: AS Details\n  provider: DShield\n  rate_limit: 86400\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.dshield.parser_asn\n
          "},{"location":"user/feeds/#block","title":"Block","text":"
          This list summarizes the top 20 attacking class C (/24) subnets over the last three days. The number of 'attacks' indicates the number of targets reporting scans from this subnet.
           
          Public: yes
           
          Revision: 2018-01-20
           
          Documentation: https://www.dshield.org/reports.html
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://www.dshield.org/block.txt\n  name: Block\n  provider: DShield\n  rate_limit: 86400\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.dshield.parser_block\n
          "},{"location":"user/feeds/#danger-rulez","title":"Danger Rulez","text":""},{"location":"user/feeds/#bruteforce-blocker","title":"Bruteforce Blocker","text":"
          Its main purpose is to block SSH bruteforce attacks via firewall.
           
          Public: yes
           
          Revision: 2018-01-20
           
          Documentation: http://danger.rulez.sk/index.php/bruteforceblocker/
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: http://danger.rulez.sk/projects/bruteforceblocker/blist.php\n  name: Bruteforce Blocker\n  provider: Danger Rulez\n  rate_limit: 3600\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.danger_rulez.parser\n
          "},{"location":"user/feeds/#dataplane","title":"Dataplane","text":""},{"location":"user/feeds/#dns-recursion-desired","title":"DNS Recursion Desired","text":"
          Entries consist of fields with identifying characteristics of a source IP address that has been seen performing a DNS recursion desired query to a remote host. This report lists hosts that are suspicious of more than just port scanning. The host may be DNS server cataloging or searching for hosts to use for DNS-based DDoS amplification.
           
          Public: yes
           
          Revision: 2021-09-09
           
          Documentation: https://dataplane.org/
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://dataplane.org/dnsrd.txt\n  name: DNS Recursion Desired\n  provider: Dataplane\n  rate_limit: 3600\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.dataplane.parser\n
          "},{"location":"user/feeds/#dns-recursion-desired-any","title":"DNS Recursion Desired ANY","text":"
          Entries consist of fields with identifying characteristics of a source IP address that has been seen performing a DNS recursion desired IN ANY query to a remote host. This report lists hosts that are suspicious of more than just port scanning. The host may be DNS server cataloging or searching for hosts to use for DNS-based DDoS amplification.
           
          Public: yes
           
          Revision: 2021-09-09
           
          Documentation: https://dataplane.org/
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://dataplane.org/dnsrdany.txt\n  name: DNS Recursion Desired ANY\n  provider: Dataplane\n  rate_limit: 3600\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.dataplane.parser\n
          "},{"location":"user/feeds/#dns-version","title":"DNS Version","text":"
          Entries consist of fields with identifying characteristics of a source IP address that has been seen performing a DNS CH TXT version.bind query to a remote host. This report lists hosts that are suspicious of more than just port scanning. The host may be DNS server cataloging or searching for vulnerable DNS servers.
           
          Public: yes
           
          Revision: 2021-09-09
           
          Documentation: https://dataplane.org/
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://dataplane.org/dnsversion.txt\n  name: DNS Version\n  provider: Dataplane\n  rate_limit: 3600\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.dataplane.parser\n
          "},{"location":"user/feeds/#protocol-41","title":"Protocol 41","text":"
          Entries consist of fields with identifying characteristics of a host that has been detected to offer open IPv6 over IPv4 tunneling. This could allow for the host to be used a public proxy against IPv6 hosts.
           
          Public: yes
           
          Revision: 2021-09-09
           
          Documentation: https://dataplane.org/
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://dataplane.org/proto41.txt\n  name: Protocol 41\n  provider: Dataplane\n  rate_limit: 3600\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.dataplane.parser\n
          "},{"location":"user/feeds/#sip-query","title":"SIP Query","text":"
          Entries consist of fields with identifying characteristics of a source IP address that has been seen initiating a SIP OPTIONS query to a remote host. This report lists hosts that are suspicious of more than just port scanning. The hosts may be SIP server cataloging or conducting various forms of telephony abuse. Report is updated hourly.
           
          Public: yes
           
          Revision: 2018-01-20
           
          Documentation: https://dataplane.org/
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://dataplane.org/sipquery.txt\n  name: SIP Query\n  provider: Dataplane\n  rate_limit: 3600\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.dataplane.parser\n
          "},{"location":"user/feeds/#sip-registration","title":"SIP Registration","text":"
          Entries consist of fields with identifying characteristics of a source IP address that has been seen initiating a SIP REGISTER operation to a remote host. This report lists hosts that are suspicious of more than just port scanning. The hosts may be SIP client cataloging or conducting various forms of telephony abuse. Report is updated hourly.
           
          Public: yes
           
          Revision: 2018-01-20
           
          Documentation: https://dataplane.org/
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://dataplane.org/sipregistration.txt\n  name: SIP Registration\n  provider: Dataplane\n  rate_limit: 3600\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.dataplane.parser\n
          "},{"location":"user/feeds/#smtp-data","title":"SMTP Data","text":"
          Entries consist of fields with identifying characteristics of a host that has been seen initiating a SMTP DATA operation to a remote host. The source report lists hosts that are suspicious of more than just port scanning. The host may be SMTP server cataloging or conducting various forms of email abuse.
           
          Public: yes
           
          Revision: 2021-09-09
           
          Documentation: https://dataplane.org/
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://dataplane.org/smtpdata.txt\n  name: SMTP Data\n  provider: Dataplane\n  rate_limit: 3600\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.dataplane.parser\n
          "},{"location":"user/feeds/#smtp-greet","title":"SMTP Greet","text":"
          Entries consist of fields with identifying characteristics of a host that has been seen initiating a SMTP HELO/EHLO operation to a remote host. The source report lists hosts that are suspicious of more than just port scanning. The host may be SMTP server cataloging or conducting various forms of email abuse.
           
          Public: yes
           
          Revision: 2021-09-09
           
          Documentation: https://dataplane.org/
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://dataplane.org/smtpgreet.txt\n  name: SMTP Greet\n  provider: Dataplane\n  rate_limit: 3600\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.dataplane.parser\n
          "},{"location":"user/feeds/#ssh-client-connection","title":"SSH Client Connection","text":"
          Entries below consist of fields with identifying characteristics of a source IP address that has been seen initiating an SSH connection to a remote host. This report lists hosts that are suspicious of more than just port scanning. The hosts may be SSH server cataloging or conducting authentication attack attempts. Report is updated hourly.
           
          Public: yes
           
          Revision: 2018-01-20
           
          Documentation: https://dataplane.org/
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://dataplane.org/sshclient.txt\n  name: SSH Client Connection\n  provider: Dataplane\n  rate_limit: 3600\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.dataplane.parser\n
          "},{"location":"user/feeds/#ssh-password-authentication","title":"SSH Password Authentication","text":"
          Entries below consist of fields with identifying characteristics of a source IP address that has been seen attempting to remotely login to a host using SSH password authentication. The report lists hosts that are highly suspicious and are likely conducting malicious SSH password authentication attacks. Report is updated hourly.
           
          Public: yes
           
          Revision: 2018-01-20
           
          Documentation: https://dataplane.org/
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://dataplane.org/sshpwauth.txt\n  name: SSH Password Authentication\n  provider: Dataplane\n  rate_limit: 3600\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.dataplane.parser\n
          "},{"location":"user/feeds/#telnet-login","title":"Telnet Login","text":"
          Entries consist of fields with identifying characteristics of a host that has been seen initiating a telnet connection to a remote host. The source report lists hosts that are suspicious of more than just port scanning. The host may be telnet server cataloging or conducting authentication attack attempts.
           
          Public: yes
           
          Revision: 2021-09-09
           
          Documentation: https://dataplane.org/
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://dataplane.org/telnetlogin.txt\n  name: Telnet Login\n  provider: Dataplane\n  rate_limit: 3600\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.dataplane.parser\n
          "},{"location":"user/feeds/#vncrfb-login","title":"VNC/RFB Login","text":"
          Entries consist of fields with identifying characteristics of a host that has been seen initiating a VNC remote buffer session to a remote host. The source report lists hosts that are suspicious of more than just port scanning. The host may be VNC/RFB server cataloging or conducting authentication attack attempts.
           
          Public: yes
           
          Revision: 2021-09-09
           
          Documentation: https://dataplane.org/
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://dataplane.org/vncrfb.txt\n  name: VNC/RFB Login\n  provider: Dataplane\n  rate_limit: 3600\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.dataplane.parser\n
          "},{"location":"user/feeds/#eset","title":"ESET","text":""},{"location":"user/feeds/#eti-domains","title":"ETI Domains","text":"
          Domain data from ESET's TAXII API.
           
          Public: no
           
          Revision: 2020-06-30
           
          Documentation: https://www.eset.com/int/business/services/threat-intelligence/
           
          Collector configuration
           
          module: intelmq.bots.collectors.eset.collector\nparameters:\n  collection: ei.domains v2 (json)\n  endpoint: eti.eset.com\n  password: <password>\n  time_delta: 3600\n  username: <username>\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.eset.parser\n
          "},{"location":"user/feeds/#eti-urls","title":"ETI URLs","text":"
          URL data from ESET's TAXII API.
           
          Public: no
           
          Revision: 2020-06-30
           
          Documentation: https://www.eset.com/int/business/services/threat-intelligence/
           
          Collector configuration
           
          module: intelmq.bots.collectors.eset.collector\nparameters:\n  collection: ei.urls (json)\n  endpoint: eti.eset.com\n  password: <password>\n  time_delta: 3600\n  username: <username>\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.eset.parser\n
          "},{"location":"user/feeds/#fireeye","title":"Fireeye","text":""},{"location":"user/feeds/#malware-analysis-system","title":"Malware Analysis System","text":"
          Process data from Fireeye mail and file analysis appliances. SHA1 and MD5 malware hashes are extracted and if there is network communication, also URLs and domains.
           
          Public: no
           
          Revision: 2021-05-03
           
          Documentation: https://www.fireeye.com/products/malware-analysis.html
           
          Collector configuration
           
          module: intelmq.bots.collectors.fireeye.collector_mas\nparameters:\n  host: <hostname of your appliance>\n  http_password: <your password>\n  http_username: <your username>\n  request_duration: <how old date should be fetched eg 24_hours or 48_hours>\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.fireeye.parser\n
          "},{"location":"user/feeds/#fraunhofer","title":"Fraunhofer","text":""},{"location":"user/feeds/#dga-archive","title":"DGA Archive","text":"
          Fraunhofer DGA collector fetches data from Fraunhofer's domain generation archive.
           
          Public: no
           
          Revision: 2018-01-20
           
          Documentation: https://dgarchive.caad.fkie.fraunhofer.de/welcome/
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_password: {{ your password }}\n  http_url: https://dgarchive.caad.fkie.fraunhofer.de/today\n  http_username: {{ your username }}\n  name: DGA Archive\n  provider: Fraunhofer\n  rate_limit: 10800\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.fraunhofer.parser_dga\n
          "},{"location":"user/feeds/#have-i-been-pwned","title":"Have I Been Pwned","text":""},{"location":"user/feeds/#enterprise-callback","title":"Enterprise Callback","text":"
          With the Enterprise Subscription of 'Have I Been Pwned' you are able to provide a callback URL and any new leak data is submitted to it. It is recommended to put a webserver with Authorization check, TLS etc. in front of the API collector.
           
          Public: no
           
          Revision: 2019-09-11
           
          Documentation: https://haveibeenpwned.com/EnterpriseSubscriber/
           
          Additional Information: A minimal nginx configuration could look like: 
          server {\n    listen 443 ssl http2;\n    server_name [your host name];\n    client_max_body_size 50M;\n\n    ssl_certificate [path to your key];\n    ssl_certificate_key [path to your certificate];\n\n    location /[your private url] {\n         if ($http_authorization != '[your private password]') {\n             return 403;\n         }\n         proxy_pass http://localhost:5001/intelmq/push;\n         proxy_read_timeout 30;\n         proxy_connect_timeout 30;\n     }\n}\n
           
          Collector configuration
           
          module: intelmq.bots.collectors.api.collector_api\nparameters:\n  name: Enterprise Callback\n  port: 5001\n  provider: Have I Been Pwned\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.hibp.parser_callback\n
          "},{"location":"user/feeds/#malwarepatrol","title":"MalwarePatrol","text":""},{"location":"user/feeds/#dansguardian","title":"DansGuardian","text":"
          Malware block list with URLs
           
          Public: no
           
          Revision: 2018-01-20
           
          Documentation: https://www.malwarepatrol.net/non-commercial/
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://lists.malwarepatrol.net/cgi/getfile?receipt={{ your API key }}&product=8&list=dansguardian\n  name: DansGuardian\n  provider: MalwarePatrol\n  rate_limit: 180000\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.malwarepatrol.parser_dansguardian\n
          "},{"location":"user/feeds/#malwareurl","title":"MalwareURL","text":""},{"location":"user/feeds/#latest-malicious-activity","title":"Latest malicious activity","text":"
          Latest malicious domains/IPs.
           
          Public: yes
           
          Revision: 2018-02-05
           
          Documentation: https://www.malwareurl.com/
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://www.malwareurl.com/\n  name: Latest malicious activity\n  provider: MalwareURL\n  rate_limit: 86400\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.malwareurl.parser\n
          "},{"location":"user/feeds/#mcafee-advanced-threat-defense","title":"McAfee Advanced Threat Defense","text":""},{"location":"user/feeds/#sandbox-reports","title":"Sandbox Reports","text":"
          Processes reports from McAfee's sandboxing solution via the openDXL API.
           
          Public: no
           
          Revision: 2018-07-05
           
          Documentation: https://www.mcafee.com/enterprise/en-us/products/advanced-threat-defense.html
           
          Collector configuration
           
          module: intelmq.bots.collectors.opendxl.collector\nparameters:\n  dxl_config_file: {{ location of dxl configuration file }}\n  dxl_topic: /mcafee/event/atd/file/report\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.mcafee.parser_atd\nparameters:\n  verdict_severity: 4\n
          "},{"location":"user/feeds/#microsoft","title":"Microsoft","text":""},{"location":"user/feeds/#bingmurls-via-interflow","title":"BingMURLs via Interflow","text":"
          Collects Malicious URLs detected by Bing from the Interflow API. The feed is available via Microsoft\u2019s Government Security Program (GSP).
           
          Public: no
           
          Revision: 2018-05-29
           
          Documentation: https://docs.microsoft.com/en-us/security/gsp/informationsharingandexchange
           
          Additional Information: Depending on the file sizes you may need to increase the parameter 'http_timeout_sec' of the collector.
           
          Collector configuration
           
          module: intelmq.bots.collectors.microsoft.collector_interflow\nparameters:\n  api_key: {{ your API key }}\n  file_match: ^bingmurls_\n  http_timeout_sec: 300\n  name: BingMURLs via Interflow\n  not_older_than: 2 days\n  provider: Microsoft\n  rate_limit: 3600\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.microsoft.parser_bingmurls\n
          "},{"location":"user/feeds/#ctip-c2-via-azure","title":"CTIP C2 via Azure","text":"
          Collects the CTIP C2 feed from a shared Azure Storage. The feed is available via Microsoft\u2019s Government Security Program (GSP).
           
          Public: no
           
          Revision: 2020-05-29
           
          Documentation: https://docs.microsoft.com/en-us/security/gsp/informationsharingandexchange
           
          Additional Information: The cache is needed for memorizing which files have already been processed, the TTL should be higher than the oldest file available in the storage (currently the last three days are available). The connection string contains endpoint as well as authentication information.
           
          Collector configuration
           
          module: intelmq.bots.collectors.microsoft.collector_azure\nparameters:\n  connection_string: {{ your connection string }}\n  container_name: ctip-c2\n  name: CTIP C2 via Azure\n  provider: Microsoft\n  rate_limit: 3600\n  redis_cache_db: 5\n  redis_cache_host: 127.0.0.1\n  redis_cache_port: 6379\n  redis_cache_ttl: 864000\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.microsoft.parser_ctip\n
          "},{"location":"user/feeds/#ctip-infected-via-azure","title":"CTIP Infected via Azure","text":"
          Collects the CTIP (Sinkhole data) from a shared Azure Storage. The feed is available via Microsoft\u2019s Government Security Program (GSP).
           
          Public: no
           
          Revision: 2022-06-01
           
          Documentation: https://docs.microsoft.com/en-us/security/gsp/informationsharingandexchange http://www.dcuctip.com/
           
          Additional Information: The cache is needed for memorizing which files have already been processed, the TTL should be higher than the oldest file available in the storage (currently the last three days are available). The connection string contains endpoint as well as authentication information. As many IPs occur very often in the data, you may want to use a deduplicator specifically for the feed. More information about the feed can be found on www.dcuctip.com after login with your GSP account.
           
          Collector configuration
           
          module: intelmq.bots.collectors.microsoft.collector_azure\nparameters:\n  connection_string: {{ your connection string }}\n  container_name: ctip-infected-summary\n  name: CTIP Infected via Azure\n  provider: Microsoft\n  rate_limit: 3600\n  redis_cache_db: 5\n  redis_cache_host: 127.0.0.1\n  redis_cache_port: 6379\n  redis_cache_ttl: 864000\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.microsoft.parser_ctip\n
          "},{"location":"user/feeds/#ctip-infected-via-interflow","title":"CTIP Infected via Interflow","text":"
          Collects the CTIP Infected feed (Sinkhole data for your country) files from the Interflow API.The feed is available via Microsoft\u2019s Government Security Program (GSP).
           
          Public: no
           
          Revision: 2018-03-06
           
          Documentation: https://docs.microsoft.com/en-us/security/gsp/informationsharingandexchange http://www.dcuctip.com/
           
          Additional Information: Depending on the file sizes you may need to increase the parameter 'http_timeout_sec' of the collector. As many IPs occur very often in the data, you may want to use a deduplicator specifically for the feed. More information about the feed can be found on www.dcuctip.com after login with your GSP account.
           
          Collector configuration
           
          module: intelmq.bots.collectors.microsoft.collector_interflow\nparameters:\n  api_key: {{ your API key }}\n  file_match: ^ctip_\n  http_timeout_sec: 300\n  name: CTIP Infected via Interflow\n  not_older_than: 2 days\n  provider: Microsoft\n  rate_limit: 3600\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.microsoft.parser_ctip\n
          "},{"location":"user/feeds/#openphish","title":"OpenPhish","text":""},{"location":"user/feeds/#premium-feed","title":"Premium Feed","text":"
          OpenPhish is a fully automated self-contained platform for phishing intelligence. It identifies phishing sites and performs intelligence analysis in real time without human intervention and without using any external resources, such as blacklists.
           
          Public: no
           
          Revision: 2018-02-06
           
          Documentation: https://www.openphish.com/phishing_feeds.html
           
          Additional Information: Discounts available for Government and National CERTs a well as for Nonprofit and Not-for-Profit organizations.
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_password: {{ your password }}\n  http_url: https://openphish.com/prvt-intell/\n  http_username: {{ your username }}\n  name: Premium Feed\n  provider: OpenPhish\n  rate_limit: 86400\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.openphish.parser_commercial\n
          "},{"location":"user/feeds/#public-feed","title":"Public feed","text":"
          OpenPhish is a fully automated self-contained platform for phishing intelligence. It identifies phishing sites and performs intelligence analysis in real time without human intervention and without using any external resources, such as blacklists.
           
          Public: yes
           
          Revision: 2018-01-20
           
          Documentation: https://www.openphish.com/
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://www.openphish.com/feed.txt\n  name: Public feed\n  provider: OpenPhish\n  rate_limit: 86400\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.openphish.parser\n
          "},{"location":"user/feeds/#phishtank","title":"PhishTank","text":""},{"location":"user/feeds/#online","title":"Online","text":"
          PhishTank is a collaborative clearing house for data and information about phishing on the Internet.
           
          Public: no
           
          Revision: 2022-11-21
           
          Documentation: https://www.phishtank.com/developer_info.php
           
          Additional Information: Updated hourly as per the documentation. Download is possible without API key, but limited to few downloads per day.
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  extract_files: True\n  http_url: https://data.phishtank.com/data/{{ your API key }}/online-valid.json.gz\n  name: Online\n  provider: PhishTank\n  rate_limit: 3600\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.phishtank.parser\n
          "},{"location":"user/feeds/#precisionsec","title":"PrecisionSec","text":""},{"location":"user/feeds/#agent-tesla","title":"Agent Tesla","text":"
          Agent Tesla IoCs, URLs where the malware is hosted.
           
          Public: yes
           
          Revision: 2019-04-02
           
          Documentation: https://precisionsec.com/threat-intelligence-feeds/agent-tesla/
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://precisionsec.com/threat-intelligence-feeds/agent-tesla/\n  name: Agent Tesla\n  provider: PrecisionSec\n  rate_limit: 86400\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.html_table.parser\nparameters:\n  columns: [\"source.ip|source.url\", \"time.source\"]\n  default_url_protocol: http://\n  defaults_fields: {'classification.type': 'malware-distribution'}\n  skip_table_head: True\n
          "},{"location":"user/feeds/#shadowserver","title":"Shadowserver","text":""},{"location":"user/feeds/#via-api","title":"Via API","text":"
          Shadowserver sends out a variety of reports to subscribers, see documentation.
           
          Public: no
           
          Revision: 2020-01-08
           
          Documentation: https://www.shadowserver.org/what-we-do/network-reporting/api-documentation/
           
          Additional Information: This configuration fetches user-configurable reports from the Shadowserver Reports API. For a list of reports, have a look at the Shadowserver collector and parser documentation.
           
          Collector configuration
           
          module: intelmq.bots.collectors.shadowserver.collector_reports_api\nparameters:\n  api_key: <API key>\n  country: <CC>\n  rate_limit: 86400\n  redis_cache_db: 12\n  redis_cache_host: 127.0.0.1\n  redis_cache_port: 6379\n  redis_cache_ttl: 864000\n  secret: <API secret>\n  types: <single report or list of reports>\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.shadowserver.parser_json\n
          "},{"location":"user/feeds/#via-imap","title":"Via IMAP","text":"
          Shadowserver sends out a variety of reports (see https://www.shadowserver.org/wiki/pmwiki.php/Services/Reports).
           
          Public: no
           
          Revision: 2018-01-20
           
          Documentation: https://www.shadowserver.org/what-we-do/network-reporting/
           
          Additional Information: The configuration retrieves the data from a e-mails via IMAP from the attachments.
           
          Collector configuration
           
          module: intelmq.bots.collectors.mail.collector_mail_attach\nparameters:\n  attach_regex: csv.zip\n  extract_files: True\n  folder: INBOX\n  mail_host: __HOST__\n  mail_password: __PASSWORD__\n  mail_ssl: True\n  mail_user: __USERNAME__\n  name: Via IMAP\n  provider: Shadowserver\n  rate_limit: 86400\n  subject_regex: __REGEX__\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.shadowserver.parser\n
          "},{"location":"user/feeds/#via-request-tracker","title":"Via Request Tracker","text":"
          Shadowserver sends out a variety of reports (see https://www.shadowserver.org/wiki/pmwiki.php/Services/Reports).
           
          Public: no
           
          Revision: 2018-01-20
           
          Documentation: https://www.shadowserver.org/what-we-do/network-reporting/
           
          Additional Information: The configuration retrieves the data from a RT/RTIR ticketing instance via the attachment or an download.
           
          Collector configuration
           
          module: intelmq.bots.collectors.rt.collector_rt\nparameters:\n  attachment_regex: \\\\.csv\\\\.zip$\n  extract_attachment: True\n  extract_download: False\n  http_password: {{ your HTTP Authentication password or null }}\n  http_username: {{ your HTTP Authentication username or null }}\n  password: __PASSWORD__\n  provider: Shadowserver\n  rate_limit: 3600\n  search_not_older_than: {{ relative time or null }}\n  search_owner: nobody\n  search_queue: Incident Reports\n  search_requestor: autoreports@shadowserver.org\n  search_status: new\n  search_subject_like: \\[__COUNTRY__\\] Shadowserver __COUNTRY__\n  set_status: open\n  take_ticket: True\n  uri: http://localhost/rt/REST/1.0\n  url_regex: https://dl.shadowserver.org/[a-zA-Z0-9?_-]*\n  user: __USERNAME__\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.shadowserver.parser\n
          "},{"location":"user/feeds/#shodan","title":"Shodan","text":""},{"location":"user/feeds/#country-stream","title":"Country Stream","text":"
          Collects the Shodan stream for one or multiple countries from the Shodan API.
           
          Public: no
           
          Revision: 2021-03-22
           
          Documentation: https://developer.shodan.io/api/stream
           
          Additional Information: A Shodan account with streaming permissions is needed.
           
          Collector configuration
           
          module: intelmq.bots.collectors.shodan.collector_stream\nparameters:\n  api_key: <API key>\n  countries: <comma-separated list of country codes>\n  error_retry_delay: 0\n  name: Country Stream\n  provider: Shodan\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.shodan.parser\nparameters:\n  error_retry_delay: 0\n  ignore_errors: False\n  minimal_mode: False\n
          "},{"location":"user/feeds/#spamhaus","title":"Spamhaus","text":""},{"location":"user/feeds/#asn-drop","title":"ASN Drop","text":"
          ASN-DROP contains a list of Autonomous System Numbers controlled by spammers or cyber criminals, as well as \"hijacked\" ASNs. ASN-DROP can be used to filter BGP routes which are being used for malicious purposes.
           
          Public: yes
           
          Revision: 2018-01-20
           
          Documentation: https://www.spamhaus.org/drop/
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://www.spamhaus.org/drop/asndrop.txt\n  name: ASN Drop\n  provider: Spamhaus\n  rate_limit: 3600\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.spamhaus.parser_drop\n
          "},{"location":"user/feeds/#cert","title":"CERT","text":"
          Spamhaus CERT Insight Portal. Access limited to CERTs and CSIRTs with national or regional responsibility. .
           
          Public: no
           
          Revision: 2018-01-20
           
          Documentation: https://www.spamhaus.org/news/article/705/spamhaus-launches-cert-insight-portal
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: {{ your CERT portal URL }}\n  name: CERT\n  provider: Spamhaus\n  rate_limit: 3600\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.spamhaus.parser_cert\n
          "},{"location":"user/feeds/#drop","title":"Drop","text":"
          The DROP list will not include any IP address space under the control of any legitimate network - even if being used by \"the spammers from hell\". DROP will only include netblocks allocated directly by an established Regional Internet Registry (RIR) or National Internet Registry (NIR) such as ARIN, RIPE, AFRINIC, APNIC, LACNIC or KRNIC or direct RIR allocations.
           
          Public: yes
           
          Revision: 2018-01-20
           
          Documentation: https://www.spamhaus.org/drop/
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://www.spamhaus.org/drop/drop.txt\n  name: Drop\n  provider: Spamhaus\n  rate_limit: 3600\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.spamhaus.parser_drop\n
          "},{"location":"user/feeds/#dropv6","title":"Dropv6","text":"
          The DROPv6 list includes IPv6 ranges allocated to spammers or cyber criminals. DROPv6 will only include IPv6 netblocks allocated directly by an established Regional Internet Registry (RIR) or National Internet Registry (NIR) such as ARIN, RIPE, AFRINIC, APNIC, LACNIC or KRNIC or direct RIR allocations.
           
          Public: yes
           
          Revision: 2018-01-20
           
          Documentation: https://www.spamhaus.org/drop/
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://www.spamhaus.org/drop/dropv6.txt\n  name: Dropv6\n  provider: Spamhaus\n  rate_limit: 3600\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.spamhaus.parser_drop\n
          "},{"location":"user/feeds/#edrop","title":"EDrop","text":"
          EDROP is an extension of the DROP list that includes sub-allocated netblocks controlled by spammers or cyber criminals. EDROP is meant to be used in addition to the direct allocations on the DROP list.
           
          Public: yes
           
          Revision: 2018-01-20
           
          Documentation: https://www.spamhaus.org/drop/
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://www.spamhaus.org/drop/edrop.txt\n  name: EDrop\n  provider: Spamhaus\n  rate_limit: 3600\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.spamhaus.parser_drop\n
          "},{"location":"user/feeds/#strangereal-intel","title":"Strangereal Intel","text":""},{"location":"user/feeds/#dailyioc","title":"DailyIOC","text":"
          Daily IOC from tweets and articles
           
          Public: yes
           
          Revision: 2019-12-05
           
          Documentation: https://github.com/StrangerealIntel/DailyIOC
           
          Additional Information: collector's extra_fields parameter may be any of fields from the github content API response <https://developer.github.com/v3/repos/contents/>_
           
          Collector configuration
           
          module: intelmq.bots.collectors.github_api.collector_github_contents_api\nparameters:\n  personal_access_token: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token\n  regex: .*.json\n  repository: StrangerealIntel/DailyIOC\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.github_feed\n
          "},{"location":"user/feeds/#surbl","title":"Surbl","text":""},{"location":"user/feeds/#malicious-domains","title":"Malicious Domains","text":"
          Detected malicious domains. Note that you have to opened up Sponsored Datafeed Service (SDS) access to the SURBL data via rsync for your IP address.
           
          Public: no
           
          Revision: 2018-09-04
           
          Collector configuration
           
          module: intelmq.bots.collectors.rsync.collector_rsync\nparameters:\n  file: wild.surbl.org.rbldnsd\n  rsync_path: blacksync.prolocation.net::surbl-wild/\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.surbl.parser\n
          "},{"location":"user/feeds/#team-cymru","title":"Team Cymru","text":""},{"location":"user/feeds/#cap","title":"CAP","text":"
          Team Cymru provides daily lists of compromised or abused devices for the ASNs and/or netblocks with a CSIRT's jurisdiction. This includes such information as bot infected hosts, command and control systems, open resolvers, malware urls, phishing urls, and brute force attacks
           
          Public: no
           
          Revision: 2018-01-20
           
          Documentation: https://www.team-cymru.com/CSIRT-AP.html https://www.cymru.com/$certname/report_info.txt
           
          Additional Information: \"Two feeds types are offered:  * The new https://www.cymru.com/$certname/$certname_{time[%Y%m%d]}.txt  * and the old https://www.cymru.com/$certname/infected_{time[%Y%m%d]}.txt  Both formats are supported by the parser and the new one is recommended.  As of 2019-09-12 the old format will be retired soon.\"
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_password: {{ your password }}\n  http_url: https://www.cymru.com/$certname/$certname_{time[%Y%m%d]}.txt\n  http_url_formatting: True\n  http_username: {{ your username }}\n  name: CAP\n  provider: Team Cymru\n  rate_limit: 86400\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.cymru.parser_cap_program\n
          "},{"location":"user/feeds/#full-bogons-ipv4","title":"Full Bogons IPv4","text":"
          Fullbogons are a larger set which also includes IP space that has been allocated to an RIR, but not assigned by that RIR to an actual ISP or other end-user. IANA maintains a convenient IPv4 summary page listing allocated and reserved netblocks, and each RIR maintains a list of all prefixes that they have assigned to end-users. Our bogon reference pages include additional links and resources to assist those who wish to properly filter bogon prefixes within their networks.
           
          Public: yes
           
          Revision: 2018-01-20
           
          Documentation: https://www.team-cymru.com/bogon-reference-http.html
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://www.team-cymru.org/Services/Bogons/fullbogons-ipv4.txt\n  name: Full Bogons IPv4\n  provider: Team Cymru\n  rate_limit: 86400\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.cymru.parser_full_bogons\n
          "},{"location":"user/feeds/#full-bogons-ipv6","title":"Full Bogons IPv6","text":"
          Fullbogons are a larger set which also includes IP space that has been allocated to an RIR, but not assigned by that RIR to an actual ISP or other end-user. IANA maintains a convenient IPv4 summary page listing allocated and reserved netblocks, and each RIR maintains a list of all prefixes that they have assigned to end-users. Our bogon reference pages include additional links and resources to assist those who wish to properly filter bogon prefixes within their networks.
           
          Public: yes
           
          Revision: 2018-01-20
           
          Documentation: https://www.team-cymru.com/bogon-reference-http.html
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://www.team-cymru.org/Services/Bogons/fullbogons-ipv6.txt\n  name: Full Bogons IPv6\n  provider: Team Cymru\n  rate_limit: 86400\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.cymru.parser_full_bogons\n
          "},{"location":"user/feeds/#threatminer","title":"Threatminer","text":""},{"location":"user/feeds/#recent-domains","title":"Recent domains","text":"
          Latest malicious domains.
           
          Public: yes
           
          Revision: 2018-02-06
           
          Documentation: https://www.threatminer.org/
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://www.threatminer.org/\n  name: Recent domains\n  provider: Threatminer\n  rate_limit: 86400\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.threatminer.parser\n
          "},{"location":"user/feeds/#turris","title":"Turris","text":""},{"location":"user/feeds/#greylist","title":"Greylist","text":"
          The data are processed and classified every week and behaviour of IP addresses that accessed a larger number of Turris routers is evaluated. The result is a list of addresses that have tried to obtain information about services on the router or tried to gain access to them. The list also contains a list of tags for each address which indicate what behaviour of the address was observed.
           
          Public: yes
           
          Revision: 2023-06-13
           
          Documentation: https://project.turris.cz/en/greylist
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://view.sentinel.turris.cz/greylist-data/greylist-latest.csv\n  name: Greylist\n  provider: Turris\n  rate_limit: 43200\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.turris.parser\n
          "},{"location":"user/feeds/#greylist-with-pgp-signature-verification","title":"Greylist with PGP signature verification","text":"
          The data are processed and classified every week and behaviour of IP addresses that accessed a larger number of Turris routers is evaluated. The result is a list of addresses that have tried to obtain information about services on the router or tried to gain access to them. The list also contains a list of tags for each address which indicate what behaviour of the address was observed.
           
          The Turris Greylist feed provides PGP signatures for the provided files. You will need to import the public PGP key from the linked documentation page, currently available at https://pgp.mit.edu/pks/lookup?op=vindex&search=0x10876666 or from below. See the URL Fetcher Collector documentation for more information on PGP signature verification.
           
          PGP Public key:
           
          -----BEGIN PGP PUBLIC KEY BLOCK-----\nVersion: SKS 1.1.6\nComment: Hostname: pgp.mit.edu\n\nmQINBFRl7D8BEADaRFoDa/+r27Gtqrdn8sZL4aSYTU4Q3gDr3TfigK8H26Un/Y79a/DUL1o0\no8SRae3uwVcjJDHZ6KDnxThbqF7URfpuCcCYxOs8p/eu3dSueqEGTODHWF4ChIh2japJDc4t\n3FQHbIh2e3GHotVqJGhvxMmWqBFoZ/mlWvhjs99FFBZ87qbUNk7l1UAGEXeWeECgz9nGox40\n3YpCgEsnJJsKC53y5LD/wBf4z+z0GsLg2GMRejmPRgrkSE/d9VjF/+niifAj2ZVFoINSVjjI\n8wQFc8qLiExdzwLdgc+ggdzk5scY3ugI5IBt1zflxMIOG4BxKj/5IWsnhKMG2NLVGUYOODoG\npKhcY0gCHypw1bmkp2m+BDVyg4KM2fFPgQ554DAX3xdukMCzzZyBxR3UdT4dN7xRVhpph3Y2\nAmh1E/dpde9uwKFk1oRHkRZ3UT1XtpbXtFNY0wCiGXPt6KznJAJcomYFkeLHjJo3nMK0hISV\nGSNetVLfNWlTkeo93E1innbSaDEN70H4jPivjdVjSrLtIGfr2IudUJI84dGmvMxssWuM2qdg\nFSzoTHw9UE9KT3SltKPS+F7u9x3h1J492YaVDncATRjPZUBDhbvo6Pcezhup7XTnI3gbRQc2\noEUDb933nwuobHm3VsUcf9686v6j8TYehsbjk+zdA4BoS/IdCwARAQABtC5UdXJyaXMgR3Jl\neWxpc3QgR2VuZXJhdG9yIDxncmV5bGlzdEB0dXJyaXMuY3o+iQI4BBMBAgAiBQJUZew/AhsD\nBgsJCAcDAgYVCAIJCgsEFgIDAQIeAQIXgAAKCRDAQrU3EIdmZoH4D/9Jo6j9RZxCAPTaQ9WZ\nWOdb1Eqd/206bObEX+xJAago+8vuy+waatHYBM9/+yxh0SIg2g5whd6J7A++7ePpt5XzX6hq\nbzdG8qGtsCRu+CpDJ40UwHep79Ck6O/A9KbZcZW1z/DhbYT3z/ZVWALy4RtgmyC67Vr+j/C7\nKNQ529bs3kP9AzvEIeBC4wdKl8dUSuZIPFbgf565zRNKLtHVgVhiuDPcxKmBEl4/PLYF30a9\n5Tgp8/PNa2qp1DV/EZjcsxvSRIZB3InGBvdKdSzvs4N/wLnKWedj1GGm7tJhSkJa4MLBSOIx\nyamhTS/3A5Cd1qoDhLkp7DGVXSdgEtpoZDC0jR7nTS6pXojcgQaF7SfJ3cjZaLI5rjsx0YLk\nG4PzonQKCAAQG1G9haCDniD8NrrkZ3eFiafoKEECRFETIG0BJHjPdSWcK9jtNCupBYb7JCiz\nQ0hwLh2wrw/wCutQezD8XfsBFFIQC18TsJAVgdHLZnGYkd5dIbV/1scOcm52w6EGIeMBBYlB\nJ2+JNukH5sJDA6zAXNl2I1H1eZsP4+FSNIfB6LdovHVPAjn7qXCw3+IonnQK8+g8YJkbbhKJ\nsPejfg+ndpe5u0zX+GvQCFBFu03muANA0Y/OOeGIQwU93d/akN0P1SRfq+bDXnkRIJQOD6XV\n0ZPKVXlNOjy/z2iN2A==\n=wjkM\n-----END PGP PUBLIC KEY BLOCK-----\n
           
          Public: yes
           
          Revision: 2018-01-20
           
          Documentation: https://project.turris.cz/en/greylist
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://www.turris.cz/greylist-data/greylist-latest.csv\n  name: Greylist\n  provider: Turris\n  rate_limit: 43200\n  signature_url: https://www.turris.cz/greylist-data/greylist-latest.csv.asc\n  verify_pgp_signatures: True\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.turris.parser\n
          "},{"location":"user/feeds/#university-of-toulouse","title":"University of Toulouse","text":""},{"location":"user/feeds/#blacklist","title":"Blacklist","text":"
          Various blacklist feeds
           
          Public: yes
           
          Revision: 2018-01-20
           
          Documentation: https://dsi.ut-capitole.fr/blacklists/
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  extract_files: true\n  http_url: https://dsi.ut-capitole.fr/blacklists/download/{collection name}.tar.gz\n  name: Blacklist\n  provider: University of Toulouse\n  rate_limit: 43200\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.generic.parser_csv\nparameters:\n  columns: {depends on a collection}\n  defaults_fields: {'classification.type': '{depends on a collection}'}\n  delimiter: false\n
          "},{"location":"user/feeds/#vxvault","title":"VXVault","text":""},{"location":"user/feeds/#urls","title":"URLs","text":"
          This feed provides IP addresses hosting Malware.
           
          Public: yes
           
          Revision: 2018-01-20
           
          Documentation: http://vxvault.net/ViriList.php
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: http://vxvault.net/URL_List.php\n  name: URLs\n  provider: VXVault\n  rate_limit: 3600\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.vxvault.parser\n
          "},{"location":"user/feeds/#viriback","title":"ViriBack","text":""},{"location":"user/feeds/#c2-tracker","title":"C2 Tracker","text":"
          Latest detected C2 servers.
           
          Public: yes
           
          Revision: 2022-11-15
           
          Documentation: https://viriback.com/
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: https://tracker.viriback.com/dump.php\n  name: C2 Tracker\n  provider: ViriBack\n  rate_limit: 86400\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.generic.csv_parser\nparameters:\n  columns: [\"malware.name\", \"source.url\", \"source.ip\", \"time.source\"]\n  defaults_fields: {'classification.type': 'malware-distribution'}\n  skip_header: True\n
          "},{"location":"user/feeds/#zoneh","title":"ZoneH","text":""},{"location":"user/feeds/#defacements","title":"Defacements","text":"
          all the information contained in Zone-H's cybercrime archive were either collected online from public sources or directly notified anonymously to us.
           
          Public: no
           
          Revision: 2018-01-20
           
          Documentation: https://zone-h.org/
           
          Collector configuration
           
          module: intelmq.bots.collectors.mail.collector_mail_attach\nparameters:\n  attach_regex: csv\n  extract_files: False\n  folder: INBOX\n  mail_host: __HOST__\n  mail_password: __PASSWORD__\n  mail_ssl: True\n  mail_user: __USERNAME__\n  name: Defacements\n  provider: ZoneH\n  rate_limit: 3600\n  sent_from: datazh@zone-h.org\n  subject_regex: Report\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.zoneh.parser\n
          "},{"location":"user/feeds/#capture","title":"cAPTure","text":""},{"location":"user/feeds/#ponmocup-domains-cif-format","title":"Ponmocup Domains CIF Format","text":"
          List of ponmocup malware redirection domains and infected web-servers from cAPTure. See also http://security-research.dyndns.org/pub/botnet-links.htm and http://c-apt-ure.blogspot.com/search/label/ponmocup The data in the CIF format is not equal to the Shadowserver CSV format. Reasons are unknown.
           
          Public: yes
           
          Revision: 2018-01-20
           
          Documentation: http://security-research.dyndns.org/pub/malware-feeds/
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: http://security-research.dyndns.org/pub/malware-feeds/ponmocup-infected-domains-CIF-latest.txt\n  name: Infected Domains\n  provider: cAPTure\n  rate_limit: 10800\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.dyn.parser\n
          "},{"location":"user/feeds/#ponmocup-domains-shadowserver-format","title":"Ponmocup Domains Shadowserver Format","text":"
          List of ponmocup malware redirection domains and infected web-servers from cAPTure. See also http://security-research.dyndns.org/pub/botnet-links.htm and http://c-apt-ure.blogspot.com/search/label/ponmocup The data in the Shadowserver CSV is not equal to the CIF format format. Reasons are unknown.
           
          Public: yes
           
          Revision: 2020-07-08
           
          Documentation: http://security-research.dyndns.org/pub/malware-feeds/
           
          Collector configuration
           
          module: intelmq.bots.collectors.http.collector_http\nparameters:\n  http_url: http://security-research.dyndns.org/pub/malware-feeds/ponmocup-infected-domains-shadowserver.csv\n  name: Infected Domains\n  provider: cAPTure\n  rate_limit: 10800\n
           
          Parser configuration
           
          module: intelmq.bots.parsers.generic.parser_csv\nparameters:\n  columns: [\"time.source\", \"source.ip\", \"source.fqdn\", \"source.urlpath\", \"source.port\", \"protocol.application\", \"extra.tag\", \"extra.redirect_target\", \"extra.category\"]\n  compose_fields: {'source.url': 'http://{0}{1}'}\n  defaults_fields: {'classification.type': 'malware-distribution'}\n  delimiter: ,\n  skip_header: True\n
          "},{"location":"user/intro/","title":"Intro","text":""},{"location":"user/intro/#intro","title":"Intro","text":"
          The User Guide provides information on how to use installed IntelMQ and it's components. Let's start with a basic not-so-technical description of how IntelMQ works and the used terminology:
           
             
          • It consists of small (python) programs called bots.
            •  
          • Bots communicate witch each other (using something called message broker) by passing so called events (JSON objects).
            •  
          • An example event can look like this:
            •  
           
          {\n    \"source.geolocation.cc\": \"JO\",\n    \"malware.name\": \"qakbot\",\n    \"source.ip\": \"82.212.115.188\",\n    \"source.asn\": 47887,\n    \"classification.type\": \"c2-server\",\n    \"extra.status\": \"offline\",\n    \"source.port\": 443,\n    \"classification.taxonomy\": \"malicious-code\",\n    \"source.geolocation.latitude\": 31.9522,\n    \"feed.accuracy\": 100,\n    \"extra.last_online\": \"2023-02-16\",\n    \"time.observation\": \"2023-02-16T09:55:12+00:00\",\n    \"source.geolocation.city\": \"amman\",\n    \"source.network\": \"82.212.115.0/24\",\n    \"time.source\": \"2023-02-15T14:19:09+00:00\",\n    \"source.as_name\": \"NEU-AS\",\n    \"source.geolocation.longitude\": 35.939,\n    \"feed.name\": \"abusech-feodo-c2-tracker\"\n  }\n
           
             
          •  
            Bots are divided into following groups:
             
               
            • Collectors - bots that collect data from sources such as website, mailbox, api, etc.
              •  
            • Parsers - bots that split and parse collected data into individual events.
              •  
            • Experts - bots that can do additional processing of events such as enriching, filtering, etc.
              •  
            • Outputs - bots that can output events to files, databases, etc.
              •  
             
            •  
          •  
            Data sources supported by IntelMQ are called feeds.
             
               
            • IntelMQ provides recommended configuration of collector and parser bot combinations for selected feeds.
              •  
             
            •  
          • The collection of all configured bots and their communication paths is called pipeline (or botnet).
            •  
          • Individual bots as well as the complete pipeline can be configured, managed and monitored via:
               
            • Web interface called IntelMQ Manager (best suited for regular users).
              •  
            • Command line tool called intelmqctl (best suited for administrators).
              •  
            • REST API provided by the IntelMQ API extension (best suited for other programs).
              •  
             
            •  
          "},{"location":"user/manager/","title":"Manager","text":""},{"location":"user/manager/#using-intelmq-manager","title":"Using IntelMQ Manager","text":"
          IntelMQ Manager is a graphical interface to manage configurations for IntelMQ. It's goal is to provide an intuitive tool to allow non-programmers to specify the data flow in IntelMQ.
          "},{"location":"user/manager/#configuration-pages","title":"Configuration Pages","text":""},{"location":"user/manager/#pipeline","title":"Pipeline","text":"
          This interface lets you visually configure the whole IntelMQ pipeline and the parameters of every single bot. You will be able to see the pipeline in a graph-like visualisation similar to the following screenshot (click to enlarge):
           
          "},{"location":"user/manager/#named-queues-paths","title":"Named queues / paths","text":"
          With IntelMQ Manager you can set the name of certain paths by double-clicking on the line which connects two bots:
           
           
          The name is then displayed along the edge:
           
          "},{"location":"user/manager/#bots-configuration","title":"Bots Configuration","text":"
          When you add a node or edit one you will be presented with a form with the available parameters for the bot. There you can easily change the parameters as shown in the screenshot:
           
           
          After editing the bot's configuration and pipeline, simply click Save Configuration to automatically write the changes to the correct files. The configurations are now ready to be deployed.
           
          Warning
           
          Without saving the configuration your changes will be lost whenever you reload the web page or move between different tabs within the IntelMQ manager page.
          "},{"location":"user/manager/#botnet-management","title":"Botnet Management","text":"
          When you save a configuration you can go to the Management section to see what bots are running and start/stop the entire botnet, or a single bot.
           
          "},{"location":"user/manager/#botnet-monitoring","title":"Botnet Monitoring","text":"
          You can also monitor the logs of individual bots or see the status of the queues for the entire system or for single bots.
           
          In this next example we can see the number of queued messages for all the queues in the system.
           
           
          The following example we can see the status information of a single bot. Namely, the number of queued messages in the queues that are related to that bot and also the last 20 log lines of that single bot.
           
          "},{"location":"user/manager/#keyboard-shortcuts","title":"Keyboard Shortcuts","text":"
          Any underscored letter denotes access key shortcut. The needed shortcut-keyboard is different per Browser:
           
             
          • Firefox: Ctrl+Alt + Letter
            •  
          • Chrome & Chromium: Alt + Letter
            •  
          "}]}
\ No newline at end of file