Skip to content

Releases: idaholab/Malcolm

Malcolm v25.02.0

27 Feb 22:01
39154f6
Compare
Choose a tag to compare

Malcolm v25.02.0 contains some major performance improvements, a few smaller new features and enhancements, several component version updates, bug fixes, and documentation updates.

v25.01.0...v25.02.0

NOTE: As a result of some of the changes to environment variables made for decoupling Redis from NetBox (#580), environment variables from previous version may cause NetBox to fail to connect to Redis which prevents successful startup. To fix this, you should perform the following steps once prior to starting Malcolm:

  1. Stop Malcolm (./scripts/stop)
  2. Change the values for REDIS_CACHE_HOST and REDIS_HOST, removing the netbox- prefix from the values, so that they look like REDIS_HOST=redis and REDIS_CACHE_HOST=redis-cache, respectively.
    • These values were found in netbox.env in previous versions, but are found in redis.env in this release.
    • Alternatively, you may remove the lines for REDIS_HOST and REDIS_CACHE_HOST completely and they will be restored with correct values the next time the control script is run.
  3. Run ./scripts/status which will check the .env files and restore the missing values if you removed them.
  4. Start Malcolm (./scripts/start)
  • ✨ Features and enhancements
    • performance improvements for NetBox enrichment (#547)
      • NetBox enrichment and autopopulation is now approximately 4x faster than it was before (depending on resources)
    • performance improvements for Suricata's processing of uploaded PCAP files (#457)
      • Suricata's processing of large sets of uploaded PCAP files is now approximately 18x faster faster than it was before (depending on resources)
    • add validate_local_site_policy.sh script for validating Zeek local site policy (#598)
    • include corelight/zeek-long-connections plugin to log long connections (#585)
      • new zeek.conn.long field is available to indicate long connections
      • Connections dashboard updated to include this new field
      • see notes below on environment variable additions for configuring this plugin
    • standardize container health checks into scripts for all containers (#491)
      • added container health checks for containers that did not previously have them (live capture containers)
    • significant work-in-progress towards support for Sigma rules via OpenSearch Security Analytics (still incomplete due to some blocking issues upstream, see #475 for details)
      • changed normalization of Windows event log records (evtx) to more closely match Winlogbeat fields which are closer to what the Sigma rules for Windows events use, and updated corresponding Windows Event Logs dashboard
    • dnp3_control.log now includes clear_bit field to indicate if control code clear bit is set or unset
    • improved shared-object-creation.sh's cURL commands so that import failures for OpenSearch/Elasticsearch shared objects are printed to the debug logs rather than being redirected to /dev/null
  • ✅ Component version updates
  • 🐛 Bug fixes
    • ANSI color codes from croc displayed in ssl-client-transmit (#559)
    • clear screen after auth_setup when using Dialog mode (#574)
    • warn and prompt user before changing NetBox database passwords out from underneath existing database (#565)
    • UFW software firewall for Malcolm ISO should automatically open ports for syslog (#560)
      • removed default port allowances (e.g., 5044/tcp, 9200/tcp, etc.) so that they could be set dynamically as part of configuration
  • 📄 Configuration changes (in environment variables in ./config/) for Malcolm and in control_vars.conf for Hedgehog Linux
  • 🧹 Code and project maintenance
    • use arm-hosted runners for GitHub build actions for arm64 images (#557)
    • decouple redis from netbox (#580)
    • document standards for supply chain and code provenance checking (#555)
    • document incorporating new Suricata rules (and removing old ones) without restarting the Suricata containers (#589)
    • updates to documentation for Docker-ba...
Read more

Malcolm v25.01.0

17 Jan 20:26
Compare
Choose a tag to compare

Malcolm v25.01.0 contains quite a few UI/UX improvements; new parsers; a bevy of component version updates including to Arkime, Zeek, NetBox; and several bug fixes.

v24.12.0...v25.01.0

  • ✨ Features and enhancements
  • ✅ Component version updates
  • 🐛 Bug fixes
    • Extracted File Downloads interface not working with some filenames (cisagov#524)
    • user-defined custom field formats for index patterns are overwritten (cisagov#542)
    • port numbers should not be shown with commas in Dashboards (cisagov#540)
    • pivoting between Arkime and Dashboards doesn't work when Malcolm is behind a reverse proxy (e.g., traefik) (cisagov#552)
    • opensearch.keystore not created when running in Hedgehog run profile (cisagov#533)
    • ensure all conn.log entries are tagged ics for OT protocols (cisagov#541)
  • 📄 Configuration changes (in environment variables in ./config/) for Malcolm and in control_vars.conf for Hedgehog Linux
    • The following variables in ./config/filebeat.env configure Malcolm's ability to accept syslog messages:
      • FILEBEAT_SYSLOG_TCP_LISTEN and FILEBEAT_SYSLOG_UDP_LISTEN - if set to true, Malcolm will accept syslog messages over TCP and/or UDP, respectively
      • FILEBEAT_SYSLOG_TCP_PORT and FILEBEAT_SYSLOG_UDP_PORT - the port on which Malcolm will accept syslog messages over TCP and/or UDP, respectively
      • FILEBEAT_SYSLOG_TCP_FORMAT and FILEBEAT_SYSLOG_UDP_FORMAT - one of auto, rfc3164, or rfc5424, to specify the allowed format for syslog messages over TCP and/or UDP, respectively (default auto)
      • FILEBEAT_SYSLOG_TCP_MAX_MESSAGE_SIZE and FILEBEAT_SYSLOG_UDP_MAX_MESSAGE_SIZE - defines the maximum message size of the message received over TCP and/or UDP, respectively (default: 10KiB for UDP, 20MiB for TCP)
      • FILEBEAT_SYSLOG_TCP_MAX_CONNECTIONS - specifies the maximum current number of TCP connections for syslog messages
      • FILEBEAT_SYSLOG_TCP_SSL - if set to true, syslog messages over TCP will require the use of TLS. When ./scripts/auth_setup is run, self-signed certificates are generated which may be used by remote log forwarders. Located in Malcolm's ./filebeat/certs/ directory, the certificate authority and client certificate and key files should be copied to the host on which the forwarder is running and used when defining its settings for connecting to Malcolm.
    • The following variables in ./config/zeek.env for Malcolm and control_vars.conf for Hedgehog Linux pertain to the new Omron FINS protocol parser:
      • ZEEK_DISABLE_ICS_OMRON_FINS - if set to true, the Omron FINS parser will be disabled
      • ZEEK_OMRON_FINS_DETAILED - if set to true, a verbose Omron FINS details log (omron_fins_detail.log) will be created
  • 🧹 Code and project maintenance
    • Changed ⓒ year to 2025

Malcolm is a powerful, easily deployable network 🖧 traffic analysis tool suite for network security monitoring 🕵🏻‍♀️.

Malcolm operates as a cluster of containers 📦, isolated sandboxes which each serve a dedicated function of the system. This makes Malcolm deployable with frameworks like Docker 🐋, Podman 🦭, and Kubernetes ⎈. Check out the Quick Start guide for examples on how to get up and running.

Alternatively, dedicated official ISO installer images 💿 for Malcolm and Hedgehog Linux 🦔 can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split 🪓 into 2GB chunks and can be reassembled with scripts provided for both Bash 🐧 (release_cleaver.sh) and PowerShell 🪟 (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

As always, join us on the Malcolm discussions board 💬 to engage with the community, or pop some corn 🍿 and watch a video 📼.

Malcolm v24.12.0

19 Dec 15:28
d8dabe0
Compare
Choose a tag to compare

Malcolm v24.12.0 contains several improvements to the Malcolm configuration script, the Malcolm user interface, and the Malcolm API, as well as component version updates and bug fixes. This release also corresponds with the release of malcolm-test (cisagov#486), a Malcolm system testing framework.

v24.11.0...v24.12.0

  • Features and enhancements
  • Component version updates
  • Bug fixes
    • Zeek DNS records don't open correctly in Arkime sessions (cisagov#509)
      • Fixes to some Zeek dns.log parsing conflicts between ECS's DNS fields and what the Arkime schema is expecting
    • Mandiant threat intel source doesn't get split correctly when using JSON zeek log format (cisagov#494)
    • Set indices.query.bool.max_clause_count to 8192 to reflect maximum number of fields
    • Increase Java stack size (-Xss) for Logstash from 1536k to 2048k
    • Minor fixes for parsing Zeek intel.log (some fields not named correctly with Zeek JSON-formatted logs)
    • Fixed setting the Signature event severity tags
  • Code and project maintenance
    • Replaced hard-coded Malcolm version number in documentation markdown files with variable-based replacer populated during generation
    • Documentation and screenshot updates

Official ISO installer images for Malcolm and Hedgehog Linux can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh) and PowerShell (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

Malcolm v24.11.0

18 Nov 17:01
2383599
Compare
Choose a tag to compare

Malcolm v24.11.0 contains a new threat intelligence feed integration, a few new API calls, other minor improvements, bug fixes, and component version updates.

v24.10.1...v24.11.0

Official ISO installer images for Malcolm and Hedgehog Linux can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh) and PowerShell (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

Malcolm v24.10.1

24 Oct 16:04
4104256
Compare
Choose a tag to compare

Malcolm v24.10.1 contains some minor improvements, a few component version updates, a fix for a regression bug, and a fair amount of code cleanup.

v24.10.0...v24.10.1

  • Features and enhancements
  • Component version updates
  • Bug fixes
    • Fixed OpenSearch anomaly detection default detectors not being created (regression, #596)
  • Configuration changes (in environment variables in ./config/) for Malcolm and in control_vars.conf for Hedgehog Linux
    • Malcolm
      • ZEEK_JA4SSH_PACKET_COUNT (with a default of 200) has been added to ./config/zeek.env, which can be used to set logging interval number of packets for ja4ssh.log (#508)
    • Hedgehog Linux
      • ZEEK_JA4SSH_PACKET_COUNT has been added to control_vars.conf for the same purpose as described above
  • Code and project maintenance
    • Examine distro hardening, fix and update documentation as needed for Malcolm and Hedgehog Linux ISO-installed environments (#328)
    • Refactoring and code cleanup in the Logstash Zeek pipeline (#592)
    • Logstash container initialization code now automatically ensures that the Zeek TSV log parsing filters (dissect and split filters) in these files are looking for TAB characters (i.e., automatically replace spaces with tabs in these filter files in case the author forgot to do so) (#592)
    • Did some code cleanup in the ./shared/bin directory, mostly moving things that were specific to either the Malcolm or Hedgehog Installer ISO environments out of shared and into their respective locations for the ISO installer build.
    • When doing the aquasecurity/trivy-action action, use TRIVY_DB_REPOSITORY: ghcr.io/aquasecurity/trivy-db,public.ecr.aws/aquasecurity/trivy-db to try to fall back to an alternative official location for the vulnerability database if the first one fails. Also, pin this action to the v0.28.0 release rather than setting it to master.
    • As it's used pretty ubiquitously in shared scripts by many of the Malcolm containers, the jq utility is now installed across the board during the container image build.
    • Added a script to gather GitHub API metrics for Malcolm downloads (#594)

Official ISO installer images for Malcolm and Hedgehog Linux can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh) and PowerShell (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

Malcolm v24.10.0

09 Oct 14:57
26d0d66
Compare
Choose a tag to compare

Malcolm v24.10.0 contains fixes for a few regression bugs, minor improvements, and a few component updates.

v24.09.0...v24.10.0

  • Features and enhancements
    • Enable Zeek's parsing of HTTP server and client header names as zeek.http.client_header_names and zeek.http.server_header_names
    • Bumped maximum field limit in OpenSearch templates from 5000 to 6000
    • Some documentation improvements
    • Build improvement: fall back to alternative Zeek .deb download URL (#585)
    • Build improvement: limit threads for spicy build processes during Zeek package installation (#571)
  • Component version updates
  • Bug fixes
    • Fix broken dashboards regression from v24.09.0 (#588)
    • Fix Zeek-extracted files not getting saved to correct location for live Zeek capture (#590)
    • Fix for building Hedgehog Linux for Raspberry Pi 4 on an M2 MacBook

Official ISO installer images for Malcolm and Hedgehog Linux can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh) and PowerShell (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

Malcolm v24.09.0

19 Sep 19:53
2f94ef9
Compare
Choose a tag to compare

Malcolm v24.09.0 contains new features and enhancements, component version updates, and bug fixes.

v24.08.0...v24.09.0

  • Features and enhancements
    • When building Docker images and the Hedgehog Linux ISO, allow specifying alternate download URL for MaxMind GeoIP database files (#565)
    • Allow total index size-based pruning for opensearch-remote and elasticsearch-remote database modes (#446)
    • Allow splitting out indexes by other field values (#450)
    • Allow users to use the Arkime Lua plugin without having to create new bind volume mounts manually (#533)
    • Automatically create empty document on startup to avoid "no data" message spamming by Dashboards (#527 and #567)
    • Improvements to documentation and install.py for Linux performance tweaks (#495)
    • Include netbox-topology-views plugin by default (#553)
    • Integrate HART-IP parser (#561)
    • Add option to go backwards in Malcolm's dialog-based install.py installation and configuration script (#487)
    • Added Podman support (#407)
    • Update EtherNet/IP and CIP to account for new packet correlation ID (#558)
    • Update Network Traffic Analysis with Malcolm slides
  • Component version updates
  • Bug fixes
    • Filtering on hunt ID in Arkime not working (#554)
    • Hedgehog with OOB/VPN connection sets ARKIME_NODE_HOST incorrectly (#560 and #559, thanks @divinehawk)
    • Offline suricata Docker container does not initialize suricata.yml config file (#564)
  • Configuration changes (in environment variables in ./config/) for Malcolm and in control_vars.conf for Hedgehog Linux
    • Malcolm
      • The MALCOLM_NETWORK_INDEX_SUFFIX and MALCOLM_OTHER_INDEX_SUFFIX variables in ./config/opensearch.env now also support expanding dot-delimited field names in {{ }} (e.g., {{event.provider}}%{%y%m%d}).
      • MALCOLM_CONTAINER_RUNTIME has been added to ./config/process.env to indicate docker, podman, or kubernetes. This value only currently used in the install, configuration, and control scripts, not inside the containers themselves.
      • ZEEK_DISABLE_ICS_HART_IP has been added to ./config/zeek.env and can be set to true to disable the new HART-IP protocol parser.
    • Hedgehog Linux
      • ZEEK_DISABLE_ICS_HART_IP has been added to control_vars.conf and can be set to true to disable the new HART-IP protocol parser.

Official ISO installer images for Malcolm and Hedgehog Linux can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh) and PowerShell (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

Malcolm v24.08.0

27 Aug 19:38
2bacdae
Compare
Choose a tag to compare

Malcolm v24.08.0 contains minor improvements, some component version updates, and bug fixes.

v24.07.0...v24.08.0

  • Features and enhancements
    • in ISO installer, prompt to format other drives for artifact storage rather than just doing it automatically (#529)
    • allow users to more easily add NetBox plugins (#530)
    • run netbox-initializers plugin on startup even if we're doing a netbox database backup preload (#531)
    • during auth_setup "all" operation, do required operations without prompting if the files don't already exist (#536)
    • some containers need resource request specified for Kubernetes (#539)
    • add "public" pseudo-segments for public IP addresses (#542)
    • reworked Windows Event dashboard
    • some documentation updates
    • added netbox tag to any logs that are passed into the netbox_enrich.rb script in the Logstash enrichment pipeline
  • Component version updates
  • Bug fixes
    • dashboards-helper container's use of curl fails internal container name resolution when host has invalid DNS settings, prevents Malcolm initialization (#499)
    • Netbox service templates not populating (#522)
    • kubernetes manifest for netbox refers to netbox-netmap-json configmap which no longer exists (#540)
    • don't try to expose the OpenSearch port 9200 in docker-compose.yml when the database mode is not opensearch-local
    • improved the liveness check for the offline Zeek container so that it returns "healthy" if the intel thread feeds are still pulling before the monitoring processes start up
    • missing cracklib-runtime package prevents ISO service account password from being updated by non-root user (#548)

Official ISO installer images for Malcolm and Hedgehog Linux can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh) and PowerShell (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

Malcolm v24.07.0

30 Jul 21:38
7b7f401
Compare
Choose a tag to compare

Malcolm v24.07.0 contains minor improvements, some component version updates, and a few bug fixes.

v24.06.0...v24.07.0

  • Features and enhancements
    • integrated the ICSNPP GE SRTP network analyzer (#516)
    • Changed the way docker compose does bind mounts of files and directories to avoid creating empty directories when the source is missing, returning an error instead (#473)
      • This changed necessitated a switch from Python's built-in YAML library to ruamel.yaml
    • code to pull from MISP feeds should specify JSON as preferred format in HTTP headers (#520)
    • add optional service argument to restart script (#521)
    • replace API link on landing page with extracted-files (#524)
    • exclude private IP space Intel::ADDR items when populating Zeek intel (#528)
    • updated some screenshots for the documentation
  • Component version updates
  • Bug fixes
    • tarball-based installation should not depend on UID inside of tarball, prevents installation if UID with which tarball's contents were created don't match installing user's (#519)
    • bacnet discovery log not parsed correctly (#523)
    • resolved issue with the build.sh helper script when building non-AMD64 Docker images
  • Configuration changes (in environment variables in ./config/)
    • The variable ZEEK_DISABLE_ICS_GE_SRTP has been added to zeek.env and control_vars.conf to control enabling the network analyzer for the GE SRTP protocol. It's default value is true (indicating that the analyzer is disabled) as it is a somewhat uncommon OT protocol that likely won't be needed by most Malcolm users.
  • Other
    • Removed long-deprecated net-map.json file support (#517)

Official ISO installer images for Malcolm and Hedgehog Linux can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh) and PowerShell (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

Malcolm v24.06.0

27 Jun 01:18
75fe54b
Compare
Choose a tag to compare

Malcolm v24.06.0 contains new features, improvements, component version updates, and a few bug fixes.

v24.05.0...v24.06.0

NetBox: backwards compatibility-breaking change: This release of Malcolm updates NetBox from v3.6.7 to v4.0.6, for bug fixes, security updates, and requirements for Malcolm to support enrichment with multiple NetBox sites. However, NetBox's built-in migrations do not appear to work handle going from v3.6.7 to v4.0.6. It is likely that if you are using NetBox that you will encounter errors upon updating to this release of Malcolm. Prior to upgrading it is recommended that you navigate to Sites, IPAM > Prefixes, DCIM > Devices, and anywhere else you've populated NetBox data and click Export > All Data (CSV) and save those in case you need to recreate your NetBox inventory after upgrading. Malcolm's NetBox backup and restore will not work in this case. If you find NetBox has data errors after upgrading Malcolm, stop Malcolm and clear the NetBox inventory from your Malcolm installation directory (e.g., rm -rf ./netbox/postgres/* ./netbox/redis/*), then start Malcolm and recreate your NetBox inventory.

  • Features and enhancements
    • Support for multiple NetBox sites (#449)
      • Malcolm now supports enrichment from a NetBox inventory for asset interaction analysis across multiple sites. The NetBox site can be specified for uploaded PCAP, for a Hedgehog Linux sensor, and for Malcolm live capture.
    • JA4+ replaces the JA3 TLS fingerprinting standard from 2017 (see also this blog post) (#419)
    • Support uploading Windows Event Log evtx files (#465) and update associated dashboard
    • Document using GitHub runners to build Malcolm images (for contributors' guide, #491)
    • Generate new forwarder SSL keys on-the-fly when transferring between Malcolm and Hedgehog Linux (#492)
    • Incorporate ATT&CK-based Control-system Indicator Detection for Zeek (ACID) (#489), a collection of Operational Techonology (OT) protocol indicators developed to alert on specific ATT&CK for ICS behaviors
    • Add platform architecture and machine boot time to Malcolm version API
    • Add links to the navigation pane of most dashboards to "other" dashboards for non-network log data (e.g., resource monitoring, Windows Event logs, etc.)
  • Component version updates
  • Bug fixes
    • Arkime viewer not rolling PCAPs (#484)
    • Free up space in GitHub runner environment building ISO images to avoid errors due to exhausted disk space
  • Configuration changes in environment variables
    • There are no significant changes or additions to the ./config/*.env environment variable files in Malcolm v24.06.0

Official ISO installer images for Malcolm and Hedgehog Linux can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh) and PowerShell (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.