Skip to content

Commit

Permalink
Malcolm v3.3.1 development (#174)
Browse files Browse the repository at this point in the history 
Minor Malcolm release with the following updates:

* Bump capa to [v3.0.2](https://github.com/mandiant/capa/releases/tag/v3.0.2) which now includes ELF scanning capabilities
* Bump zeek to [v4.0.4](https://github.com/zeek/zeek/releases/tag/v4.0.4)
* Incorporate Corelight's ["OMIGOD" (CVE-2021-38647)](https://github.com/corelight/CVE-2021-38647) plugin
* minor fix on race condition creating default anomaly detectors
* minor tweak to `build.sh` script for building docker images
  • Loading branch information
@mmguero
mmguero authored Sep 30, 2021
1 parent bf3da93 commit e57292a
Show file tree
Hide file tree
Showing 12 changed files with 107 additions and 88 deletions.
2 changes: 1 addition & 1 deletion Dockerfiles/file-monitor.Dockerfile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ ENV YARA_URL "https://github.com/VirusTotal/yara/archive/v${YARA_VERSION}.tar.gz
ENV YARA_RULES_URL "https://github.com/Neo23x0/signature-base"
ENV YARA_RULES_DIR "/yara-rules"
ENV YARA_RULES_SRC_DIR "$SRC_BASE_DIR/signature-base"
ENV CAPA_VERSION "2.0.0"
ENV CAPA_VERSION "3.0.2"
ENV CAPA_URL "https://github.com/fireeye/capa/releases/download/v${CAPA_VERSION}/capa-v${CAPA_VERSION}-linux.zip"
ENV CAPA_DIR "/opt/capa"
ENV CAPA_BIN "${CAPA_DIR}/capa"
Expand Down
8 changes: 4 additions & 4 deletions Dockerfiles/zeek.Dockerfile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ ENV PUSER_PRIV_DROP true

# for download and install
ARG ZEEK_LTS=1
ARG ZEEK_VERSION=4.0.3-0
ARG ZEEK_VERSION=4.0.4-0
ARG SPICY_VERSION=1.2.1

ENV ZEEK_LTS $ZEEK_LTS
Expand Down Expand Up @@ -152,9 +152,9 @@ ADD zeek/config/*.txt ${ZEEK_DIR}/share/zeek/site/
# these ENVs should match the number of third party scripts/plugins installed by zeek_install_plugins.sh
# todo: Bro::LDAP is broken right now, disabled
ENV ZEEK_THIRD_PARTY_PLUGINS_COUNT 19
ENV ZEEK_THIRD_PARTY_PLUGINS_GREP "(_Zeek::Spicy|ANALYZER_SPICY_DHCP|ANALYZER_SPICY_DNS|ANALYZER_SPICY_HTTP|ANALYZER_SPICY_OPENVPN|ANALYZER_SPICY_IPSEC|ANALYZER_SPICY_TFTP|ANALYZER_SPICY_WIREGUARD|ANALYZER_SPICY_LDAP_TCP|Corelight::CommunityID|Corelight::PE_XOR|ICSNPP::BACnet|ICSNPP::BSAP_IP|ICSNPP::BSAP_SERIAL|ICSNPP::ENIP|Salesforce::GQUIC|Zeek::PROFINET|Zeek::S7comm|Zeek::TDS)"
ENV ZEEK_THIRD_PARTY_SCRIPTS_COUNT 15
ENV ZEEK_THIRD_PARTY_SCRIPTS_GREP "(bzar/main|callstranger-detector/callstranger|cve-2020-0601/cve-2020-0601|cve-2020-13777/cve-2020-13777|CVE-2020-16898/CVE-2020-16898|CVE-2021-31166/detect|hassh/hassh|ja3/ja3|pingback/detect|ripple20/ripple20|SIGRed/CVE-2020-1350|zeek-EternalSafety/main|zeek-httpattacks/main|zeek-sniffpass/__load__|zerologon/main)\.(zeek|bro)"
ENV ZEEK_THIRD_PARTY_PLUGINS_GREP "(_Zeek::Spicy|ANALYZER_SPICY_DHCP|ANALYZER_SPICY_DNS|ANALYZER_SPICY_HTTP|ANALYZER_SPICY_OPENVPN_UDP\b|ANALYZER_SPICY_IPSEC_UDP\b|ANALYZER_SPICY_TFTP|ANALYZER_SPICY_WIREGUARD|ANALYZER_SPICY_LDAP_TCP|Corelight::CommunityID|Corelight::PE_XOR|ICSNPP::BACnet|ICSNPP::BSAP|ICSNPP::ENIP|ICSNPP::ETHERCAT|Salesforce::GQUIC|Zeek::PROFINET|Zeek::S7comm|Zeek::TDS)"
ENV ZEEK_THIRD_PARTY_SCRIPTS_COUNT 16
ENV ZEEK_THIRD_PARTY_SCRIPTS_GREP "(bzar/main|callstranger-detector/callstranger|cve-2020-0601/cve-2020-0601|cve-2020-13777/cve-2020-13777|CVE-2020-16898/CVE-2020-16898|CVE-2021-38647/omigod|CVE-2021-31166/detect|hassh/hassh|ja3/ja3|pingback/detect|ripple20/ripple20|SIGRed/CVE-2020-1350|zeek-EternalSafety/main|zeek-httpattacks/main|zeek-sniffpass/__load__|zerologon/main)\.(zeek|bro)"

RUN mkdir -p /tmp/logs && \
cd /tmp/logs && \
Expand Down
71 changes: 36 additions & 35 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -141,21 +141,21 @@ You can then observe that the images have been retrieved by running `docker imag
```
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
malcolmnetsec/arkime 3.3.0 xxxxxxxxxxxx 39 hours ago 683MB
malcolmnetsec/elasticsearch-od 3.3.0 xxxxxxxxxxxx 40 hours ago 690MB
malcolmnetsec/file-monitor 3.3.0 xxxxxxxxxxxx 39 hours ago 470MB
malcolmnetsec/file-upload 3.3.0 xxxxxxxxxxxx 39 hours ago 199MB
malcolmnetsec/filebeat-oss 3.3.0 xxxxxxxxxxxx 39 hours ago 555MB
malcolmnetsec/freq 3.3.0 xxxxxxxxxxxx 39 hours ago 390MB
malcolmnetsec/htadmin 3.3.0 xxxxxxxxxxxx 39 hours ago 180MB
malcolmnetsec/kibana-helper 3.3.0 xxxxxxxxxxxx 40 hours ago 141MB
malcolmnetsec/kibana-od 3.3.0 xxxxxxxxxxxx 40 hours ago 1.16GB
malcolmnetsec/logstash-oss 3.3.0 xxxxxxxxxxxx 39 hours ago 1.41GB
malcolmnetsec/name-map-ui 3.3.0 xxxxxxxxxxxx 39 hours ago 137MB
malcolmnetsec/nginx-proxy 3.3.0 xxxxxxxxxxxx 39 hours ago 120MB
malcolmnetsec/pcap-capture 3.3.0 xxxxxxxxxxxx 39 hours ago 111MB
malcolmnetsec/pcap-monitor 3.3.0 xxxxxxxxxxxx 39 hours ago 157MB
malcolmnetsec/zeek 3.3.0 xxxxxxxxxxxx 39 hours ago 887MB
malcolmnetsec/arkime 3.3.1 xxxxxxxxxxxx 39 hours ago 683MB
malcolmnetsec/elasticsearch-od 3.3.1 xxxxxxxxxxxx 40 hours ago 690MB
malcolmnetsec/file-monitor 3.3.1 xxxxxxxxxxxx 39 hours ago 470MB
malcolmnetsec/file-upload 3.3.1 xxxxxxxxxxxx 39 hours ago 199MB
malcolmnetsec/filebeat-oss 3.3.1 xxxxxxxxxxxx 39 hours ago 555MB
malcolmnetsec/freq 3.3.1 xxxxxxxxxxxx 39 hours ago 390MB
malcolmnetsec/htadmin 3.3.1 xxxxxxxxxxxx 39 hours ago 180MB
malcolmnetsec/kibana-helper 3.3.1 xxxxxxxxxxxx 40 hours ago 141MB
malcolmnetsec/kibana-od 3.3.1 xxxxxxxxxxxx 40 hours ago 1.16GB
malcolmnetsec/logstash-oss 3.3.1 xxxxxxxxxxxx 39 hours ago 1.41GB
malcolmnetsec/name-map-ui 3.3.1 xxxxxxxxxxxx 39 hours ago 137MB
malcolmnetsec/nginx-proxy 3.3.1 xxxxxxxxxxxx 39 hours ago 120MB
malcolmnetsec/pcap-capture 3.3.1 xxxxxxxxxxxx 39 hours ago 111MB
malcolmnetsec/pcap-monitor 3.3.1 xxxxxxxxxxxx 39 hours ago 157MB
malcolmnetsec/zeek 3.3.1 xxxxxxxxxxxx 39 hours ago 887MB
```

#### Import from pre-packaged tarballs
Expand Down Expand Up @@ -218,10 +218,11 @@ Malcolm leverages the following excellent open source tools, among others.
* Andrew Klaus's [zeek-httpattacks](https://github.com/precurse/zeek-httpattacks) plugin for detecting noncompliant HTTP requests
* ICS protocol analyzers for Zeek published by [DHS CISA](https://github.com/cisagov/ICSNPP) and [Idaho National Lab](https://github.com/idaholab/ICSNPP)
* Corelight's [bro-xor-exe](https://github.com/corelight/bro-xor-exe-plugin) plugin
* Corelight's ["bad neighbor" (CVE-2020-16898)](https://github.com/corelight/CVE-2020-16898) plugin
* Corelight's [HTTP protocol stack vulnerability (CVE-2021-31166)](https://github.com/corelight/CVE-2021-31166) plugin
* Corelight's ["bad neighbor" (CVE-2020-16898)](https://github.com/corelight/CVE-2020-16898) plugin
* Corelight's [HTTP protocol stack vulnerability (CVE-2021-31166)](https://github.com/corelight/CVE-2021-31166) plugin
* Corelight's [callstranger-detector](https://github.com/corelight/callstranger-detector) plugin
* Corelight's [community ID](https://github.com/corelight/zeek-community-id) flow hashing plugin
* Corelight's ["OMIGOD" (CVE-2021-38647)](https://github.com/corelight/CVE-2021-38647) plugin
* Corelight's [pingback](https://github.com/corelight/pingback) plugin
* Corelight's [ripple20](https://github.com/corelight/ripple20) plugin
* Corelight's [SIGred](https://github.com/corelight/SIGred) plugin
Expand Down Expand Up @@ -876,7 +877,7 @@ A remote network sensor appliance can be used to monitor network traffic, captur
* monitor network interfaces
* capture packets to PCAP files
* detect file transfers in network traffic and extract and scan those files for threats
* generate and forward Zeek logs, Arkime sessions, and other information to [Malcolm](https://github.com/cisagov/malcolm)
* generate and forward Zeek logs, Arkime sessions, and other information to [Malcolm](https://github.com/cisagov/Malcolm)

Please see the [Hedgehog Linux README](https://github.com/cisagov/Malcolm/blob/main/sensor-iso/README.md) for more information.

Expand Down Expand Up @@ -1510,7 +1511,7 @@ Building the ISO may take 30 minutes or more depending on your system. As the bu

```
…
Finished, created "/malcolm-build/malcolm-iso/malcolm-3.3.0.iso"
Finished, created "/malcolm-build/malcolm-iso/malcolm-3.3.1.iso"
…
```

Expand Down Expand Up @@ -1893,21 +1894,21 @@ Pulling zeek ... done
user@host:~/Malcolm$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
malcolmnetsec/arkime 3.3.0 xxxxxxxxxxxx 39 hours ago 683MB
malcolmnetsec/elasticsearch-od 3.3.0 xxxxxxxxxxxx 40 hours ago 690MB
malcolmnetsec/file-monitor 3.3.0 xxxxxxxxxxxx 39 hours ago 470MB
malcolmnetsec/file-upload 3.3.0 xxxxxxxxxxxx 39 hours ago 199MB
malcolmnetsec/filebeat-oss 3.3.0 xxxxxxxxxxxx 39 hours ago 555MB
malcolmnetsec/freq 3.3.0 xxxxxxxxxxxx 39 hours ago 390MB
malcolmnetsec/htadmin 3.3.0 xxxxxxxxxxxx 39 hours ago 180MB
malcolmnetsec/kibana-helper 3.3.0 xxxxxxxxxxxx 40 hours ago 141MB
malcolmnetsec/kibana-od 3.3.0 xxxxxxxxxxxx 40 hours ago 1.16GB
malcolmnetsec/logstash-oss 3.3.0 xxxxxxxxxxxx 39 hours ago 1.41GB
malcolmnetsec/name-map-ui 3.3.0 xxxxxxxxxxxx 39 hours ago 137MB
malcolmnetsec/nginx-proxy 3.3.0 xxxxxxxxxxxx 39 hours ago 120MB
malcolmnetsec/pcap-capture 3.3.0 xxxxxxxxxxxx 39 hours ago 111MB
malcolmnetsec/pcap-monitor 3.3.0 xxxxxxxxxxxx 39 hours ago 157MB
malcolmnetsec/zeek 3.3.0 xxxxxxxxxxxx 39 hours ago 887MB
malcolmnetsec/arkime 3.3.1 xxxxxxxxxxxx 39 hours ago 683MB
malcolmnetsec/elasticsearch-od 3.3.1 xxxxxxxxxxxx 40 hours ago 690MB
malcolmnetsec/file-monitor 3.3.1 xxxxxxxxxxxx 39 hours ago 470MB
malcolmnetsec/file-upload 3.3.1 xxxxxxxxxxxx 39 hours ago 199MB
malcolmnetsec/filebeat-oss 3.3.1 xxxxxxxxxxxx 39 hours ago 555MB
malcolmnetsec/freq 3.3.1 xxxxxxxxxxxx 39 hours ago 390MB
malcolmnetsec/htadmin 3.3.1 xxxxxxxxxxxx 39 hours ago 180MB
malcolmnetsec/kibana-helper 3.3.1 xxxxxxxxxxxx 40 hours ago 141MB
malcolmnetsec/kibana-od 3.3.1 xxxxxxxxxxxx 40 hours ago 1.16GB
malcolmnetsec/logstash-oss 3.3.1 xxxxxxxxxxxx 39 hours ago 1.41GB
malcolmnetsec/name-map-ui 3.3.1 xxxxxxxxxxxx 39 hours ago 137MB
malcolmnetsec/nginx-proxy 3.3.1 xxxxxxxxxxxx 39 hours ago 120MB
malcolmnetsec/pcap-capture 3.3.1 xxxxxxxxxxxx 39 hours ago 111MB
malcolmnetsec/pcap-monitor 3.3.1 xxxxxxxxxxxx 39 hours ago 157MB
malcolmnetsec/zeek 3.3.1 xxxxxxxxxxxx 39 hours ago 887MB
```

Finally, we can start Malcolm. When Malcolm starts it will stream informational and debug messages to the console. If you wish, you can safely close the console or use `Ctrl+C` to stop these messages; Malcolm will continue running in the background.
Expand Down Expand Up @@ -1987,7 +1988,7 @@ If you checked out a working copy of the Malcolm repository from GitHub with a `

### Scenario 2: Malcolm was installed from a packaged tarball

If you installed Malcolm from [pre-packaged installation files](https://github.com/cisagov/malcolm#Packager), here are the basic steps to perform an upgrade:
If you installed Malcolm from [pre-packaged installation files](https://github.com/cisagov/Malcolm#Packager), here are the basic steps to perform an upgrade:

1. stop Malcolm
* `./scripts/stop`
Expand Down
30 changes: 15 additions & 15 deletions docker-compose-standalone.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ x-pcap-capture-variables: &pcap-capture-variables

services:
elasticsearch:
image: malcolmnetsec/elasticsearch-od:3.3.0
image: malcolmnetsec/elasticsearch-od:3.3.1
restart: "no"
stdin_open: false
tty: true
Expand Down Expand Up @@ -165,7 +165,7 @@ services:
retries: 3
start_period: 180s
kibana-helper:
image: malcolmnetsec/kibana-helper:3.3.0
image: malcolmnetsec/kibana-helper:3.3.1
restart: "no"
stdin_open: false
tty: true
Expand Down Expand Up @@ -193,7 +193,7 @@ services:
retries: 3
start_period: 30s
kibana:
image: malcolmnetsec/kibana-od:3.3.0
image: malcolmnetsec/kibana-od:3.3.1
restart: "no"
stdin_open: false
tty: true
Expand All @@ -214,7 +214,7 @@ services:
retries: 3
start_period: 210s
logstash:
image: malcolmnetsec/logstash-oss:3.3.0
image: malcolmnetsec/logstash-oss:3.3.1
restart: "no"
stdin_open: false
tty: true
Expand Down Expand Up @@ -249,7 +249,7 @@ services:
retries: 3
start_period: 600s
filebeat:
image: malcolmnetsec/filebeat-oss:3.3.0
image: malcolmnetsec/filebeat-oss:3.3.1
restart: "no"
stdin_open: false
tty: true
Expand Down Expand Up @@ -286,7 +286,7 @@ services:
retries: 3
start_period: 60s
arkime:
image: malcolmnetsec/arkime:3.3.0
image: malcolmnetsec/arkime:3.3.1
restart: "no"
stdin_open: false
tty: true
Expand Down Expand Up @@ -325,7 +325,7 @@ services:
retries: 3
start_period: 210s
zeek:
image: malcolmnetsec/zeek:3.3.0
image: malcolmnetsec/zeek:3.3.1
restart: "no"
stdin_open: false
tty: true
Expand All @@ -351,7 +351,7 @@ services:
retries: 3
start_period: 60s
file-monitor:
image: malcolmnetsec/file-monitor:3.3.0
image: malcolmnetsec/file-monitor:3.3.1
restart: "no"
stdin_open: false
tty: true
Expand All @@ -374,7 +374,7 @@ services:
retries: 3
start_period: 60s
pcap-capture:
image: malcolmnetsec/pcap-capture:3.3.0
image: malcolmnetsec/pcap-capture:3.3.1
restart: "no"
stdin_open: false
tty: true
Expand All @@ -394,7 +394,7 @@ services:
volumes:
- ./pcap/upload:/pcap
pcap-monitor:
image: malcolmnetsec/pcap-monitor:3.3.0
image: malcolmnetsec/pcap-monitor:3.3.1
restart: "no"
stdin_open: false
tty: true
Expand All @@ -417,7 +417,7 @@ services:
retries: 3
start_period: 90s
upload:
image: malcolmnetsec/file-upload:3.3.0
image: malcolmnetsec/file-upload:3.3.1
restart: "no"
stdin_open: false
tty: true
Expand All @@ -443,7 +443,7 @@ services:
retries: 3
start_period: 60s
htadmin:
image: malcolmnetsec/htadmin:3.3.0
image: malcolmnetsec/htadmin:3.3.1
restart: "no"
stdin_open: false
tty: true
Expand All @@ -465,7 +465,7 @@ services:
retries: 3
start_period: 60s
freq:
image: malcolmnetsec/freq:3.3.0
image: malcolmnetsec/freq:3.3.1
restart: "no"
stdin_open: false
tty: true
Expand All @@ -483,7 +483,7 @@ services:
retries: 3
start_period: 60s
name-map-ui:
image: malcolmnetsec/name-map-ui:3.3.0
image: malcolmnetsec/name-map-ui:3.3.1
restart: "no"
stdin_open: false
tty: true
Expand All @@ -504,7 +504,7 @@ services:
retries: 3
start_period: 60s
nginx-proxy:
image: malcolmnetsec/nginx-proxy:3.3.0
image: malcolmnetsec/nginx-proxy:3.3.1
restart: "no"
stdin_open: false
tty: true
Expand Down
Loading

0 comments on commit e57292a

Please sign in to comment.