Malcolm 2.4.0 release development (#154)

Malcolm 2.4.0 release development

    Extracted file scanning
        added Capa as an optional extracted file scanner
        multiple file scanners can now be enabled
    Version updates
        updated Moloch to 2.4.1
        updated Zeek to 3.0.10
        updated Linux Kernel for ISO installers to 5.7
    Zeek plugins
        added Corelight's Zerologon plugin to detect CVE-2020-1472
    Tweaks and bug fixes
        Don't allow docker to mess with firewall rules in Malcolm ISO
        Fix idaholab#26, ISO installers result in blank screen when booting with BIOS
        Fix idaholab#24, install.py won't prompt to change ownership of extracted directory correctly if run as root
        Leave some development packages in place in Hedgehog ISO so that Spicy plugins can be compiled

* update sha for malcolm iso

* Allow multiple file scanners (VirusTotal, ClamAV, Malass) to be run instead of having to just select one.

* working on idaholab#148, adding Yara as a file scanner

* working on idaholab#148, some fixes to signatures dashboard/parsing

* revert dashboard change from previous commit

* working on idaholab#148, change file pipeline from push/pull to publish/subscribe (but still screwy for multiple engines)

* working on idaholab#148, change file pipeline from push/pull to publish/subscribe (but still screwy for multiple engines)

* working on idaholab#148, more work on allowing multiple file scanners to be able to run nicely

* working on idaholab#148, adding Yara as a file scanner to the hedgehog iso

* working on idaholab#148, adding Yara as a file scanner to the hedgehog iso

* update pip3 python versions

* fix configure-capture for multiple scanners

* fix configure-capture for multiple scanners

* fix configure-capture for multiple scanners

* bump version for 2.3.0

* bump moloch to 2.4.0

* fix moloch 2.4.0 build

* adjust footer for malcolm

* fix footer

* fix footer

* readme updates

* fix idaholab#150 by decreasing the value of COMPOSE_HTTP_TIMEOUT slightly

* reduce image size for freq

* remove old kernels when done with installs

* remove old kernels when done with installs

* update kbuild version for iso

* added new environment variables for disabling certain zeek features to hedgehog

* fix issue with zeek_init in my local.zeek not having priority set so it executes prior to the parser plugins being loaded

* don't depend on scanner for file-monitor health check

* update shas

* provide more than just filename to scanners from carved file watcher (additionally send size and mime type)

* add capa (https://github.com/fireeye/capa) binary to file-monitor docker image

* working on idaholab#152, implement capa file scanner for EXE Files

* working on idaholab#152, implement capa file scanner for EXE Files

* working on idaholab#152, implement capa file scanner for EXE Files

* improve Capa detection by logging mitre att&ck techniques by default

* fix verbose capa results

* improve Capa detection by logging mitre att&ck techniques by default, but allow original rules as well

* update zeek to 3.0.10 (https://github.com/zeek/zeek/blob/v3.0.10/NEWS)

* update version to 2.4.0

* fix bug idaholab#24, install.py won't prompt to change ownership of extracted directory correctly if run as root

* fix kbuild install

* update capa version

* retain some build stuff

* keep spicy build available in ISO

* added detection for Zerologon (CVE-2020-1472) via corelight zeek script

https://github.com/corelight/zerologon

* this *should* fix the issues with the installer not working in BIOS. it should work in both bios and efi. fixes issue idaholab#26. testing in progress

* added menu entries for bios installer

* install capa from pip rather than binary

* bump moloch to 2.4.1; https://github.com/aol/moloch/blob/v2.4.1/CHANGELOG#L21-L42

* fix url for zeek-EternalSafety github repo

* fix lgtm python alerts for unused variable and exception handling

* fix issue where capa rules directory is not specified correctly

* fix issue where capa rules directory is not specified correctly

* fix lgtm python alerts for unused variable and exception handling

* fix issue running spicy where noexec is defined for /tmp; related somewhat to issue zeek/spicy#521

* fix issue running spicy where noexec is defined for /tmp

* fix ufw/iptables/docker networking issue

* fix ufw/iptables/docker networking issue

* fix ufw/iptables/docker networking issue (specify default docker address pool)

* fix ufw/iptables/docker networking issue (specify default docker address pool)

* change documentation to reflect resolution of issue idaholab#26

* change documentation to reflect resolution of issue idaholab#26

* fix BIOS installer to use preseed correctly

* fix BIOS installer to use preseed correctly

* in malcolm installer, make /var partition larger to give more room for docker images

* tweak spicy HLTO TMP location so it doesn't try to use capture path on hedgehog ISO

* make sure file doesn't get moved out from underneath capa scanner by making move logic more robust

* make sure that capa-rules matches the capa git rules submodule revision

Dockerfiles/curator.Dockerfile

@@ -2,7 +2,7 @@ FROM debian:buster-slim
 
 # Copyright (c) 2020 Battelle Energy Alliance, LLC.  All rights reserved.
 LABEL maintainer="[email protected]"
-
+LABEL org.opencontainers.image.authors='[email protected]'
 LABEL org.opencontainers.image.url='https://github.com/cisagov/Malcolm'
 LABEL org.opencontainers.image.documentation='https://github.com/cisagov/Malcolm/blob/master/README.md'
 LABEL org.opencontainers.image.source='https://github.com/cisagov/Malcolm'

Dockerfiles/elastalert.Dockerfile

@@ -2,7 +2,7 @@ FROM mmguero/elastalert:2.0.5
 
 # Copyright (c) 2020 Battelle Energy Alliance, LLC.  All rights reserved.
 LABEL maintainer="[email protected]"
-
+LABEL org.opencontainers.image.authors='[email protected]'
 LABEL org.opencontainers.image.url='https://github.com/cisagov/Malcolm'
 LABEL org.opencontainers.image.documentation='https://github.com/cisagov/Malcolm/blob/master/README.md'
 LABEL org.opencontainers.image.source='https://github.com/cisagov/Malcolm'

Dockerfiles/elasticsearch.Dockerfile

@@ -2,7 +2,7 @@ FROM docker.elastic.co/elasticsearch/elasticsearch-oss:7.6.2
 
 # Copyright (c) 2020 Battelle Energy Alliance, LLC.  All rights reserved.
 LABEL maintainer="[email protected]"
-
+LABEL org.opencontainers.image.authors='[email protected]'
 LABEL org.opencontainers.image.url='https://github.com/cisagov/Malcolm'
 LABEL org.opencontainers.image.documentation='https://github.com/cisagov/Malcolm/blob/master/README.md'
 LABEL org.opencontainers.image.source='https://github.com/cisagov/Malcolm'

Dockerfiles/file-monitor.Dockerfile

@@ -2,7 +2,7 @@ FROM debian:buster-slim
 
 # Copyright (c) 2020 Battelle Energy Alliance, LLC.  All rights reserved.
 LABEL maintainer="[email protected]"
-
+LABEL org.opencontainers.image.authors='[email protected]'
 LABEL org.opencontainers.image.url='https://github.com/cisagov/Malcolm'
 LABEL org.opencontainers.image.documentation='https://github.com/cisagov/Malcolm/blob/master/README.md'
 LABEL org.opencontainers.image.source='https://github.com/cisagov/Malcolm'
@@ -42,6 +42,8 @@ ARG EXTRACTED_FILE_PIPELINE_DEBUG_EXTRA=false
 ARG CLAMD_SOCKET_FILE=/tmp/clamd.ctl
 ARG EXTRACTED_FILE_ENABLE_YARA=false
 ARG EXTRACTED_FILE_YARA_CUSTOM_ONLY=false
+ARG EXTRACTED_FILE_ENABLE_CAPA=false
+ARG EXTRACTED_FILE_CAPA_VERBOSE=false
 
 ENV ZEEK_EXTRACTOR_PATH $ZEEK_EXTRACTOR_PATH
 ENV ZEEK_LOG_DIRECTORY $ZEEK_LOG_DIRECTORY
@@ -64,10 +66,14 @@ ENV EXTRACTED_FILE_PIPELINE_DEBUG_EXTRA $EXTRACTED_FILE_PIPELINE_DEBUG_EXTRA
 ENV CLAMD_SOCKET_FILE $CLAMD_SOCKET_FILE
 ENV EXTRACTED_FILE_ENABLE_YARA $EXTRACTED_FILE_ENABLE_YARA
 ENV EXTRACTED_FILE_YARA_CUSTOM_ONLY $EXTRACTED_FILE_YARA_CUSTOM_ONLY
+ENV EXTRACTED_FILE_ENABLE_CAPA $EXTRACTED_FILE_ENABLE_CAPA
+ENV EXTRACTED_FILE_CAPA_VERBOSE $EXTRACTED_FILE_CAPA_VERBOSE
 ENV YARA_VERSION "4.0.2"
 ENV YARA_URL "https://github.com/VirusTotal/yara/archive/v${YARA_VERSION}.tar.gz"
 ENV YARA_RULES_URL "https://codeload.github.com/Neo23x0/signature-base/tar.gz/master"
 ENV YARA_RULES_DIR "/yara-rules"
+ENV CAPA_URL "https://github.com/fireeye/capa"
+ENV CAPA_RULES_DIR "/opt/capa-rules"
 ENV SRC_BASE_DIR "/usr/local/src"
 
 RUN sed -i "s/buster main/buster main contrib non-free/g" /etc/apt/sources.list && \
@@ -80,6 +86,7 @@ RUN sed -i "s/buster main/buster main contrib non-free/g" /etc/apt/sources.list
       clamav-freshclam \
       curl \
       gcc \
+      git \
       libclamunrar9 \
       libjansson-dev \
       libjansson4 \
@@ -89,19 +96,25 @@ RUN sed -i "s/buster main/buster main contrib non-free/g" /etc/apt/sources.list
       libssl1.1 \
       libtool \
       make \
-      pkg-config && \
+      pkg-config \
+      unzip && \
     apt-get  -y -q install \
       inotify-tools \
       libzmq5 \
       psmisc \
+      python \
+      python-dev \
+      python-pip \
+      python-backports-shutil-get-terminal-size \
       python3 \
       python3-bs4 \
       python3-dev \
       python3-pip \
       python3-pyinotify \
       python3-requests \
       python3-zmq && \
-    pip3 install clamd supervisor yara-python && \
+    pip3 install clamd supervisor yara-python python-magic psutil && \
+    pip2 install flare-capa && \
     mkdir -p "${SRC_BASE_DIR}" && \
     cd "${SRC_BASE_DIR}" && \
       curl -sSL "${YARA_URL}" | tar xzf - -C "${SRC_BASE_DIR}" && \
@@ -114,35 +127,47 @@ RUN sed -i "s/buster main/buster main contrib non-free/g" /etc/apt/sources.list
         --enable-dotnet && \
       make && \
       make install && \
-    cd /tmp && \
     rm -rf "${SRC_BASE_DIR}"/yara* && \
-    mkdir -p ./Neo23x0 && \
+    cd /tmp && \
+      mkdir -p ./Neo23x0 && \
       curl -sSL "$YARA_RULES_URL" | tar xzvf - -C ./Neo23x0 --strip-components 1 && \
       mkdir -p "${YARA_RULES_DIR}" && \
       cp ./Neo23x0/yara/* ./Neo23x0/vendor/yara/* "${YARA_RULES_DIR}"/ && \
       cp ./Neo23x0/LICENSE "${YARA_RULES_DIR}"/_LICENSE && \
       rm -rf /tmp/Neo23x0 && \
+    cd /tmp && \
+      git clone --depth 1 --single-branch --branch "v$(/usr/local/bin/capa --version 2>&1 | awk '{print $2}')" "${CAPA_URL}" /tmp/capa && \
+      cd /tmp/capa && \
+      git submodule init rules && \
+      git submodule update --depth 1 rules && \
+      cd /tmp && \
+      rm -rf "${CAPA_RULES_DIR}" && \
+      mv /tmp/capa/rules "${CAPA_RULES_DIR}" && \
+      rm -rf "${CAPA_RULES_DIR}"/.git* /tmp/capa && \
     apt-get -y -q --allow-downgrades --allow-remove-essential --allow-change-held-packages --purge remove \
         automake \
         build-essential \
         gcc \
         gcc-8 \
+        git \
         libc6-dev \
         libgcc-8-dev \
         libjansson-dev \
         libmagic-dev \
         libssl-dev \
         libtool \
         make \
-        python3-dev && \
+        python-dev \
+        python3-dev \
+        unzip && \
       apt-get -y -q --allow-downgrades --allow-remove-essential --allow-change-held-packages autoremove && \
       apt-get clean && \
       rm -rf /var/lib/apt/lists/* && \
     curl -s -S -L -o /var/lib/clamav/main.cvd http://database.clamav.net/main.cvd && \
       curl -s -S -L -o /var/lib/clamav/daily.cvd http://database.clamav.net/daily.cvd && \
       curl -s -S -L -o /var/lib/clamav/bytecode.cvd http://database.clamav.net/bytecode.cvd && \
     groupadd --gid ${DEFAULT_GID} ${PGROUP} && \
-      useradd -M --uid ${DEFAULT_UID} --gid ${DEFAULT_GID} ${PUSER} && \
+      useradd -m --uid ${DEFAULT_UID} --gid ${DEFAULT_GID} ${PUSER} && \
       usermod -a -G tty ${PUSER} && \
     mkdir -p /var/log/clamav /var/lib/clamav && \
       chown -R ${PUSER}:${PGROUP} /var/log/clamav  /var/lib/clamav && \
@@ -161,6 +186,7 @@ RUN sed -i "s/buster main/buster main contrib non-free/g" /etc/apt/sources.list
       ln -r -s /usr/local/bin/zeek_carve_scanner.py /usr/local/bin/vtot_scan.py && \
       ln -r -s /usr/local/bin/zeek_carve_scanner.py /usr/local/bin/clam_scan.py && \
       ln -r -s /usr/local/bin/zeek_carve_scanner.py /usr/local/bin/yara_scan.py && \
+      ln -r -s /usr/local/bin/zeek_carve_scanner.py /usr/local/bin/capa_scan.py && \
       ln -r -s /usr/local/bin/zeek_carve_scanner.py /usr/local/bin/malass_scan.py
 
 ADD shared/bin/docker-uid-gid-setup.sh /usr/local/bin/

Dockerfiles/file-upload.Dockerfile

@@ -26,7 +26,7 @@ RUN apt-get update && \
 FROM debian:buster-slim AS runtime
 
 LABEL maintainer="[email protected]"
-
+LABEL org.opencontainers.image.authors='[email protected]'
 LABEL org.opencontainers.image.url='https://github.com/cisagov/Malcolm'
 LABEL org.opencontainers.image.documentation='https://github.com/cisagov/Malcolm/blob/master/README.md'
 LABEL org.opencontainers.image.source='https://github.com/cisagov/Malcolm'

Dockerfiles/filebeat.Dockerfile

@@ -2,7 +2,7 @@ FROM docker.elastic.co/beats/filebeat-oss:7.6.2
 
 # Copyright (c) 2020 Battelle Energy Alliance, LLC.  All rights reserved.
 LABEL maintainer="[email protected]"
-
+LABEL org.opencontainers.image.authors='[email protected]'
 LABEL org.opencontainers.image.url='https://github.com/cisagov/Malcolm'
 LABEL org.opencontainers.image.documentation='https://github.com/cisagov/Malcolm/blob/master/README.md'
 LABEL org.opencontainers.image.source='https://github.com/cisagov/Malcolm'

Dockerfiles/freq.Dockerfile

@@ -2,7 +2,7 @@ FROM debian:buster-slim
 
 # Copyright (c) 2020 Battelle Energy Alliance, LLC.  All rights reserved.
 LABEL maintainer="[email protected]"
-
+LABEL org.opencontainers.image.authors='[email protected]'
 LABEL org.opencontainers.image.url='https://github.com/cisagov/Malcolm'
 LABEL org.opencontainers.image.documentation='https://github.com/cisagov/Malcolm/blob/master/README.md'
 LABEL org.opencontainers.image.source='https://github.com/cisagov/Malcolm'

Dockerfiles/htadmin.Dockerfile

@@ -2,7 +2,7 @@ FROM debian:buster-slim
 
 # Copyright (c) 2020 Battelle Energy Alliance, LLC.  All rights reserved.
 LABEL maintainer="[email protected]"
-
+LABEL org.opencontainers.image.authors='[email protected]'
 LABEL org.opencontainers.image.url='https://github.com/cisagov/Malcolm'
 LABEL org.opencontainers.image.documentation='https://github.com/cisagov/Malcolm/blob/master/README.md'
 LABEL org.opencontainers.image.source='https://github.com/cisagov/Malcolm'

Dockerfiles/kibana.Dockerfile

@@ -2,7 +2,7 @@ FROM docker.elastic.co/kibana/kibana-oss:7.6.2
 
 # Copyright (c) 2020 Battelle Energy Alliance, LLC.  All rights reserved.
 LABEL maintainer="[email protected]"
-
+LABEL org.opencontainers.image.authors='[email protected]'
 LABEL org.opencontainers.image.url='https://github.com/cisagov/Malcolm'
 LABEL org.opencontainers.image.documentation='https://github.com/cisagov/Malcolm/blob/master/README.md'
 LABEL org.opencontainers.image.source='https://github.com/cisagov/Malcolm'

Dockerfiles/logstash.Dockerfile

@@ -25,7 +25,7 @@ RUN /bin/bash -lc "command curl -sSL https://rvm.io/mpapis.asc | gpg2 --import -
 FROM docker.elastic.co/logstash/logstash-oss:7.6.2
 
 LABEL maintainer="[email protected]"
-
+LABEL org.opencontainers.image.authors='[email protected]'
 LABEL org.opencontainers.image.url='https://github.com/cisagov/Malcolm'
 LABEL org.opencontainers.image.documentation='https://github.com/cisagov/Malcolm/blob/master/README.md'
 LABEL org.opencontainers.image.source='https://github.com/cisagov/Malcolm'

Dockerfiles/moloch.Dockerfile

@@ -4,7 +4,7 @@ FROM debian:buster-slim AS build
 
 ENV DEBIAN_FRONTEND noninteractive
 
-ENV MOLOCH_VERSION "2.4.0"
+ENV MOLOCH_VERSION "2.4.1"
 ENV MOLOCHDIR "/data/moloch"
 ENV MOLOCH_URL "https://codeload.github.com/aol/moloch/tar.gz/v${MOLOCH_VERSION}"
 ENV MOLOCH_LOCALELASTICSEARCH no
@@ -90,7 +90,7 @@ RUN sed -i "s/buster main/buster main contrib non-free/g" /etc/apt/sources.list
 FROM debian:buster-slim
 
 LABEL maintainer="[email protected]"
-
+LABEL org.opencontainers.image.authors='[email protected]'
 LABEL org.opencontainers.image.url='https://github.com/cisagov/Malcolm'
 LABEL org.opencontainers.image.documentation='https://github.com/cisagov/Malcolm/blob/master/README.md'
 LABEL org.opencontainers.image.source='https://github.com/cisagov/Malcolm'

Dockerfiles/name-map-ui.Dockerfile

@@ -2,7 +2,7 @@ FROM alpine:3.11
 
 # Copyright (c) 2020 Battelle Energy Alliance, LLC.  All rights reserved.
 LABEL maintainer="[email protected]"
-
+LABEL org.opencontainers.image.authors='[email protected]'
 LABEL org.opencontainers.image.url='https://github.com/cisagov/Malcolm'
 LABEL org.opencontainers.image.documentation='https://github.com/cisagov/Malcolm/blob/master/README.md'
 LABEL org.opencontainers.image.source='https://github.com/cisagov/Malcolm'

Dockerfiles/nginx.Dockerfile

@@ -49,7 +49,7 @@ RUN set -x ; \
 FROM alpine:3.11
 
 LABEL maintainer="[email protected]"
-
+LABEL org.opencontainers.image.authors='[email protected]'
 LABEL org.opencontainers.image.url='https://github.com/cisagov/Malcolm'
 LABEL org.opencontainers.image.documentation='https://github.com/cisagov/Malcolm/blob/master/README.md'
 LABEL org.opencontainers.image.source='https://github.com/cisagov/Malcolm'

Dockerfiles/pcap-capture.Dockerfile

@@ -2,7 +2,7 @@ FROM debian:buster-slim
 
 # Copyright (c) 2020 Battelle Energy Alliance, LLC.  All rights reserved.
 LABEL maintainer="[email protected]"
-
+LABEL org.opencontainers.image.authors='[email protected]'
 LABEL org.opencontainers.image.url='https://github.com/cisagov/Malcolm'
 LABEL org.opencontainers.image.documentation='https://github.com/cisagov/Malcolm/blob/master/README.md'
 LABEL org.opencontainers.image.source='https://github.com/cisagov/Malcolm'

Dockerfiles/pcap-monitor.Dockerfile

@@ -2,7 +2,7 @@ FROM debian:buster-slim
 
 # Copyright (c) 2020 Battelle Energy Alliance, LLC.  All rights reserved.
 LABEL maintainer="[email protected]"
-
+LABEL org.opencontainers.image.authors='[email protected]'
 LABEL org.opencontainers.image.url='https://github.com/cisagov/Malcolm'
 LABEL org.opencontainers.image.documentation='https://github.com/cisagov/Malcolm/blob/master/README.md'
 LABEL org.opencontainers.image.source='https://github.com/cisagov/Malcolm'

Dockerfiles/zeek.Dockerfile

@@ -16,7 +16,7 @@ ENV SRC_BASE_DIR "/usr/local/src"
 ENV ZEEK_DIR "/opt/zeek"
 ENV ZEEK_PATCH_DIR "${SRC_BASE_DIR}/zeek-patches"
 ENV ZEEK_SRC_DIR "${SRC_BASE_DIR}/zeek-${ZEEK_VERSION}"
-ENV ZEEK_VERSION "3.0.8"
+ENV ZEEK_VERSION "3.0.10"
 
 # using clang now instead of gcc because Spicy depends on it
 ENV LLVM_VERSION "10"
@@ -90,7 +90,7 @@ RUN sed -i "s/buster main/buster main contrib non-free/g" /etc/apt/sources.list
 FROM debian:buster-slim
 
 LABEL maintainer="[email protected]"
-
+LABEL org.opencontainers.image.authors='[email protected]'
 LABEL org.opencontainers.image.url='https://github.com/cisagov/Malcolm'
 LABEL org.opencontainers.image.documentation='https://github.com/cisagov/Malcolm/blob/master/README.md'
 LABEL org.opencontainers.image.source='https://github.com/cisagov/Malcolm'