Skip to content

Commit

Permalink
Malcolm v1.5.2 development (#61)
Browse files Browse the repository at this point in the history
* remove bit about not being tested on Windows

* vagrant.vbguest autorebuild is giving me fits

* set DEBIAN_FRONTEND=noninteractive in vagrant reprovision

* replaced sample rule for elastalert with one that uses email; also, create dummy smtp-auth.yaml file on startup and map it into volumes so elastalert can use email credentials if desired

* update version to 1.5.2, moloch 2.0.1

* fix issue with new cyberchef not serving up Regex.js correctly

* replace stretch-slim with buster-slim as Moloch base image

* fix API calls to kibana in nginx

* sync sensor shared script with malcolm shared script

* rewrite kibana links in a more elegant way

* improvements to Moloch -> Kibana filter shortcut

* improvements to Moloch -> Kibana filter shortcut

* improvements to Moloch -> Kibana filter shortcut
  • Loading branch information
mmguero authored Sep 25, 2019
1 parent 20fe82d commit d453713
Show file tree
Hide file tree
Showing 18 changed files with 230 additions and 143 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
/nginx/htpasswd
/htadmin/config.ini
/htadmin/metadata
/elastalert/config/smtp-auth.yaml

# development
.vagrant
Expand Down
20 changes: 7 additions & 13 deletions Dockerfiles/moloch.Dockerfile
Original file line number Diff line number Diff line change
@@ -1,15 +1,14 @@
FROM debian:stretch-slim AS build
FROM debian:buster-slim AS build

# Copyright (c) 2019 Battelle Energy Alliance, LLC. All rights reserved.
LABEL maintainer="[email protected]"

ENV DEBIAN_FRONTEND noninteractive

ENV MOLOCH_VERSION "2.0.0"
ENV MOLOCH_VERSION "2.0.1"
ENV MOLOCHDIR "/data/moloch"
ENV ZEEK_VERSION "2.6.4"
ENV ZEEK_DIR "/opt/bro"
ENV CYBERCHEF_VERSION "8.30.1"
ENV ZEEK_CORELIGHT_COMMUNITY_ID_PLUGIN_VER "1.2"

ADD moloch/scripts/bs4_remove_div.py /data/
Expand All @@ -18,11 +17,10 @@ ADD README.md $MOLOCHDIR/doc/
ADD doc.css $MOLOCHDIR/doc/
ADD docs/images $MOLOCHDIR/doc/images/
ADD https://github.com/aol/moloch/archive/v$MOLOCH_VERSION.tar.gz /data/moloch.tar.gz
ADD https://github.com/gchq/CyberChef/releases/download/v$CYBERCHEF_VERSION/cyberchef.htm $MOLOCHDIR/doc/cyberchef.htm
ADD https://www.zeek.org/downloads/bro-$ZEEK_VERSION.tar.gz /data/bro.tar.gz
ADD https://github.com/corelight/bro-community-id/archive/$ZEEK_CORELIGHT_COMMUNITY_ID_PLUGIN_VER.tar.gz /data/bro-community-id.tar.gz

RUN sed -i "s/stretch main/stretch main contrib non-free/g" /etc/apt/sources.list && \
RUN sed -i "s/buster main/buster main contrib non-free/g" /etc/apt/sources.list && \
apt-get -q update && \
apt-get install -q -y --no-install-recommends \
bison \
Expand All @@ -42,7 +40,7 @@ RUN sed -i "s/stretch main/stretch main contrib non-free/g" /etc/apt/sources.lis
libkrb5-dev \
libmaxminddb-dev \
libpcap0.8-dev \
libssl1.0-dev \
libssl-dev \
libtool \
libwww-perl \
libyaml-dev \
Expand Down Expand Up @@ -108,10 +106,6 @@ RUN sed -i "s/stretch main/stretch main contrib non-free/g" /etc/apt/sources.lis
cp -v $MOLOCHDIR/doc/images/moloch/header_logo.png ./viewer/public/header_logo.png && \
cp -v $MOLOCHDIR/doc/images/moloch/header_logo.png ./viewer/vueapp/src/assets/logo.png && \
find $MOLOCHDIR/doc/images/screenshots -name "*.png" -delete && \
cp -v $MOLOCHDIR/doc/cyberchef.htm ./viewer/public/cyberchef.htm && \
rm -f ./viewer/public/cyberchef.htm.gz && \
gzip ./viewer/public/cyberchef.htm && \
chmod 664 ./viewer/public/cyberchef.htm.gz $MOLOCHDIR/doc/cyberchef.htm && \
export PATH="$MOLOCHDIR/bin:${PATH}" && \
ln -sf $MOLOCHDIR/bin/npm /usr/local/bin/npm && \
ln -sf $MOLOCHDIR/bin/node /usr/local/bin/node && \
Expand All @@ -130,7 +124,7 @@ RUN sed -i "s/stretch main/stretch main contrib non-free/g" /etc/apt/sources.lis
/tmp/* \
/var/tmp/*

FROM debian:stretch-slim AS runtime
FROM debian:buster-slim AS runtime

# Copyright (c) 2019 Battelle Energy Alliance, LLC. All rights reserved.
LABEL maintainer="[email protected]"
Expand Down Expand Up @@ -183,7 +177,7 @@ ENV ZEEK_EXTRACTOR_PATH $ZEEK_EXTRACTOR_PATH
COPY --from=build $MOLOCHDIR $MOLOCHDIR
COPY --from=build $ZEEK_DIR $ZEEK_DIR

RUN sed -i "s/stretch main/stretch main contrib non-free/" /etc/apt/sources.list && \
RUN sed -i "s/buster main/buster main contrib non-free/" /etc/apt/sources.list && \
apt-get -q update && \
apt-get install -q -y --no-install-recommends \
cron \
Expand Down Expand Up @@ -218,7 +212,7 @@ RUN sed -i "s/stretch main/stretch main contrib non-free/" /etc/apt/sources.list
ln -sf $MOLOCHDIR/bin/npm /usr/local/bin/npm && \
ln -sf $MOLOCHDIR/bin/node /usr/local/bin/node && \
ln -sf $MOLOCHDIR/bin/npx /usr/local/bin/npx && \
apt-get -q -y --purge remove gcc gcc-6 cpp cpp-6 libssl1.0-dev && \
apt-get -q -y --purge remove gcc gcc-8 cpp cpp-8 libssl-dev && \
apt-get -q -y autoremove && \
apt-get clean && \
rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
Expand Down
49 changes: 25 additions & 24 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,17 +112,17 @@ You can then observe that the images have been retrieved by running `docker imag
```
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
malcolmnetsec/moloch 1.5.1 xxxxxxxxxxxx 27 minutes ago 517MB
malcolmnetsec/htadmin 1.5.1 xxxxxxxxxxxx 2 hours ago 180MB
malcolmnetsec/nginx-proxy 1.5.1 xxxxxxxxxxxx 4 hours ago 53MB
malcolmnetsec/file-upload 1.5.1 xxxxxxxxxxxx 24 hours ago 198MB
malcolmnetsec/pcap-capture 1.5.1 xxxxxxxxxxxx 24 hours ago 111MB
malcolmnetsec/file-monitor 1.5.1 xxxxxxxxxxxx 24 hours ago 355MB
malcolmnetsec/logstash-oss 1.5.1 xxxxxxxxxxxx 25 hours ago 1.24GB
malcolmnetsec/curator 1.5.1 xxxxxxxxxxxx 25 hours ago 303MB
malcolmnetsec/kibana-oss 1.5.1 xxxxxxxxxxxx 33 hours ago 944MB
malcolmnetsec/filebeat-oss 1.5.1 xxxxxxxxxxxx 11 days ago 459MB
malcolmnetsec/elastalert 1.5.1 xxxxxxxxxxxx 11 days ago 276MB
malcolmnetsec/moloch 1.5.2 xxxxxxxxxxxx 27 minutes ago 517MB
malcolmnetsec/htadmin 1.5.2 xxxxxxxxxxxx 2 hours ago 180MB
malcolmnetsec/nginx-proxy 1.5.2 xxxxxxxxxxxx 4 hours ago 53MB
malcolmnetsec/file-upload 1.5.2 xxxxxxxxxxxx 24 hours ago 198MB
malcolmnetsec/pcap-capture 1.5.2 xxxxxxxxxxxx 24 hours ago 111MB
malcolmnetsec/file-monitor 1.5.2 xxxxxxxxxxxx 24 hours ago 355MB
malcolmnetsec/logstash-oss 1.5.2 xxxxxxxxxxxx 25 hours ago 1.24GB
malcolmnetsec/curator 1.5.2 xxxxxxxxxxxx 25 hours ago 303MB
malcolmnetsec/kibana-oss 1.5.2 xxxxxxxxxxxx 33 hours ago 944MB
malcolmnetsec/filebeat-oss 1.5.2 xxxxxxxxxxxx 11 days ago 459MB
malcolmnetsec/elastalert 1.5.2 xxxxxxxxxxxx 11 days ago 276MB
docker.elastic.co/elasticsearch/elasticsearch-oss 6.8.3 xxxxxxxxxxxx 5 weeks ago 769MB
```

Expand Down Expand Up @@ -231,7 +231,7 @@ Then, go take a walk or something since it will be a while. When you're done, yo
* `malcolmnetsec/htadmin` (based on `debian:buster-slim`)
* `malcolmnetsec/kibana-oss` (based on `docker.elastic.co/kibana/kibana-oss`)
* `malcolmnetsec/logstash-oss` (based on `centos:7`)
* `malcolmnetsec/moloch` (based on `debian:stretch-slim`)
* `malcolmnetsec/moloch` (based on `debian:buster-slim`)
* `malcolmnetsec/nginx-proxy` (based on `jwilder/nginx-proxy:alpine`)
* `malcolmnetsec/pcap-capture` (based on `debian:buster-slim`)

Expand Down Expand Up @@ -312,7 +312,8 @@ Run `install.py malcolm_XXXXXXXX_XXXXXX_XXXXXXX.tar.gz` and follow the prompts.
## <a name="Preparing"></a>Preparing your system

### <a name="SystemRequirements"></a>Recommended system requirements
Malcolm needs a reasonably up-to-date version of [Docker](https://www.docker.com/) and [Docker Compose](https://docs.docker.com/compose/). In theory this should be possible on Linux, macOS, and recent Windows 10 releases, although so far it's only been tested on Linux and macOS hosts.

Malcolm runs on top of [Docker](https://www.docker.com/) which runs on recent releases of Linux, Apple macOS and Microsoft Windows 10.

To quote the [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/guide/current/hardware.html), "If there is one resource that you will run out of first, it will likely be memory." The same is true for Malcolm: you will want at least 16 gigabytes of RAM to run Malcolm comfortably. For processing large volumes of traffic, I'd recommend at a bare minimum a dedicated server with 16 cores and 16 gigabytes of RAM. Malcolm can run on less, but more is better. You're going to want as much hard drive space as possible, of course, as the amount of PCAP data you're able to analyze and store will be limited by your hard drive.

Expand Down Expand Up @@ -1359,17 +1360,17 @@ Pulling nginx-proxy ... done
user@host:~/Malcolm$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
malcolmnetsec/moloch 1.5.1 xxxxxxxxxxxx 27 minutes ago 517MB
malcolmnetsec/htadmin 1.5.1 xxxxxxxxxxxx 2 hours ago 180MB
malcolmnetsec/nginx-proxy 1.5.1 xxxxxxxxxxxx 4 hours ago 53MB
malcolmnetsec/file-upload 1.5.1 xxxxxxxxxxxx 24 hours ago 198MB
malcolmnetsec/pcap-capture 1.5.1 xxxxxxxxxxxx 24 hours ago 111MB
malcolmnetsec/file-monitor 1.5.1 xxxxxxxxxxxx 24 hours ago 355MB
malcolmnetsec/logstash-oss 1.5.1 xxxxxxxxxxxx 25 hours ago 1.24GB
malcolmnetsec/curator 1.5.1 xxxxxxxxxxxx 25 hours ago 303MB
malcolmnetsec/kibana-oss 1.5.1 xxxxxxxxxxxx 33 hours ago 944MB
malcolmnetsec/filebeat-oss 1.5.1 xxxxxxxxxxxx 11 days ago 459MB
malcolmnetsec/elastalert 1.5.1 xxxxxxxxxxxx 11 days ago 276MB
malcolmnetsec/moloch 1.5.2 xxxxxxxxxxxx 27 minutes ago 517MB
malcolmnetsec/htadmin 1.5.2 xxxxxxxxxxxx 2 hours ago 180MB
malcolmnetsec/nginx-proxy 1.5.2 xxxxxxxxxxxx 4 hours ago 53MB
malcolmnetsec/file-upload 1.5.2 xxxxxxxxxxxx 24 hours ago 198MB
malcolmnetsec/pcap-capture 1.5.2 xxxxxxxxxxxx 24 hours ago 111MB
malcolmnetsec/file-monitor 1.5.2 xxxxxxxxxxxx 24 hours ago 355MB
malcolmnetsec/logstash-oss 1.5.2 xxxxxxxxxxxx 25 hours ago 1.24GB
malcolmnetsec/curator 1.5.2 xxxxxxxxxxxx 25 hours ago 303MB
malcolmnetsec/kibana-oss 1.5.2 xxxxxxxxxxxx 33 hours ago 944MB
malcolmnetsec/filebeat-oss 1.5.2 xxxxxxxxxxxx 11 days ago 459MB
malcolmnetsec/elastalert 1.5.2 xxxxxxxxxxxx 11 days ago 276MB
docker.elastic.co/elasticsearch/elasticsearch-oss 6.8.3 xxxxxxxxxxxx 5 weeks ago 769MB
```

Expand Down
23 changes: 12 additions & 11 deletions docker-compose-standalone-zeek-live.yml
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ services:
- ./elasticsearch:/usr/share/elasticsearch/data:delegated
- ./elasticsearch-backup:/opt/elasticsearch/backup:delegated
kibana:
image: malcolmnetsec/kibana-oss:1.5.1
image: malcolmnetsec/kibana-oss:1.5.2
restart: "no"
hostname: kibana
environment:
Expand All @@ -126,7 +126,7 @@ services:
retries: 3
start_period: 200s
elastalert:
image: malcolmnetsec/elastalert:1.5.1
image: malcolmnetsec/elastalert:1.5.2
restart: "no"
hostname: elastalert
environment:
Expand All @@ -147,10 +147,11 @@ services:
start_period: 200s
volumes:
- ./elastalert/config/elastalert.yaml:/opt/elastalert/config.yaml
- ./elastalert/config/smtp-auth.yaml:/opt/elastalert/smtp-auth.yaml:ro
- ./elastalert/config/config.json:/opt/elastalert-server/config/config.json
- ./elastalert/rules/:/opt/elastalert/rules/
curator:
image: malcolmnetsec/curator:1.5.1
image: malcolmnetsec/curator:1.5.2
restart: "no"
hostname: curator
environment:
Expand All @@ -160,7 +161,7 @@ services:
depends_on:
- elasticsearch
logstash:
image: malcolmnetsec/logstash-oss:1.5.1
image: malcolmnetsec/logstash-oss:1.5.2
restart: "no"
hostname: logstash
environment:
Expand Down Expand Up @@ -188,7 +189,7 @@ services:
- ./cidr-map.txt:/usr/share/logstash/config/cidr-map.txt:ro
- ./host-map.txt:/usr/share/logstash/config/host-map.txt:ro
filebeat:
image: malcolmnetsec/filebeat-oss:1.5.1
image: malcolmnetsec/filebeat-oss:1.5.2
restart: "no"
hostname: filebeat
environment:
Expand All @@ -213,7 +214,7 @@ services:
- ./filebeat/certs/client.crt:/certs/client.crt:ro
- ./filebeat/certs/client.key:/certs/client.key:ro
moloch:
image: malcolmnetsec/moloch:1.5.1
image: malcolmnetsec/moloch:1.5.2
restart: "no"
hostname: moloch
env_file:
Expand Down Expand Up @@ -245,7 +246,7 @@ services:
- ./moloch-logs:/data/moloch/logs
- ./moloch-raw:/data/moloch/raw
file-monitor:
image: malcolmnetsec/file-monitor:1.5.1
image: malcolmnetsec/file-monitor:1.5.2
restart: "no"
hostname: filemon
environment:
Expand All @@ -256,7 +257,7 @@ services:
- ./zeek-logs/extract_files:/data/zeek/extract_files
- ./zeek-logs/current:/data/zeek/logs
pcap-capture:
image: malcolmnetsec/pcap-capture:1.5.1
image: malcolmnetsec/pcap-capture:1.5.2
restart: "no"
network_mode: host
ulimits:
Expand All @@ -273,7 +274,7 @@ services:
volumes:
- ./pcap/upload:/pcap
upload:
image: malcolmnetsec/file-upload:1.5.1
image: malcolmnetsec/file-upload:1.5.2
restart: "no"
hostname: upload
env_file:
Expand All @@ -290,7 +291,7 @@ services:
volumes:
- ./pcap/upload:/var/www/upload/server/php/chroot/files
htadmin:
image: malcolmnetsec/htadmin:1.5.1
image: malcolmnetsec/htadmin:1.5.2
restart: "no"
hostname: htadmin
environment:
Expand All @@ -302,7 +303,7 @@ services:
- ./htadmin/metadata:/var/www/htadmin/config/metadata:rw
- ./nginx/htpasswd:/var/www/htadmin/config/htpasswd:rw
nginx-proxy:
image: malcolmnetsec/nginx-proxy:1.5.1
image: malcolmnetsec/nginx-proxy:1.5.2
restart: "no"
hostname: nginx-proxy
depends_on:
Expand Down
23 changes: 12 additions & 11 deletions docker-compose-standalone.yml
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ services:
- ./elasticsearch:/usr/share/elasticsearch/data:delegated
- ./elasticsearch-backup:/opt/elasticsearch/backup:delegated
kibana:
image: malcolmnetsec/kibana-oss:1.5.1
image: malcolmnetsec/kibana-oss:1.5.2
restart: "no"
hostname: kibana
environment:
Expand All @@ -126,7 +126,7 @@ services:
retries: 3
start_period: 200s
elastalert:
image: malcolmnetsec/elastalert:1.5.1
image: malcolmnetsec/elastalert:1.5.2
restart: "no"
hostname: elastalert
environment:
Expand All @@ -147,10 +147,11 @@ services:
start_period: 200s
volumes:
- ./elastalert/config/elastalert.yaml:/opt/elastalert/config.yaml
- ./elastalert/config/smtp-auth.yaml:/opt/elastalert/smtp-auth.yaml:ro
- ./elastalert/config/config.json:/opt/elastalert-server/config/config.json
- ./elastalert/rules/:/opt/elastalert/rules/
curator:
image: malcolmnetsec/curator:1.5.1
image: malcolmnetsec/curator:1.5.2
restart: "no"
hostname: curator
environment:
Expand All @@ -160,7 +161,7 @@ services:
depends_on:
- elasticsearch
logstash:
image: malcolmnetsec/logstash-oss:1.5.1
image: malcolmnetsec/logstash-oss:1.5.2
restart: "no"
hostname: logstash
environment:
Expand Down Expand Up @@ -188,7 +189,7 @@ services:
- ./cidr-map.txt:/usr/share/logstash/config/cidr-map.txt:ro
- ./host-map.txt:/usr/share/logstash/config/host-map.txt:ro
filebeat:
image: malcolmnetsec/filebeat-oss:1.5.1
image: malcolmnetsec/filebeat-oss:1.5.2
restart: "no"
hostname: filebeat
environment:
Expand All @@ -213,7 +214,7 @@ services:
- ./filebeat/certs/client.crt:/certs/client.crt:ro
- ./filebeat/certs/client.key:/certs/client.key:ro
moloch:
image: malcolmnetsec/moloch:1.5.1
image: malcolmnetsec/moloch:1.5.2
restart: "no"
hostname: moloch
env_file:
Expand Down Expand Up @@ -245,7 +246,7 @@ services:
- ./moloch-logs:/data/moloch/logs
- ./moloch-raw:/data/moloch/raw
file-monitor:
image: malcolmnetsec/file-monitor:1.5.1
image: malcolmnetsec/file-monitor:1.5.2
restart: "no"
hostname: filemon
environment:
Expand All @@ -256,7 +257,7 @@ services:
- ./zeek-logs/extract_files:/data/zeek/extract_files
- ./zeek-logs/current:/data/zeek/logs
pcap-capture:
image: malcolmnetsec/pcap-capture:1.5.1
image: malcolmnetsec/pcap-capture:1.5.2
restart: "no"
network_mode: host
ulimits:
Expand All @@ -273,7 +274,7 @@ services:
volumes:
- ./pcap/upload:/pcap
upload:
image: malcolmnetsec/file-upload:1.5.1
image: malcolmnetsec/file-upload:1.5.2
restart: "no"
hostname: upload
env_file:
Expand All @@ -290,7 +291,7 @@ services:
volumes:
- ./pcap/upload:/var/www/upload/server/php/chroot/files
htadmin:
image: malcolmnetsec/htadmin:1.5.1
image: malcolmnetsec/htadmin:1.5.2
restart: "no"
hostname: htadmin
environment:
Expand All @@ -302,7 +303,7 @@ services:
- ./htadmin/metadata:/var/www/htadmin/config/metadata:rw
- ./nginx/htpasswd:/var/www/htadmin/config/htpasswd:rw
nginx-proxy:
image: malcolmnetsec/nginx-proxy:1.5.1
image: malcolmnetsec/nginx-proxy:1.5.2
restart: "no"
hostname: nginx-proxy
depends_on:
Expand Down
Loading

0 comments on commit d453713

Please sign in to comment.