Skip to content

Commit 3851526

Browse files
aaronkaplanaaronkaplan
committed
initial
0 parents  commit 3851526

36 files changed

+2901
-0
lines changed

‎.gitignore

+33
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
*.bak
2+
*.swp
3+
*.pyc
4+
*.log
5+
*.*~
6+
*.mmdb
7+
*.profile
8+
.profile
9+
intelmq.egg-info
10+
build
11+
dist
12+
*.old
13+
.vagrant/
14+
*~
15+
.coverage
16+
.idea/
17+
htmlcov/
18+
*.pem
19+
*.key
20+
.eggs
21+
22+
# Debian build filed
23+
debian/files
24+
debian/intelmq.postinst.debhelper
25+
debian/intelmq.prerm.debhelper
26+
debian/intelmq.substvars
27+
debian/intelmq/
28+
/.pc/*
29+
30+
# custom stuff for this repo
31+
scratch/*
32+
*/~$*.pptx
33+
src/data/*.sql

‎LICENSE

+661
Large diffs are not rendered by default.

‎README.md

+107
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
# Stats portal
2+
3+
The stats portal is a component in the certtools series. It connectes to the [eventDB](https://github.com/certtools/intelmq/blob/develop/intelmq/bin/intelmq_psql_initdb.py) (the database of all incident events which got processed by IntelMQ). The following picture explains the place of the stats-portal within the certtools components:
4+
5+
![architecture-overview-stats-portal](architecture-overview-stats-portal-screen.png)
6+
7+
8+
The stats portal is the presentation layer for the aggregation tables.
9+
However, this code repository also contains the scripts to create the aggregation tables on periodic basis (for example via cron-jobs).
10+
11+
# Overview of this source code repository
12+
13+
This repo contains two parts:
14+
15+
1. the [code](src/) to aggregate the eventDB (creating aggregation tables)
16+
2. the [Grafana dashboard](src/grafana) which allows you to connect to the aggregation tables
17+
18+
19+
# How to install
20+
21+
## Aggregation tables
22+
23+
Take a look at the [cronjob example](src/crontab) and run the shell scripts on the eventDB server.
24+
The cron job creates aggregation tables, dumps them, pushes them to the server which will host the aggregation tables, loads them into postgresql there, transforms them to a timescaleDB format and cleans up some permissions. If you want to read the source code of this, please start by reading the (trivial) [make_all.sh](src/make_all.sh)
25+
26+
Assumptions:
27+
28+
We will call the host serving the aggregation tables "stats server".
29+
We will call the host doing the aggregation tables "eventDB server".
30+
31+
The stats server creates the visual graphs via Grafana which pulls its data from the (timescale) aggregation tables.
32+
Please note that it assumes a postgresql user "statsro" on the stats server (Grafana needs this)
33+
Furthermore, we assume an (unix) ssh user "stats-sync" on both servers. ssh keys are used to push the data from the eventDB server to the stats server.
34+
35+
36+
## Grafana dashboard
37+
38+
For the Grafana dashboard (on the stats server), please see the [README](src/grafana/README.md) in the grafana directory.
39+
40+
41+
# General topics
42+
43+
44+
## How to count correctly?
45+
46+
Alas, this question is anything but trivial.
47+
First, let's settle on a couple of definitions:
48+
49+
* **Measurement**: some observation of the Internet which results in multiple events. Measurements usually are [internet-wide scans](https://en.wikipedia.org/wiki/Port_scanner) or [sinkholes](https://en.wikipedia.org/wiki/DNS_sinkhole) of botnets or [honeypot](https://en.wikipedia.org/wiki/Honeypot_(computing)) events.
50+
* **Feed**: some data feed usually consisting of usually multiple rows ("events") which **must** always contain a time stamp (with time zone) and an IP address or a hostname / url.
51+
* **Event**: a row in the feed.
52+
* **EventDB**: the pre-processed (via IntelMQ or some other Extract Transform Load (ETL) tool) events are stored in a database, the eventDB.
53+
* **Aggregation**: the events in the EventDB are grouped by some criteria and counted.
54+
* **Constituency portal**: a database ("contactDB") of a CERTs constituency. Usually contains contact information (email addresses, PGP keys, etc) for the constituency's security team. But often also contains relevant network information ("net objects") such as ASN, netblocks etc. Given the netobjects, each network operator can gain insight into his/her statistics for a specific ASN or some netblocks.
55+
* **ASN**: Autonomous System Number. See [wikipedia:ASN](https://en.wikipedia.org/wiki/Autonomous_system_(Internet))
56+
57+
Now that we defined some terms, we can discuss how to count correctly, in other words: how to aggregate events properly. First of all, there is no "correct". We can only argue that certain ways of counting make more sense than others for specific questions. Secondly, the internet is a dynamic space. Effects such as [dynamic IP addresses](https://en.wikipedia.org/wiki/IP_address#Dynamic_IP) or [NAT](https://en.wikipedia.org/wiki/Network_address_translation) complicate the any measurement. Next to NAT and dynamic IPs, we also have the effect that answers to a measurement might be faked (especially with UDP based scans), packets might be dropped (for example with high scanning speeds) or the effect which is to be measured changes state during the measurement (a server gets turned off, a firewall rule triggers on the scan, etc.)
58+
So, in other words: measuring is hard. We always must keep in mind that the underlying data of the measurement might be biased or skewed somehow. However, most of the time, a national CERT will take the data feeds by feed providers (such as shadowserver) and simply treat it as indication (a possibility) of an event and pass it on to the network operator as a hint that something might be wrong. So, in a sense, the feed is treated as ground truth from the national CERT perspective.
59+
60+
Understanding and aggregating the measurement is the second hard problem which is directly connected with the way we pose a question to the measurement data set (the eventDB).
61+
62+
In our case, we are mostly interested in the following questions:
63+
64+
* what are the trends over time for:
65+
* malware infections
66+
* types of problems ([classification.taxonomy](https://www.enisa.europa.eu/publications/reference-incident-classification-taxonomy), classification.type, classification.identifier)
67+
* involved applications and protocols (for example UDP amplifiers)
68+
* specific ASNs and network operators (how well is a network operator cleaning up his part of the Internet)
69+
* feed providers (internal metric): are we receiving a a constant feed or are there big variations within a feed provider?
70+
* number of events per day (overall)
71+
* number of unique (distinct) IP addresses
72+
73+
74+
Use cases
75+
-----------
76+
77+
As can be seen above, one can vary the time interval (we will settle on one day), the way of counting (do we count number of events or number of distinct IP addresses?).
78+
Given the assumption that we take the feed provider's data as ground truth, we will try to answer the following questions:
79+
80+
### Internal view (national CSIRT's overview page)
81+
82+
* how many events per day per taxonomy (taxonomy, type, identifier) exist?
83+
* how many events are we getting per ASN per feed provider?
84+
* what are the top-N ASNs per taxonomy (taxonomy, type, identifier)?
85+
* where (ASN) are we seeing steady increases of problems (versus the regularl slow decline of issues)?
86+
* what are the top-N problems in our country?
87+
88+
### Network operator's perspective
89+
90+
* How many events per day are there in my ASN? (--> trendline)
91+
* How many events per day per taxonomy are there in my ASN? (--> trendline)
92+
* If settling on specific taxonomy, how many types are there in my ASN? (--> trendline)
93+
* What malware infections exist in my ASN? (trendline)
94+
* What are the trends for vulnerabilities in my ASN? (trendline)
95+
96+
97+
These questions are being answered by the stats portal.
98+
99+
100+
101+
Funded by
102+
=========
103+
104+
This project was partially funded by the CEF framework
105+
![cef logo](fundedby.png)
106+
107+

‎architecture-overview-stats-portal-screen.png

189 KB
Loading

‎architecture-overview-stats-portal.png

229 KB
Loading

‎docs/Screenshot 2019-03-18 at 13.06.42.png

377 KB
Loading

‎docs/Screenshot 2019-03-18 at 13.10.48.png

548 KB
Loading

‎docs/Screenshot 2019-03-18 at 13.11.20.png

515 KB
Loading

‎docs/Screenshot 2019-03-18 at 13.11.57.png

541 KB
Loading

‎docs/Screenshot 2019-03-18 at 13.12.19.png

511 KB
Loading

‎docs/arch-overview.png

135 KB
Loading

‎docs/arch-overview.pptx

38.9 KB
Binary file not shown.

‎docs/arch-overview_small.png

135 KB
Loading

‎fundedby.png

114 KB
Loading

‎src/README.md

+15
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
# Server backend side
2+
3+
We assume that the server backend hosts a postgreql eventDB. See [the corresponding IntelMQ output bot](https://github.com/certtools/intelmq/tree/develop/intelmq/bots/outputs/postgresql) as well as the scripts to [create the eventDB structure](https://github.com/certtools/intelmq/blob/develop/intelmq/bin/intelmq_psql_initdb.py).
4+
5+
From the eventDB, we can create aggregation tables and copy them over to a separate server.
6+
There, we convert the aggregation tables to [timescaleDB](https://www.timescale.com/) in order to speed up time window based searches.
7+
Have a look at the [make_all.sh](make_all.sh) script.
8+
9+
The constituency portal source code resides in [this repo](https://github.com/certat/do-portal)
10+
11+
The whole process can be seen in the architecture sketch below:
12+
13+
![architecture](../docs/arch-overview_small.png)
14+
15+

‎src/agg_tables.txt

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
agg_ndim_day_all_tags
2+
agg_ndim_day_netobject_tags

‎src/cleanup.sh

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
#!/bin/bash
2+
3+
olddate=$( date --iso-8601 -d '3 days ago')
4+
olddatadir="/home/stats-sync/data/${olddate}_dump_agg_tables"
5+
6+
# delete old stuff
7+
rm -rf $olddatadir

‎src/crontab

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# m h dom mon dow command
2+
03 04 * * * ( cd $HOME; date; ./make_all.sh ; date )

‎src/data/README.md

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# data directory
2+
3+
The data/ dir is used for transferring data between the eventDB server and the stats server.
4+
The files residing in this directory in the repository are meant as an example on how the asn, identifier, types and taxonomy tables look like.
5+
They get generated by the make_all.sh script however.
6+
7+

‎src/data/asns.sql

+59
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
--
2+
-- PostgreSQL database dump
3+
--
4+
5+
-- Dumped from database version 9.5.16
6+
-- Dumped by pg_dump version 9.5.16
7+
8+
SET statement_timeout = 0;
9+
SET lock_timeout = 0;
10+
SET client_encoding = 'UTF8';
11+
SET standard_conforming_strings = on;
12+
SELECT pg_catalog.set_config('search_path', '', false);
13+
SET check_function_bodies = false;
14+
SET client_min_messages = warning;
15+
SET row_security = off;
16+
17+
SET default_tablespace = '';
18+
19+
SET default_with_oids = false;
20+
21+
--
22+
-- Name: asns; Type: TABLE; Schema: public; Owner: stats-sync
23+
--
24+
25+
CREATE TABLE public.asns (
26+
"source.asn" integer
27+
);
28+
29+
30+
ALTER TABLE public.asns OWNER TO "stats-sync";
31+
32+
--
33+
-- Data for Name: asns; Type: TABLE DATA; Schema: public; Owner: stats-sync
34+
--
35+
36+
--- NOTE NOTE NOTE
37+
--- This is a list of all your ASNs in your country. We intentionally removed most so that you can simply
38+
--- see what you would need to put here.
39+
---
40+
--- Please note that you can get a list of all your ASNs in your country via stat.ripe.net:
41+
---
42+
--- https://stat.ripe.net/docs/data_api#country-asns
43+
---
44+
--- please fill in your own country list this way
45+
46+
COPY public.asns ("source.asn") FROM stdin;
47+
2025
48+
2033
49+
2036
50+
2047
51+
2055
52+
\N
53+
\.
54+
55+
56+
--
57+
-- PostgreSQL database dump complete
58+
--
59+

‎src/data/identifier.sql

+73
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
--
2+
-- PostgreSQL database dump
3+
--
4+
5+
-- Dumped from database version 9.5.16
6+
-- Dumped by pg_dump version 9.5.16
7+
8+
SET statement_timeout = 0;
9+
SET lock_timeout = 0;
10+
SET client_encoding = 'UTF8';
11+
SET standard_conforming_strings = on;
12+
SELECT pg_catalog.set_config('search_path', '', false);
13+
SET check_function_bodies = false;
14+
SET client_min_messages = warning;
15+
SET row_security = off;
16+
17+
SET default_tablespace = '';
18+
19+
SET default_with_oids = false;
20+
21+
--
22+
-- Name: identifier; Type: TABLE; Schema: public; Owner: stats-sync
23+
--
24+
25+
CREATE TABLE public.identifier (
26+
"classification.identifier" text
27+
);
28+
29+
30+
ALTER TABLE public.identifier OWNER TO "stats-sync";
31+
32+
--
33+
-- Data for Name: identifier; Type: TABLE DATA; Schema: public; Owner: stats-sync
34+
--
35+
36+
COPY public.identifier ("classification.identifier") FROM stdin;
37+
neshta
38+
openproxy
39+
open-chargen
40+
ghostpush
41+
gernidru
42+
tinynuke
43+
wordpress-vulnerabilities
44+
monerominer
45+
feodo
46+
accessible-sewage-plant
47+
wannacry
48+
sniperspy
49+
dresscode
50+
ntp-version
51+
smtpauth
52+
open-natpmp
53+
wpad
54+
androidlocker
55+
locky
56+
neutrino
57+
malware-generic
58+
apt-generic
59+
spam
60+
spamlink
61+
parama
62+
mail-password-leak
63+
unknownrat
64+
wordpress-login
65+
zeroaccess
66+
if-you-read-this-far-the-full-list-of-identifiers-are-available-from-cert-at-upon-request-by-friendly-other-CERTs
67+
\.
68+
69+
70+
--
71+
-- PostgreSQL database dump complete
72+
--
73+

‎src/data/taxonomy.sql

+54
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
--
2+
-- PostgreSQL database dump
3+
--
4+
5+
-- Dumped from database version 9.5.16
6+
-- Dumped by pg_dump version 9.5.16
7+
8+
SET statement_timeout = 0;
9+
SET lock_timeout = 0;
10+
SET client_encoding = 'UTF8';
11+
SET standard_conforming_strings = on;
12+
SELECT pg_catalog.set_config('search_path', '', false);
13+
SET check_function_bodies = false;
14+
SET client_min_messages = warning;
15+
SET row_security = off;
16+
17+
SET default_tablespace = '';
18+
19+
SET default_with_oids = false;
20+
21+
--
22+
-- Name: taxonomy; Type: TABLE; Schema: public; Owner: stats-sync
23+
--
24+
25+
CREATE TABLE public.taxonomy (
26+
"classification.taxonomy" text
27+
);
28+
29+
30+
ALTER TABLE public.taxonomy OWNER TO "stats-sync";
31+
32+
--
33+
-- Data for Name: taxonomy; Type: TABLE DATA; Schema: public; Owner: stats-sync
34+
--
35+
36+
COPY public.taxonomy ("classification.taxonomy") FROM stdin;
37+
abusive content
38+
availability
39+
fraud
40+
information content security
41+
information gathering
42+
intrusion attempts
43+
intrusions
44+
malicious code
45+
other
46+
test
47+
vulnerable
48+
\.
49+
50+
51+
--
52+
-- PostgreSQL database dump complete
53+
--
54+

0 commit comments

Comments
 (0)