Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improved Bambenek parser to better handle description #1451

Draft
wants to merge 7 commits into
base: develop
Choose a base branch
from
Draft

Improved Bambenek parser to better handle description #1451

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
debd147
Improved parser to better handle description
Sep 13, 2019
e5b4372
Removed whitespace and broke down regex expression
Sep 13, 2019
56021be
TST: add bambenek tests for commas in descriptions
Sep 18, 2019
f8dfa9b
Corrected line-break in regex expression
Sep 19, 2019
87a11ed
Merge pull request #1 from wagner-certat/pr-1451
amojamo Sep 19, 2019
4065494
Merge branch 'develop' of https://github.com/amojamo/intelmq into dev…
Sep 19, 2019
ef4ec5c
Merge branch 'develop' into pr-1451
May 20, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 13 additions & 11 deletions intelmq/bots/parsers/bambenek/parser.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# -*- coding: utf-8 -*-
""" IntelMQ parser for Bambenek DGA, Domain, and IP feeds """

import re
from intelmq.lib.bot import ParserBot


Expand Down Expand Up @@ -35,32 +36,33 @@ def parse_line(self, line, report):
self.tempdata.append(line)

else:
value = line.split(',')
m = re.match(r"(?P<ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}),(?P<description>.*), "
r"(?P<timestamp>\d{4}-\d{2}-\d{2}[ ]\d{2}[:]\d{2}),(?P<url>.*)", line)
values = m.groups()
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That line raises an exception in the tests

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It could be my line split. I'm not sure how to break the regex line into two lines in order not to trigger the "Line too long" warning when it comes to code style checking.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just split the line like this:

            m = re.match(r"(?P<ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}),(?P<description>.*), "
                         r"(?P<timestamp>\d{4}-\d{2}-\d{2}[ ]\d{2}[:]\d{2}),(?P<url>.*)", line)

event = self.new_event(report)

event.add('event_description.text', value[1])
event.add('event_description.url', value[3])
event.add('event_description.text', values[1])
event.add('event_description.url', values[3])
event.add('raw', line)

# last column is a url with malware named txt file link
malware_name = value[-1].split('/')[-1].split('.')[0]
malware_name = values[-1].split('/')[-1].split('.')[0]
event.add('malware.name', self.MALWARE_NAME_MAP.get(malware_name, malware_name))

if report['feed.url'] in BambenekParserBot.IPMASTERLIST:
event.add('source.ip', value[0])
event.add('time.source', value[2] + ' UTC')
event.add('source.ip', values[0])
event.add('time.source', values[2] + ' UTC')
event.add('classification.type', 'c2server')
event.add('status', 'online')

elif report['feed.url'] in BambenekParserBot.DOMMASTERLIST:
event.add('source.fqdn', value[0])
event.add('time.source', value[2] + ' UTC')
event.add('source.fqdn', values[0])
event.add('time.source', values[2] + ' UTC')
event.add('classification.type', 'c2server')
event.add('status', 'online')

elif report['feed.url'] in BambenekParserBot.DGA_FEED:
event.add('source.fqdn', value[0])
event.add('time.source', value[2] + ' 00:00 UTC')
event.add('source.fqdn', values[0])
event.add('time.source', values[2] + ' 00:00 UTC')
event.add('classification.type', 'dga domain')

else:
Expand Down
15 changes: 8 additions & 7 deletions intelmq/tests/bots/parsers/bambenek/c2-ipmasterlist.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,17 +1,18 @@
#############################################################
## Master Feed of known, active and non-sinkholed C&Cs IP
## Master Feed of known, active and non-sinkholed C&Cs IP
## addresses
##
## Feed generated at: 2016-11-12 18:17
##
## Feed generated at: 2016-11-12 18:17
##
## Feed Provided By: John Bambenek of Bambenek Consulting
## [email protected] // http://bambenekconsulting.com
## Use of this feed is governed by the license here:
## http://osint.bambenekconsulting.com/license.txt
## Use of this feed is governed by the license here:
## http://osint.bambenekconsulting.com/license.txt
##
## For more information on this feed go to:
## For more information on this feed go to:
## http://osint.bambenekconsulting.com/manual/c2-ipmasterlist.txt
##
##
## All times are in UTC
#############################################################
213.247.47.190,IP used by shiotob/urlzone/bebloh C&C,2016-11-12 18:02,http://osint.bambenekconsulting.com/manual/bebloh.txt
64.183.187.20,IP resolved by necurs C&C, uses encoded IP, this is not the C2 IP, 2019-09-17 06:06,http://osint.bambenekconsulting.com/manual/necurs.txt
21 changes: 18 additions & 3 deletions intelmq/tests/bots/parsers/bambenek/test_parser.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@
'time.observation': '2016-01-01T00:00:00+00:00'
}

IP_EVENTS = {'feed.url': 'http://osint.bambenekconsulting.com/feeds/c2-ipmasterlist.txt',
IP_EVENTS = [{'feed.url': 'http://osint.bambenekconsulting.com/feeds/c2-ipmasterlist.txt',
'feed.name': 'Bambenek C2 IP Feed',
'__type': 'Event',
'time.observation': '2016-01-01T00:00:00+00:00',
Expand All @@ -57,7 +57,21 @@
'time.source': '2016-11-12T18:02:00+00:00',
'event_description.text': 'IP used by shiotob/urlzone/bebloh C&C',
'event_description.url': 'http://osint.bambenekconsulting.com/manual/bebloh.txt'
}
},
{'feed.url': 'http://osint.bambenekconsulting.com/feeds/c2-ipmasterlist.txt',
'feed.name': 'Bambenek C2 IP Feed',
'__type': 'Event',
'time.observation': '2016-01-01T00:00:00+00:00',
'raw': 'NjQuMTgzLjE4Ny4yMCxJUCByZXNvbHZlZCBieSBuZWN1cnMgQyZDLCB1c2VzIGVuY29kZWQgSVAsIHRoaXMgaXMgbm90IHRoZSBDMiBJUCwgMjAxOS0wOS0xNyAwNjowNixodHRwOi8vb3NpbnQuYmFtYmVuZWtjb25zdWx0aW5nLmNvbS9tYW51YWwvbmVjdXJzLnR4dAo=',
'source.ip': '64.183.187.20',
'malware.name': 'necurs',
'classification.type': 'c2server',
'status': 'online',
'time.source': '2019-09-17T06:06:00+00:00',
'event_description.text': 'IP resolved by necurs C&C, uses encoded IP, this is not the C2 IP',
'event_description.url': 'http://osint.bambenekconsulting.com/manual/necurs.txt'
},
]

DGA_REPORT = {'feed.url': 'http://osint.bambenekconsulting.com/feeds/dga-feed.txt',
'feed.name': 'Bambenek DGA Domain Feed',
Expand Down Expand Up @@ -95,7 +109,8 @@ def test_domain(self):
def test_ip(self):
self.input_message = IP_REPORT
self.run_bot()
self.assertMessageEqual(0, IP_EVENTS)
self.assertMessageEqual(0, IP_EVENTS[0])
self.assertMessageEqual(1, IP_EVENTS[1])

def test_dga(self):
self.input_message = DGA_REPORT
Expand Down