SIEM — Security Information & Event Management System

A full-stack threat detection system built in Python that ingests logs from multiple sources, applies real-time detection rules, and surfaces alerts on a live dashboard. Integrates with AbuseIPDB for automatic IP reputation lookups on triggered alerts.

Tech stack: Python · Flask · SQLite · REST API · Chart.js

---

Features

• Log ingestion — tails Linux auth, web server, and firewall logs in real time
• Regex parser — normalizes raw log lines into structured events
• Rule engine — 6 detection rules including brute-force SSH, port scan, and web scanner detection
• Threat intelligence — AbuseIPDB API integration enriches alerts with IP reputation scores and geolocation
• Live dashboard — Flask web app with Chart.js charts, alert management, and event feed that auto-refreshes every 5 seconds

Project Structure

siem/
├── collector.py    ← Main loop: tails logs, calls parser + rules
├── parser.py       ← Regex parser, normalizes raw logs into structured dicts
├── database.py     ← All SQLite reads and writes
├── rules.py        ← Detection rule engine
├── threat_intel.py ← AbuseIPDB threat intelligence integration
├── app.py          ← Flask web server + JSON API routes
├── requirements.txt
├── logs/           ← Log source files
├── db/             ← SQLite database (auto-created)
└── templates/
└── index.html      ← Dashboard UI

Setup

# 1. Create a virtual environment
python3 -m venv venv
source venv/bin/activate        # Mac/Linux
venv\Scripts\activate           # Windows

# 2. Install dependencies
pip install -r requirements.txt

# 3. Add your AbuseIPDB API key in threat_intel.py

# 4. Run the collector (Terminal 1)
python collector.py

# 5. Run the dashboard (Terminal 2)
python app.py

# 6. Open http://localhost:5000

How it works

1. collector.py tails each log file every 2 seconds
2. New lines go to parser.py which uses regex to extract IPs, timestamps, usernames, and event types
3. Parsed events are stored in SQLite via database.py
4. Each event runs through every rule in rules.py
5. Rules that trigger create an alert with AbuseIPDB enrichment via threat_intel.py
6. app.py serves the dashboard and exposes /api/ endpoints
7. Dashboard auto-refreshes every 5 seconds

Detection Rules

Rule	Trigger	Severity
Brute Force SSH	5+ failed logins from same IP in 5 min	High
Port Scan	15+ firewall denies from same IP in 1 min	High
Web Scanner	10+ HTTP 404s from same IP in 2 min	Medium
Invalid User Scan	3+ invalid usernames from same IP in 10 min	Medium
Web Server Error	Any HTTP 500 error	Low
Sudo Usage	Any privileged command execution	Low