ThreatFox live IOC integration + CI - #4
Open
unclet67 wants to merge 2 commits into
Open
Conversation
Integrate the abuse.ch ThreatFox feed as a third intelligence source alongside MITRE ATT&CK and Malpedia, with source-aware provenance and indicator freshness handling. - Ingestion: ThreatFox client, normalization, and idempotent merge into the source-aware pipeline (ingestion.py, worker.py, config.py) - Schema: migration 0004 adds confidence_score, threat_type, malware, tags, reference_url, expires_at, and last_synced_at to indicators, plus multi-source provenance - API: /iocs and /dashboard/bootstrap expose the new fields and hide expired/inactive indicators by default (include_inactive to override) - Dashboard: IOC view becomes "Live Indicators" with malware/threat attribution, tags, ACTIVE/EXPIRED status, and last-sync freshness - Tests: add test_threatfox.py and test_ioc_multi_source.py (15/15 pass) - Docs: README/BACKEND/DEPLOYMENT updated with ThreatFox setup and the ABUSECH_AUTH_KEY requirement Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Adds abuse.ch ThreatFox as a third live intelligence source alongside MITRE ATT&CK and Malpedia, giving indicators real threat context and freshness — plus first-time CI and a couple of repo-hygiene fixes.
ThreatFox IOC integration (
03ce78f)ingestion.py,worker.py,config.py). Runs viapython -m app.worker --source threatfox(or--source all); safely skips whenABUSECH_AUTH_KEYis unset.0004_threatfox_iocs): addsconfidence_score,threat_type,malware,tags,reference_url,expires_at,last_synced_at, and multi-source provenance to indicators./iocsand/dashboard/bootstrapexpose the new fields and hide expired/inactive indicators by default (include_inactive=trueto override).ABUSECH_AUTH_KEYrequirement.CI + hygiene (
275488a)pip install -e '.[dev]'), applies Alembic migrations on SQLite, runspyteston Python 3.11 + 3.13, and builds the backend image..gitignorenow covers*.bak/*.backupso local DB dumps can't be committed.Testing
test_threatfox.py(3) andtest_ioc_multi_source.py(1).0004applies cleanly on a fresh SQLite DB.Reviewer notes
ABUSECH_AUTH_KEYand other feed keys stay backend-only; production compose hard-fails withoutPOSTGRES_PASSWORD/API_AUTH_TOKEN./docs+/openapi.json, and nginx lacks security headers.🤖 Generated with Claude Code