Skip to content

ThreatFox live IOC integration + CI - #4

Open
unclet67 wants to merge 2 commits into
mainfrom
threatfox-ioc-integration
Open

ThreatFox live IOC integration + CI#4
unclet67 wants to merge 2 commits into
mainfrom
threatfox-ioc-integration

Conversation

@unclet67

@unclet67 unclet67 commented Aug 6, 2026

Copy link
Copy Markdown
Owner

What this does

Adds abuse.ch ThreatFox as a third live intelligence source alongside MITRE ATT&CK and Malpedia, giving indicators real threat context and freshness — plus first-time CI and a couple of repo-hygiene fixes.

ThreatFox IOC integration (03ce78f)

  • Ingestion: ThreatFox client, normalization, and idempotent merge into the existing source-aware pipeline (ingestion.py, worker.py, config.py). Runs via python -m app.worker --source threatfox (or --source all); safely skips when ABUSECH_AUTH_KEY is unset.
  • Schema (0004_threatfox_iocs): adds confidence_score, threat_type, malware, tags, reference_url, expires_at, last_synced_at, and multi-source provenance to indicators.
  • API: /iocs and /dashboard/bootstrap expose the new fields and hide expired/inactive indicators by default (include_inactive=true to override).
  • Dashboard: the IOC screen becomes Live Indicators — malware/threat attribution, tags, ACTIVE/EXPIRED status, and last-sync freshness. Falls back cleanly to bundled demo data when the API is unreachable.
  • Docs: README / BACKEND / DEPLOYMENT updated with ThreatFox setup and the ABUSECH_AUTH_KEY requirement.

CI + hygiene (275488a)

  • New GitHub Actions workflow: installs the backend (pip install -e '.[dev]'), applies Alembic migrations on SQLite, runs pytest on Python 3.11 + 3.13, and builds the backend image.
  • .gitignore now covers *.bak / *.backup so local DB dumps can't be committed.

Testing

  • 15/15 backend tests pass locally, including the new test_threatfox.py (3) and test_ioc_multi_source.py (1).
  • Migration 0004 applies cleanly on a fresh SQLite DB.

Reviewer notes

  • Secrets: no credentials committed. ABUSECH_AUTH_KEY and other feed keys stay backend-only; production compose hard-fails without POSTGRES_PASSWORD / API_AUTH_TOKEN.
  • Attribution: the ThreatFox feature was developed in this working tree by a separate effort; this PR packages and verifies it (tests green) and adds the CI/gitignore hardening on top.
  • Not addressed here (follow-ups from the launch review): prod nginx publicly proxies /docs + /openapi.json, and nginx lacks security headers.

🤖 Generated with Claude Code

unclet67 and others added 2 commits August 6, 2026 06:55
Integrate the abuse.ch ThreatFox feed as a third intelligence source
alongside MITRE ATT&CK and Malpedia, with source-aware provenance and
indicator freshness handling.

- Ingestion: ThreatFox client, normalization, and idempotent merge into
  the source-aware pipeline (ingestion.py, worker.py, config.py)
- Schema: migration 0004 adds confidence_score, threat_type, malware,
  tags, reference_url, expires_at, and last_synced_at to indicators,
  plus multi-source provenance
- API: /iocs and /dashboard/bootstrap expose the new fields and hide
  expired/inactive indicators by default (include_inactive to override)
- Dashboard: IOC view becomes "Live Indicators" with malware/threat
  attribution, tags, ACTIVE/EXPIRED status, and last-sync freshness
- Tests: add test_threatfox.py and test_ioc_multi_source.py (15/15 pass)
- Docs: README/BACKEND/DEPLOYMENT updated with ThreatFox setup and the
  ABUSECH_AUTH_KEY requirement

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant