ThreatFox live IOC integration + CI

.env.production.example

@@ -1,4 +1,5 @@
 POSTGRES_PASSWORD=replace-with-a-long-random-password
 API_AUTH_TOKEN=replace-with-a-long-random-bearer-token
-STIX_FEED_URL=https://example.invalid/approved-stix-bundle.json
+ABUSECH_AUTH_KEY=replace-with-your-abuse-ch-auth-key
+THREATFOX_DAYS=7
 PUBLIC_PORT=8080

.github/workflows/ci.yml

@@ -0,0 +1,46 @@
+name: CI
+
+on:
+  push:
+    branches: ["**"]
+  pull_request:
+
+jobs:
+  backend-tests:
+    name: Backend tests (Python ${{ matrix.python-version }})
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.11", "3.13"]
+    defaults:
+      run:
+        working-directory: backend
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: pip
+          cache-dependency-path: backend/pyproject.toml
+
+      - name: Install package with dev dependencies
+        run: python -m pip install -e '.[dev]'
+
+      - name: Verify Alembic migrations apply cleanly (SQLite)
+        env:
+          DATABASE_URL: sqlite:///./ci.db
+        run: alembic upgrade head
+
+      - name: Run test suite
+        run: pytest
+
+  backend-image:
+    name: Build backend image
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Build API/worker image
+        run: docker build ./backend

.gitignore

@@ -15,3 +15,5 @@ htmlcov/
 *.db
 *.sqlite
 *.sqlite3
+*.bak
+*.backup

BACKEND.md

@@ -30,18 +30,12 @@ docker compose up --build
 
 The API is available at `http://127.0.0.1:8001`.
 
-## MITRE and Malpedia sync
+## Feed synchronization
 
-Run both source adapters once:
+Run the actor adapters independently or together:
 
 ```bash
 cd backend
-python -m app.worker --once --source all
-```
-
-Run a single adapter:
-
-```bash
 python -m app.worker --once --source mitre
 python -m app.worker --once --source malpedia
 ```
@@ -59,18 +53,40 @@ Malpedia actor endpoints do not require registration. If a token is provided,
 the backend sends `Authorization: apitoken <token>` as documented by Malpedia.
 Never put this token in `index.html` or another browser-delivered file.
 
-The identity merge order is:
+The actor identity merge order is source external ID, MITRE group ID, then
+case-insensitive name or alias. Each actor can retain multiple source profiles.
+
+### ThreatFox IOC feed
+
+Obtain a free Auth-Key from <https://auth.abuse.ch/> and configure:
+
+```dotenv
+THREATFOX_API_URL=https://threatfox-api.abuse.ch/api/v1/
+ABUSECH_AUTH_KEY=your-auth-key
+THREATFOX_DAYS=7
+```
+
+Then run:
+
+```bash
+alembic upgrade head
+python -m app.worker --once --source threatfox
+```
+
+The adapter requests recent IOCs, maps IP/port, domain, URL, and hash types,
+and upserts by type and value. It preserves per-source IDs, confidence score,
+malware name, threat type, tags, reference, first/last-seen times, last-sync
+time, and a six-month expiration date. API responses exclude inactive or
+expired indicators unless `include_inactive=true` is requested.
+
+Run every configured feed with:
 
-1. Existing source external ID
-2. MITRE ATT&CK group ID
-3. Case-insensitive canonical name or alias
-4. New actor profile
+```bash
+python -m app.worker --once --source all
+```
 
-Each actor may have multiple `actor_sources` records. These retain the source
-name, external ID, public profile URL, and last successful sync time. MITRE
-contributes descriptions and ATT&CK technique relationships; Malpedia enriches
-aliases, country/sponsor metadata when existing fields are unknown, and its
-actor profile link. Empty source fields never erase populated actor fields.
+If `ABUSECH_AUTH_KEY` is absent, an `all` run logs a warning and continues with
+MITRE and Malpedia. An explicit `--source threatfox` run fails clearly instead.
 
 To run scheduled syncs in Docker:
 

DEPLOYMENT.md

@@ -1,8 +1,8 @@
 # Production deployment
 
 The production Compose stack runs four services: Nginx for the dashboard,
-FastAPI for the API, PostgreSQL for persistence, and a scheduled STIX ingestion
-worker.
+FastAPI for the API, PostgreSQL for persistence, and a scheduled intelligence
+ingestion worker for MITRE ATT&CK, Malpedia, and ThreatFox.
 
 ## Prepare secrets
 
@@ -11,7 +11,9 @@ cp .env.production.example .env.production
 ```
 
 Replace every placeholder. Use long random values for `POSTGRES_PASSWORD` and
-`API_AUTH_TOKEN`, and configure only an approved STIX bundle URL.
+`API_AUTH_TOKEN`. Obtain `ABUSECH_AUTH_KEY` from
+<https://auth.abuse.ch/>. Keep all credentials in the backend environment or a
+managed secret store; never place them in `index.html`.
 
 ## Start the stack
 
@@ -26,8 +28,11 @@ authenticated API request and stores it only for the current browser session.
 ## Operational requirements
 
 - Terminate TLS in front of the published port before internet exposure.
+- Confirm the abuse.ch community API fair-use terms fit the deployment; obtain
+  a commercial subscription when required.
 - Use a managed secret store instead of an environment file where available.
 - Back up the PostgreSQL volume and test restoration.
 - Pin and scan container images in the deployment environment.
-- Restrict the STIX source allowlist and outbound network access.
+- Restrict feed destinations and outbound network access.
 - Monitor `/api/health`, container restarts, and ingestion run failures.
+- Validate confidence, age, and internal telemetry before blocking an IOC.

README.md

@@ -16,6 +16,7 @@ operational reporting.
 - Browse threat-actor profiles, aliases, confidence, and attribution
 - Merge MITRE ATT&CK and Malpedia metadata into one actor profile
 - Preserve per-source external IDs, profile links, and last-sync timestamps
+- Ingest recent ThreatFox IOCs with confidence, malware, tags, and expiration
 - Explore campaigns by actor, sector, and status
 - Filter IP, domain, hash, and URL indicators
 - Review ATT&CK technique coverage and report metadata
@@ -27,7 +28,7 @@ operational reporting.
 ## Architecture
 
 ```text
-MITRE ATT&CK STIX       Malpedia actor API
+MITRE ATT&CK STIX   Malpedia actor API   ThreatFox IOC API
          \                  /
           v                v
        Source-aware ingestion and identity merge
@@ -65,32 +66,50 @@ python3 -m http.server 8000
 Open [http://127.0.0.1:8000](http://127.0.0.1:8000). API documentation is
 available at [http://127.0.0.1:8001/docs](http://127.0.0.1:8001/docs).
 
-## Sync threat actors
+## Sync live intelligence
 
-With the backend environment active, fetch and merge both official sources:
+MITRE and Malpedia can be synchronized without credentials:
 
 ```bash
 cd backend
-python -m app.worker --once --source all
+python -m app.worker --once --source mitre
+python -m app.worker --once --source malpedia
 ```
 
-You can sync only one source with `--source mitre` or `--source malpedia`.
-The default MITRE URL is the latest Enterprise ATT&CK STIX 2.1 bundle. The
-Malpedia actor metadata endpoints are public; an optional `MALPEDIA_API_TOKEN`
-is supported and is sent only by the backend.
+ThreatFox requires a free abuse.ch Auth-Key. Create one through the
+[abuse.ch Authentication Portal](https://auth.abuse.ch/), then place it in
+`backend/.env` without committing it:
+
+```dotenv
+ABUSECH_AUTH_KEY=your-auth-key
+THREATFOX_DAYS=7
+```
+
+Apply migrations and synchronize recent IOCs:
+
+```bash
+alembic upgrade head
+python -m app.worker --once --source threatfox
+```
+
+`--source all` synchronizes MITRE, Malpedia, and ThreatFox when the Auth-Key is
+configured. Without the key, the worker safely skips ThreatFox. The IOC API
+hides records after their six-month freshness window by default; use
+`include_inactive=true` only for historical review.
 
 Source documentation:
 
 - [MITRE ATT&CK STIX data](https://github.com/mitre-attack/attack-stix-data)
 - [Malpedia API](https://malpedia.caad.fkie.fraunhofer.de/usage/api)
+- [ThreatFox API](https://threatfox.abuse.ch/api/)
 
 ## Docker development
 
 ```bash
 docker compose up --build
 ```
 
-Include the scheduled MITRE and Malpedia sync worker with:
+Include the scheduled actor and IOC synchronization worker with:
 
 ```bash
 docker compose --profile ingestion up --build
@@ -118,7 +137,7 @@ pytest
 ```
 
 The suite covers the API, seed collections, filters, authentication, frontend
-contract, STIX normalization, source-aware actor merging, audit records, and
+contract, STIX normalization, source-aware actor and IOC merging, freshness filtering, audit records, and
 idempotent ingestion.
 
 ## Project structure
@@ -142,7 +161,8 @@ idempotent ingestion.
 - Validate attribution, timestamps, confidence, and provenance before use.
 - Do not automatically visit indicators or execute referenced files.
 - Keep API tokens and database credentials out of Git and in a secret store.
-- Malpedia credentials are backend-only and are never sent to the dashboard.
+- Feed credentials are backend-only and are never sent to the dashboard.
+- Review confidence and age before operationally blocking any indicator.
 - Enable authentication, TLS, restricted CORS, and outbound allowlists in production.
 - Set `SEED_ON_STARTUP=false` before loading approved production data.
 - Back up PostgreSQL and monitor health and failed ingestion runs.

backend/.env.example

@@ -8,6 +8,9 @@ API_AUTH_TOKEN=
 MITRE_STIX_URL=https://raw.githubusercontent.com/mitre-attack/attack-stix-data/master/enterprise-attack/enterprise-attack.json
 MALPEDIA_BASE_URL=https://malpedia.caad.fkie.fraunhofer.de
 MALPEDIA_API_TOKEN=
+THREATFOX_API_URL=https://threatfox-api.abuse.ch/api/v1/
+ABUSECH_AUTH_KEY=
+THREATFOX_DAYS=7
 STIX_FEED_URL=
 INGESTION_INTERVAL_SECONDS=3600
 REQUEST_TIMEOUT_SECONDS=120

backend/alembic/versions/0004_threatfox_iocs.py

@@ -0,0 +1,75 @@
+"""Add live IOC context and multi-source provenance."""
+
+from alembic import op
+import sqlalchemy as sa
+
+revision = "0004_threatfox_iocs"
+down_revision = "0003_actor_sources"
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    with op.batch_alter_table("indicators") as batch:
+        batch.add_column(sa.Column("confidence_score", sa.Integer(), nullable=True))
+        batch.add_column(sa.Column("threat_type", sa.String(120), nullable=True))
+        batch.add_column(sa.Column("malware", sa.String(160), nullable=True))
+        batch.add_column(sa.Column("tags", sa.JSON(), nullable=False, server_default=sa.text("'[]'")))
+        batch.add_column(sa.Column("reference_url", sa.String(1000), nullable=True))
+        batch.add_column(sa.Column("expires_at", sa.DateTime(timezone=True), nullable=True))
+        batch.add_column(sa.Column("last_synced_at", sa.DateTime(timezone=True), nullable=True))
+        batch.add_column(sa.Column("is_active", sa.Boolean(), nullable=False, server_default=sa.true()))
+    op.create_index("ix_indicators_expires_at", "indicators", ["expires_at"])
+    op.create_index("ix_indicators_is_active", "indicators", ["is_active"])
+
+    op.create_table(
+        "indicator_sources",
+        sa.Column("id", sa.Integer(), primary_key=True),
+        sa.Column("indicator_id", sa.Integer(), sa.ForeignKey("indicators.id", ondelete="CASCADE"), nullable=False),
+        sa.Column("source_id", sa.Integer(), sa.ForeignKey("sources.id", ondelete="CASCADE"), nullable=False),
+        sa.Column("external_id", sa.String(160), nullable=False),
+        sa.Column("reference_url", sa.String(1000), nullable=True),
+        sa.Column("confidence_score", sa.Integer(), nullable=True),
+        sa.Column("first_seen", sa.DateTime(timezone=True), nullable=True),
+        sa.Column("last_seen", sa.DateTime(timezone=True), nullable=True),
+        sa.Column("expires_at", sa.DateTime(timezone=True), nullable=True),
+        sa.Column("last_synced_at", sa.DateTime(timezone=True), nullable=False),
+        sa.Column("is_active", sa.Boolean(), nullable=False, server_default=sa.true()),
+        sa.UniqueConstraint("source_id", "external_id", name="uq_indicator_sources_external_identity"),
+        sa.UniqueConstraint("indicator_id", "source_id", name="uq_indicator_sources_indicator_source"),
+    )
+    op.create_index("ix_indicator_sources_indicator_id", "indicator_sources", ["indicator_id"])
+    op.create_index("ix_indicator_sources_source_id", "indicator_sources", ["source_id"])
+
+    connection = op.get_bind()
+    connection.execute(
+        sa.text(
+            """
+            INSERT INTO indicator_sources (
+                indicator_id, source_id, external_id, reference_url,
+                first_seen, last_seen, last_synced_at, is_active
+            )
+            SELECT indicators.id, indicators.source_id, indicators.external_id, sources.url,
+                   indicators.first_seen, indicators.last_seen, CURRENT_TIMESTAMP, TRUE
+            FROM indicators JOIN sources ON sources.id = indicators.source_id
+            WHERE indicators.source_id IS NOT NULL AND indicators.external_id IS NOT NULL
+            """
+        )
+    )
+
+
+def downgrade() -> None:
+    op.drop_index("ix_indicator_sources_source_id", table_name="indicator_sources")
+    op.drop_index("ix_indicator_sources_indicator_id", table_name="indicator_sources")
+    op.drop_table("indicator_sources")
+    op.drop_index("ix_indicators_is_active", table_name="indicators")
+    op.drop_index("ix_indicators_expires_at", table_name="indicators")
+    with op.batch_alter_table("indicators") as batch:
+        batch.drop_column("is_active")
+        batch.drop_column("last_synced_at")
+        batch.drop_column("expires_at")
+        batch.drop_column("reference_url")
+        batch.drop_column("tags")
+        batch.drop_column("malware")
+        batch.drop_column("threat_type")
+        batch.drop_column("confidence_score")