CyberIntelCollector

CyberIntelCollector is a Python 3 application for defensive cyber-intelligence research. It collects configured RSS feeds, normalizes article metadata, and stores immutable source material in SQLite.

The initial release deliberately contains no AI or LLM integration.

Features

• RSS sources defined in YAML rather than source code
• Dedicated CISA KEV/advisory, MITRE ATT&CK STIX 2.1, and DC3/DCISE document collectors
• SQLite storage with an append-only articles table
• SHA-256 duplicate detection
• Per-source failure isolation
• Structured JSON Lines logging
• Repeatable command-line collection summary
• Versioned, deterministic article enrichment and daily delta reports
• Optional macOS launchd automation using project-owned templates

Installation

From the project directory:

cd ~/Desktop/PARA/01_Projects/GitHub/CyberIntelCollector
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt

All dependencies are installed inside .venv; no global installation is required.

Source configuration

Edit config/sources.yaml:

sources:
  - name: Example Security Feed
    url: https://example.org/security/feed.xml
    category: news
    enabled: true

Supported categories are government, vendors, news, and frameworks. Set enabled: false to retain a definition without collecting it.

The same YAML file also configures dedicated collectors:

collectors:
  cisa:
    enabled: true
    kev_url: https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json
    advisory_feeds:
      - name: CISA Cybersecurity Advisories
        url: https://www.cisa.gov/cybersecurity-advisories/all.xml
  mitre:
    enabled: true
    enterprise_url: https://raw.githubusercontent.com/mitre-attack/attack-stix-data/master/enterprise-attack/enterprise-attack.json
  dc3:
    enabled: true
    pages: []
    documents:
      - title: DIB-Reported Cyber Threats CY 2026 Q1
        url: https://www.dc3.mil/Portals/100/Documents/DC3/Missions/DCISE/DCISE%20Slick%20Sheets/DIB%20Cyber%20Threats/2026/DCISE-Quarterly-Sheet-CY2026-Q1-Final.pdf

NSA's official cybersecurity publication page was evaluated but is disabled because it rejects automated requests with HTTP 403 and no verified official RSS or Atom endpoint was found.

Collection

Activate the environment and run:

source .venv/bin/activate
python main.py collect

The command reports:

• sources attempted
• sources successful
• sources failed
• new items
• duplicates skipped

Default paths:

Purpose	Path
Source definitions	config/sources.yaml
SQLite database	data/cyber_intel.db
Structured log	logs/collector.jsonl

Override them when testing or operating another environment:

python main.py collect \
  --config /path/to/sources.yaml \
  --database /path/to/cyber_intel.db \
  --log-file /path/to/collector.jsonl

Each log line is a JSON object with a UTC timestamp, level, event, source, URL, and exception details when applicable.

Immutable evidence and derived intelligence

The articles table is the immutable evidence layer. It contains:

id, title, url, source, published_at, collected_at,
summary, content, category, content_hash

content_hash is a unique SHA-256 digest of the immutable collected content. When that digest already exists, the collector skips the duplicate. SQLite triggers reject direct updates and deletions, ensuring previously collected source material cannot be overwritten.

Enrichment never writes annotations into an article or changes its content hash. It creates replaceable, versioned derived records that reference the article ID instead:

• tag_definitions is a shared tag_type/canonical-name lookup, while article_tags retains the rules version and exact matched text/field evidence for every article match. tag_definitions is not a historical copy of each YAML rules version.
• article_cves retains normalized CVE matches, KEV status, and KEV date_added at correlation time.
• article_attack_matches retains exact MITRE ATT&CK object matches and their match basis.
• event_clusters and event_cluster_articles retain derived event grouping and its score/basis evidence.
• rules_versions stores the immutable canonical JSON snapshot and SHA-256 digest for every accepted rules version. Reusing a version for different tags, enabled flags, ATT&CK safety settings, scoring, or clustering settings is rejected. An upgrade reserves historical versions that predate this registry as legacy-unverified, so operators must choose a new version instead of silently attaching new semantics to old derived rows.
• enrichment_runs, article_enrichment_state, and report_runs record the rules version, timing, status, counts, reporting window, and output path. Each enrichment run also records the highest source article ID visible when the run begins.

Derived rows are correctable without changing source evidence. The current pipeline processes only articles that lack article_enrichment_state for the active rules version, so operators must bump the YAML rules version to re-enrich already processed articles after a tagging-rule, ATT&CK matching, or authoritative-catalog change. A catalog refresh alone does not recompute stored ATT&CK or CVE evidence for completed articles. The daily report dynamically joins the current KEV catalog when it renders CVE/KEV status, but the stored per-article correlation evidence can therefore be stale until that versioned re-enrichment occurs. The original articles rows must remain unchanged; they are the source evidence for all derived results.

Normalized authoritative data is stored separately:

• kev_vulnerabilities contains the latest searchable CISA KEV facts.
• attack_objects contains Enterprise ATT&CK groups, software, campaigns, techniques, tactics, and other STIX objects.
• attack_relationships contains MITRE STIX relationships.

The raw KEV catalog and ATT&CK bundle are also retained as immutable article records, preserving provenance when normalized records are refreshed.

The DC3/DCISE collector discovers public PDFs on the configured official pages, downloads the original bytes, and stores them as immutable government records. Revised documents with different bytes receive new SHA-256 hashes; unchanged files are skipped.

DC3's Akamai configuration currently returns HTTP 403 for automated listing page requests. The current configuration therefore uses verified direct official PDF URLs. Page crawling remains supported through pages if DC3 permits automated listing access in the future.

Deterministic enrichment

Tagging, scoring, and clustering configuration lives in config/tagging.yaml. The document is strictly validated and has five top-level keys:

version: "2026-07-28.3"
attack:
  aliases:
    Volt Typhoon:
      - Vanguard Panda
  safe_canonical_names: []
  minimum_canonical_alphanumeric_length: 8
  ambiguous_canonical_names:
    - Windows
tags:
  - type: actor
    canonical_name: Volt Typhoon
    aliases: []
    enabled: true
scoring_weights:
  government_source: 4
  # all eight documented criteria are required
clustering:
  threshold: 3
  max_days: 7
  weights:
    cve: 5
    # attack, actor, malware, sector, and title_similarity are also required

Supported tag types are country, actor, malware, sector, and priority. Review and edit this YAML to add a tag or toggle its optional enabled flag. ATT&CK aliases and safety decisions live only under attack; ordinary tag aliases do not authorize ATT&CK matches. Bump version before re-enriching existing articles after any tagging, ATT&CK matching, authoritative-catalog, scoring, or clustering change. Once a version has been used, its complete canonical snapshot is immutable.

Run deterministic enrichment with:

.venv/bin/python main.py enrich

Optional paths allow an isolated operation or a reviewed rules file:

.venv/bin/python main.py enrich \
  --database /path/to/cyber_intel.db \
  --rules /path/to/tagging.yaml

The engine searches title, summary, and textual content. Configured tag phrases use canonical Unicode normalization (so NFC/NFD-equivalent text matches), case-insensitive word/phrase boundaries, and retain the exact original source substring and field as evidence. CVE extraction is intentionally an ASCII CVE-YYYY-NNNN... pattern and normalizes its matched identifier to uppercase; it does not claim general Unicode phrase semantics. The engine prefers longer configured phrases. It indexes non-revoked, supported ATT&CK objects by their catalog canonical names when those names are long and unambiguous, plus aliases reviewed under attack.aliases. Short or common canonical names are excluded unless explicitly approved under attack.safe_canonical_names; the minimum length and attack.ambiguous_canonical_names form the versioned safety baseline. Unsupported or revoked ATT&CK objects are excluded. Binary content (including PDFs) is deliberately skipped; there is no PDF text extraction, probabilistic entity recognition, semantic model, or inferred relationship analysis.

Each pending article is handled in its own transaction, so one malformed row is recorded as an isolated failure instead of stopping the rest of a run. A rerun with no pending article for the active rules version is idempotent; it does not automatically revisit completed rows when authoritative inputs change. The run captures its maximum visible article ID in the same transaction that records the run and applies that cutoff to enrichment and clustering queries, including the authoritative catalog source articles used for KEV and ATT&CK correlation. Articles and catalog updates collected concurrently above the cutoff remain pending for the next run.

Scoring and clustering limits

The daily brief ranks its window's articles by the explicit scoring_weights: government source, CVE, KEV match, KEV newly added in the report date, priority tag, named actor tag, critical-infrastructure sector tag, and multi-publisher cluster corroboration. Equal scores are ordered by published time, then collection time, then article ID.

Clusters are evidence links, not merged or deduplicated articles. Candidate pairs must be no more than clustering.max_days apart by publication time (or collection time if publication time is absent). A pair receives the full configured weight for every shared CVE, ATT&CK ID, actor, malware, and sector. Its title contribution is the Jaccard similarity of normalized, stop-word-free title-token sets multiplied by title_similarity. Only pairs meeting clustering.threshold qualify. Qualifying pairs are considered in descending score order; a merge is accepted only when the resulting component's entire publication/collection span remains within clustering.max_days. A stable cluster key includes the rules version, cluster settings, article IDs, and match basis. The result is useful for transparent corroboration, but it is not a semantic determination that two reports describe the same incident.

Daily reports

Generate a report for the previous local day (by default America/Chicago):

.venv/bin/python main.py report daily

For a reproducible operator run, specify both the date and, if needed, the timezone:

.venv/bin/python main.py report daily \
  --report-date 2026-07-28 \
  --timezone America/Chicago

The report date is a local calendar date. The program converts local midnight boundaries to an explicit UTC half-open window, [start, end), and selects articles by collected_at in that window. Consequently, daylight-saving days are represented correctly. “New KEV” instead uses CISA's calendar date_added in the same local report date; it is not a permanent article tag.

The default output root is reports; a run writes both:

reports/daily/YYYY-MM-DD/daily-brief.md
reports/daily/YYYY-MM-DD/daily-brief.json

The Markdown brief is readable for operators and the JSON companion contains the same explicit window, rules version, scoring weights, delta, corpus totals, source links, tags, CVE/KEV data, ATT&CK matches, clusters, and top-development evidence. The brief is a deterministic report, not an AI-generated assessment. Use --report-root /path/to/reports to direct output elsewhere.

Every report verifies the immutable rules snapshot and reports enrichment coverage for the selected delta, including exact IDs missing enrichment or still pending clustering. Such a report is written with partial status instead of silently claiming completion. The status is present in Markdown, JSON, command output, and report_runs.

For an ordered manual workflow, run collection, enrichment, and reporting separately. run-daily is the equivalent convenience command and stops at the first failed stage:

.venv/bin/python main.py run-daily --report-date 2026-07-28

macOS launchd automation

The project supplies source templates under config/launchd/; they are not generated files. The templates schedule the three jobs in local macOS time:

Label	Command	Schedule
com.cyberintel.collect	main.py collect	06:00
com.cyberintel.enrich	main.py enrich	06:10
com.cyberintel.report-daily	main.py report daily	06:20

Install or reinstall only those known labels for the current GUI user:

.venv/bin/python scripts/install_launchd.py

The installer validates the project paths and every template's exact label, schedule, log paths, and explicit command arguments before changing anything. For each known label it preserves the prior plist, writes the expanded replacement atomically, checks bootout, bootstrap, and final service status, and restores and re-loads the prior job if replacement fails. Labels are replaced independently, and unrelated LaunchAgents are never altered.

Inspect a job and its logs with:

launchctl print "gui/$UID/com.cyberintel.collect"
launchctl print "gui/$UID/com.cyberintel.enrich"
launchctl print "gui/$UID/com.cyberintel.report-daily"
tail -n 100 logs/launchd/collect.stdout.log
tail -n 100 logs/launchd/collect.stderr.log

The enrich and report labels use matching enrich.* and report-daily.* log names. To exercise the installed sequence without waiting for its schedule, kick start the labels in collection, enrichment, then reporting order:

launchctl kickstart -k "gui/$UID/com.cyberintel.collect"
launchctl kickstart -k "gui/$UID/com.cyberintel.enrich"
launchctl kickstart -k "gui/$UID/com.cyberintel.report-daily"

Official-source evaluations

The standalone collectors.source_evaluation evaluator is intentionally not called during collection. It accepts only HTTPS endpoints on official nsa.gov and fbi.gov domains, with no embedded credentials and only the default HTTPS port. It follows at most five same-official-domain redirects and rejects any protocol downgrade. It records the status, redirect chain, declared and observed content type, structured format, entry count, and required title/link/date fields. It recommends enabling only nonempty, complete RSS, Atom, JSON, or STIX responses; HTML-only, empty, unofficial, blocked, or invalid endpoints remain disabled.

The dated evidence is in docs/source-evaluations/2026-07-28-nsa-fbi.md. As evaluated on 2026-07-29 UTC, NSA's official advisory page returned blocked HTML and had no verified machine-readable endpoint, so it remains disabled. FBI National Press Releases RSS 2.0 was valid and enabled in config/sources.yaml; the FBI News Blog RSS 2.0 endpoint was validly served but empty, so it remains disabled. Do not add HTML scraping or unofficial mirrors merely to claim coverage.

Backup and practice run

Before operating against a live database, create a timestamped backup with SQLite's backup API, record the article count, and compute a SHA-256 digest of ordered id:content_hash values. This uses only the immutable evidence-layer identity:

.venv/bin/python - <<'PY'
from datetime import datetime, timezone
from hashlib import sha256
from pathlib import Path
import sqlite3

database = Path("data/cyber_intel.db")
backup = Path("data/backups") / (
    "cyber_intel-" + datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ") + ".db"
)
backup.parent.mkdir(parents=True, exist_ok=True)
with sqlite3.connect(database) as source, sqlite3.connect(backup) as target:
    source.backup(target)
with sqlite3.connect(database) as connection:
    rows = connection.execute(
        "SELECT id, content_hash FROM articles ORDER BY id"
    )
    digest = sha256(
        "".join(f"{article_id}:{content_hash}\\n" for article_id, content_hash in rows)
        .encode("utf-8")
    ).hexdigest()
    count = connection.execute("SELECT COUNT(*) FROM articles").fetchone()[0]
print(f"backup={backup} articles={count} digest={digest}")
PY

Then follow this evidence-preserving practice run:

1. Save the backup path, count, and digest.
2. Run .venv/bin/python main.py collect and record its collection summary.
3. Record the post-collection digest; enrichment and reporting must not change that immutable-evidence digest.
4. Run .venv/bin/python main.py enrich and record tags, CVEs, KEVs, ATT&CK matches, clusters, and failures.
5. Run .venv/bin/python main.py report daily --report-date YYYY-MM-DD and save both reported output paths.
6. Install and manually kick-start the three LaunchAgents in order, then check launchctl print output and the corresponding launchd logs.
7. Run SQLite integrity and foreign-key checks, then recompute only the ordered digest. It must match the post-collection digest:

.venv/bin/python - <<'PY'
import sqlite3
with sqlite3.connect("data/cyber_intel.db") as connection:
    print(connection.execute("PRAGMA integrity_check").fetchall())
    print(connection.execute("PRAGMA foreign_key_check").fetchall())
PY

.venv/bin/python - <<'PY'
from hashlib import sha256
import sqlite3
with sqlite3.connect("data/cyber_intel.db") as connection:
    rows = connection.execute("SELECT id, content_hash FROM articles ORDER BY id")
    print(sha256(
        "".join(f"{article_id}:{content_hash}\\n" for article_id, content_hash in rows)
        .encode("utf-8")
    ).hexdigest())
PY

8. Run the final offline regression suite after the practice run:

.venv/bin/python -m unittest discover -s tests -v

Keep the backup and hand off the collection/enrichment summaries, report paths, launchd labels/status/log paths, endpoint conclusions, integrity result, and immutable digest comparison. Runtime databases, backups, logs, and reports are ignored by Git; templates, source code, documentation, and tests remain tracked. data/reports/ is retained only as a legacy/custom-output compatibility location; new default reports are written below reports/daily/.

Tests

The test suite uses temporary databases and local feed fixtures:

.venv/bin/python -m unittest discover -s tests -v

The tests do not require internet access.