Issue 4 (Backtesting 'suddenly risky' + evaluation artefacts (case studies + error analysis) etc)

[abby-ql]

Owner:

What’s already done: scoring supports trend half-life + sensitivity multiplier; trend history CSV exists; cross-match exists for current trends.

Goal: Produce the required evaluation outputs: 3 case studies, error analysis, and demonstrate “low → medium/high” under trend spikes.

Tasks

• Implement scripts/backtest_rescore.py:

  • reads historical trend snapshots from data/trend_history_uk_demo.csv
  • for each timestamp (or day), rescore a fixed set of historical items
  • output a CSV/JSON report of score changes over time

• Pick 3 concrete case studies where an item’s bucket increases due to trend spikes

• Write a short report in reports/evaluation.md:

  • scoring function summary & trade-offs (precision vs recall)
  • the 3 case studies (before/after with reasons)
  • brief error analysis: false positives/false negatives + how explanations reduce harm

Acceptance criteria

• Running python -m scripts.backtest_rescore produces:

  • reports/backtest_results.csv (or JSON)
  • clearly shows low→medium/high transitions for ≥3 items

• reports/evaluation.md exists and includes required sections (case studies + error analysis + tradeoffs)

Tests

• Unit test for backtest script core function: given a tiny synthetic trend timeline, it produces expected bucket transitions

Dependencies

• Depends on Issue 2 only if you want edge-case narrative reused in the report (not required technically)
• Can run fully offline using included trend_history CSV