Two-sided span counting: precision per prediction span, recall per annotation

CHANGELOG.md

@@ -9,6 +9,9 @@
 ### Behavior Changes
 
 - **Predictions are projected to the deepest annotated ancestor during canonical mapping** — the gold vocabulary decides the granularity, per prediction. A `NAME` prediction is mapped to `PERSON` when the dataset annotates `PERSON`, and `DATE` is mapped to `DATE_TIME` when the dataset annotates `DATE_TIME`. A prediction with no annotated ancestor is left unchanged, so a coarser prediction is never pushed down onto a finer gold label and siblings are never conflated. Datasets that annotate several depths on one branch (e.g. `PERSON` and `TITLE` in `data/synth_dataset_v2.json`) need no mapping decision: `TITLE` predictions stay `TITLE` while `NAME` predictions become `PERSON`, so every annotated depth keeps its own metrics. Mixed annotation depths are reported as an INFO issue. Low-IoU errors are attributed to the projected scoring label.
+- **Two-sided (asymmetric) span counting in `SpanEvaluator`** — recall is now counted per annotation and precision per prediction span, replacing per-annotation counting of predictions that could count one prediction span several times (once per annotation it overlapped) or count a group of spans as a single prediction. Every annotation gets exactly one verdict (`TP + FN == num_annotated`), and every prediction span enters `num_predicted` exactly once, as either credited or FP. Precision is now `(num_predicted - false_positives) / num_predicted`; `true_positives` counts covered annotations and may exceed the number of credited predictions (one wide span covering two annotations is two recall hits but one credited prediction), so `true_positives / num_predicted` is no longer a valid precision formula for downstream consumers. Practical effects: a group of same-type spans that jointly fail the combined-IoU test now counts one FP per span (previously one per group); a too-wide span missing several annotations counts one FP (previously one per missed annotation); a span that matches one annotation and merely brushes another is no longer punished twice (FN only, no extra FP). Fixes the old inconsistency where an annotation could be counted as both FN and TP, and `num_predicted` could drift above or below the actual number of predicted spans depending on gold layout.
+- **Single-span coverage uses exact pairwise IoU** — when exactly one prediction overlaps an annotation, coverage is measured with the exact pairwise `Span.iou`; the combined-IoU path (which slightly inflates values at span boundaries) is reserved for genuine multi-span coverage. Borderline single-span matches at a threshold boundary may flip compared to previous releases (e.g. IoU 0.4706 previously computed as 0.50 no longer passes τ=0.5).
+- **Annotation-centric confusion matrix** — every annotation is recorded in exactly one cell: `(type, type)` when covered, `(type, predicted type)` when a different type covers it at IoU >= threshold, `(type, "O")` otherwise, so row totals equal `num_annotated`. When several types reach the threshold on one annotation (possible at thresholds of 0.5 or below), the annotation's own type claims the cell, else the wrong type with the highest IoU; spans of the other types are false positives in the `"O"` row. The `"O"` row holds only prediction spans that appear in no annotation cell, so a wrong-type detection is no longer written to both `(type, predicted type)` and `("O", predicted type)`. A prediction covering several annotations appears in one cell per annotation while counting once in `num_predicted`. Confusion-matrix cells and `ModelError` records are now written by the per-type pass only; the global PII pass updates the `pii_*` counters and nothing else, so `calculate_score_on_df(level="both")` (the default, also used by `calculate_hierarchical_scores`) no longer records each false positive twice or adds an `("O", "PII")` cell to per-type results. Documented in `docs/span_evaluation.md`.
 
 ### Bug Fixes
 

docs/span_evaluation.md

@@ -73,35 +73,68 @@ The matching process follows these steps:
 
 ## Metric Calculation
 
-The evaluator calculates both per-entity-type metrics and global PII metrics:
+The evaluator calculates both per-entity-type metrics and global PII metrics.
+Recall is counted per annotation: each annotation is a true positive if
+predictions of its type cover it at IoU ≥ threshold, and a false negative
+otherwise. Precision is counted per prediction span: each span is counted once
+in `num_predicted`, either credited by a successful match or counted as a
+false positive.
 
 ### Per-Entity-Type Metrics
 
-- **Precision**: TP / num_predicted
+- **Precision**: (num_predicted − FP) / num_predicted
 - **Recall**: TP / num_annotated
 - **F-beta**: (1 + beta²) * (precision * recall) / (beta² * precision + recall)
 
+Note that precision is not TP / num_predicted: TP counts covered annotations,
+and a single prediction covering two annotations is two TPs but one prediction.
+
 ### Global PII Metrics
 
 - Treat every entity type as if it were a single PII type
 - Calculate global precision, recall, and F-score on PII/not PII values
 
+### Confusion Matrix and Error Records
+
+The confusion matrix (`EvaluationResult.results`) is annotation-centric:
+
+- Every annotation lands in exactly one cell: `(type, type)` when covered,
+  `(type, predicted type)` when a different type covers it at IoU ≥ threshold,
+  and `(type, "O")` when nothing does. Row totals therefore equal
+  `num_annotated` per type. When several types reach the threshold on the same
+  annotation (possible at thresholds of 0.5 or below), the annotation's own
+  type claims the cell; otherwise the wrong type with the highest IoU does.
+  Spans of the other types are false positives in the `"O"` row.
+- The `"O"` row holds prediction spans that appear in no annotation cell: false
+  positives that overlap nothing, or overlap an annotation below the threshold.
+  A prediction already represented by a `(type, predicted type)` cell is not
+  added to the `"O"` row again.
+- Column totals are not the prediction ledger. One prediction that covers two
+  annotations appears in two cells while counting once in `num_predicted`. Use
+  `num_predicted` and `false_positives` for prediction-side totals.
+
+Confusion-matrix cells and `ModelError` records are written by the per-type
+pass only. The global PII pass updates the `pii_*` counters and nothing else,
+so `calculate_score_on_df(level="both")` records each error once.
+
 
 ## Evaluation Process
 
 1. For each annotation, find all overlapping prediction spans
 2. Group overlapping spans by entity type
-3. Calculate combined IoU for each group
+3. Calculate the same-type coverage (pairwise IoU for a single span, combined
+   IoU for several)
 4. Determine match status based on IoU and entity type
-5. Mark remaining predictions (with no overlap) as FPs
+5. Count each prediction span once: credited if it participated in a
+   successful match, otherwise a false positive
 
 See more info on the [Span Matching Strategies](span_matching_strategies.md) document.
 
 ## Counting Strategy
 
-- Multiple predictions of the same type overlapping with one annotation count as a single prediction
+- Every annotation is counted once in `num_annotated` and receives one verdict
+  (TP or FN), regardless of how many predictions or types intersect with it
+- Every prediction span is counted once in `num_predicted` — a span is not
+  re-counted per annotation it overlaps, and grouped spans that jointly fail
+  count one FP each
 - Different entity types are counted separately
-- An annotation is only counted once as annotated (denominator for precision and recall),
-  regardless of how many types intersect with it
-
-

docs/span_matching_strategies.md

@@ -96,15 +96,19 @@ When an annotation overlaps with multiple prediction spans:
 
 ### 1. Multiple Spans of Same Type
 
-Spans of the same type are combined, and their collective IoU is calculated:
+Spans of the same type are combined, and their collective IoU is calculated.
+The combined IoU decides the annotation's verdict; the spans themselves are
+counted individually in `num_predicted`:
 
 - **Example**:
     - Text: "New York Mets"
     - Annotation: [ORGANIZATION, ORGANIZATION, ORGANIZATION]
     - Prediction: [ORGANIZATION, O, ORGANIZATION]
     - Combined IoU = 0.67
-    - If threshold = 0.5: Treated as a match (TP)
-    - If threshold = 0.75: Treated as a miss (FN)
+    - If threshold = 0.5: Treated as a match — 1 TP; both spans are credited
+      (num_predicted: +2, FP: 0)
+    - If threshold = 0.75: Treated as a miss — 1 FN; each failed span is its
+      own false positive (num_predicted: +2, FP: +2)
 
 ### 2. Multiple Spans of Different Types
 
@@ -118,6 +122,20 @@ Each entity type is evaluated separately against the annotation:
     - If threshold = 0.5: PERSON is a match but wrong type for LOCATION portion
     - Result: TP for PERSON, FP for LOCATION
 
+## One Prediction Overlapping Multiple Annotations
+
+The mirror case: a single prediction overlapping several annotations. Each
+annotation measures its own pairwise IoU against the prediction independently;
+the prediction itself is counted once in `num_predicted`:
+
+- Each annotation whose IoU is above the threshold is a TP; each annotation
+  whose IoU is below it is an FN.
+- The prediction is credited if it matched at least one annotation, otherwise
+  it is a single FP (not one per missed annotation).
+- **Example**: gold [John Smith] and [Mary Jones], prediction one PERSON span
+  over "John Smith met Mary Jones" (IoU ≈ 0.4 per annotation). At
+  threshold 0.3: 2 TP, num_predicted 1. At threshold 0.9: 2 FN, 1 FP.
+
 ## Real-world Examples
 
 ### Example 1: Complex Name with Multiple Parts