Two-sided span counting: precision per prediction span, recall per annotation

[omri374]

Summary

SpanEvaluator counted predictions through the annotations they overlapped: a prediction span overlapping two annotations entered num_predicted twice, and a group of same-type spans jointly covering one annotation entered it once. num_predicted therefore drifted both above and below the model's actual output depending on gold layout, an annotation could be counted as both FN and TP, and a blob prediction could earn double TP credit at lenient thresholds (a sloppy span covering two golds scored P=1.0, R=1.0 by being counted twice on both sides).

This PR replaces that with two-sided counting — each metric counted in its natural unit, each unit counted exactly once:

• Recall pass (per annotation) — semantics unchanged: each annotation independently checks whether same-type predictions cover it at IoU ≥ threshold (pairwise for a single span, combined IoU for several) → TP or FN. Predictions are not consumed: one span may satisfy several annotations, so no verdict depends on matching order.
• Precision pass (per prediction span) — num_predicted = actual span count; every span is either credited (participated in ≥1 successful match) or an FP. Precision = (num_predicted − FP) / num_predicted.

The two numerators may legitimately differ (one wide span covering two annotations = 2 recall hits, 1 credited prediction), which is why precision is no longer TP / num_predicted.

Counting contract (all covered by tests)

Scenario	Recall side	Precision side
2 preds jointly cover 1 gold, combined IoU ≥ τ	1 TP	np=2, both credited
2 preds jointly fail	1 FN	np=2, 2 FP (was 1 per group)
2 standalone preds	—	2 FP
1 pred covers 2 golds, each ≥ τ	2 TP	np=1, credited (was np=2, tp=2)
1 pred covers 2 golds, each < τ	2 FN	np=1, 1 FP (was 2)
1 pred matches long gold, swallows short one	1 TP, 1 FN	np=1, credited, 0 FP (was 1 — no more double penalty)

Invariants: TP + FN == num_annotated; num_predicted == actual emitted span count.

Also in this PR

• Single-span coverage uses exact pairwise Span.iou; the combined-IoU path (slightly inflated at boundaries) is reserved for genuine multi-span coverage. Borderline single-span matches can flip at exact threshold values (see the "Cyprus ( Greek )" example below).
• Annotation-centric confusion matrix. Every annotation is in exactly one row cell: (type, type) when covered, (type, predicted type) when a different type covers it at IoU ≥ τ, (type, "O") otherwise, so row totals equal num_annotated at every threshold. When several types reach the threshold on one annotation (possible at τ ≤ 0.5), the annotation's own type claims the cell, else the wrong type with the highest IoU; spans of the other types are false positives in the "O" row. This is the tie rule from Fix confusion matrix consistency for overlapping entity predictions #198, applied to the new matcher; Fix confusion matrix consistency for overlapping entity predictions #198's three TestConfusionMatrixConsistency tests are included unchanged, so Fix confusion matrix consistency for overlapping entity predictions #198 is superseded by this PR. The "O" row holds only prediction spans represented by no annotation cell, so a wrong-type detection is no longer written to both (type, predicted type) and ("O", predicted type). Column totals are not the prediction ledger: one prediction covering two annotations appears in two cells while counting once in num_predicted. Documented in a new "Confusion Matrix and Error Records" section of docs/span_evaluation.md.
• Confusion cells and ModelError records are written by the per-type pass only. With level="both" (the default, also used by calculate_hierarchical_scores) the global PII pass shares the same result object; it now updates the pii_* counters and nothing else. Previously every FP was recorded twice at the binary level, and per-type results gained a spurious ("O", "PII") cell plus a duplicate "PII" FP record at branch and detailed level.
• ~370 lines of matching machinery removed: the single/multiple-overlap scenario dispatch (_compare_single_overlaps, _compare_multiple_overlaps) collapsed into one uniform rule; dead helpers (_find_best_match, _check_if_matched_already, _handle_unmatched_predictions, _update_wrong_entities, _add_to_processed_predictions) deleted.
• Docs: docs/span_evaluation.md and docs/span_matching_strategies.md corrected where they described the old counting (precision formula, prediction counting, multi-span FP accounting) plus a short section on one prediction overlapping multiple annotations; typed docstrings added throughout span_evaluator.py.
• CHANGELOG: three entries under "Unreleased > Behavior Changes" (0.3.2 was released after this branch was opened).

Merge with main

main has been merged into this branch. Conflicts were limited to the CHANGELOG section placement and the calculate_score_on_df docstring, which now also documents the allow_generic_entities parameter added on main. One test from the hierarchy projection work (#196), test_low_iou_descendants_use_the_projected_type, encoded the old per-group FP count; it now expects one FP per failed span (num_predicted == 2, false_positives == 2), still attributed to the projected PERSON label.

Behaviour across hierarchical levels

calculate_hierarchical_scores runs the same evaluator on the binary, branch and detailed projections, each with level="both". The following hold at every level, verified by test_hierarchical_levels_share_one_ledger at τ ∈ {0.5, 0.75, 0.9} and on the full synth_dataset_v2 run described below:

• tp + fn == num_annotated for every type; num_predicted equals the independently counted span total.
• The number of FP and FN ModelError records equals the FP and FN counters.
• Confusion-matrix row totals equal num_annotated at every threshold; only the level's own labels appear in cells and error records (no "PII" leakage into branch or detailed results).
• The pii_* counters are identical at all three levels and equal the binary level's PII per-type counts, because both collapse labels to PII before span creation.
• At binary level no wrong-entity cell is possible, so ("PII", "PII"), ("PII", "O") and ("O", "PII") equal TP, FN and FP exactly.

Effect on a real evaluation run

Notebook 5's pipeline was run on data/synth_dataset_v2.json (1500 sentences): predict, CanonicalMapper with suppress_prediction_only() and the notebook's issue resolutions, then calculate_hierarchical_scores(beta=2). The model was the default Presidio AnalyzerEngine with spaCy en_core_web_lg (the OpenMed Hugging Face model used in the notebook was not available in the environment). Predictions were generated once, and the identical mapped DataFrame was scored with the old evaluator (main) and this branch at τ ∈ {0.5, 0.75, 0.9}.

Headline numbers at τ = 0.75

Metric	Old	New
PII precision	0.680	0.616
PII recall	0.575	0.575
PII F2	0.593	0.582
PII num_predicted	1564	1730
PII true positives	1063	1063
PERSON precision (detailed)	0.708	0.720
PERSON num_predicted (detailed)	821	808
DATE_TIME precision	0.266	0.264
PHONE_NUMBER precision	0.673	0.661
CONTACT precision (branch)	0.827	0.819

• Recall is unchanged at τ = 0.75 and 0.9: every TP and FN count is identical at all three levels.
• num_predicted is now the true span count. An independent count of prediction spans from the span builder gives 1730 at the binary level and, per type at the detailed level, DATE_TIME 375, LOCATION 356, PERSON 808, PHONE_NUMBER 56. The new evaluator reports exactly these; the old one was off in both directions (PII −166, PERSON +13, DATE_TIME −3, PHONE_NUMBER −1, CONTACT −1).
• The binary level moves most because collapsing all types to PII turns fragments of different types (a DATE_TIME house number next to a LOCATION street) into same-type fragments of one annotation, so the old per-group under-count compounds there. Per type, under-counting of fragments and over-counting of blobs partly cancel, which is why those shifts stay within about 0.01.
• At τ = 0.5 the recall side changes on LOCATION only, from two old-code defects: five annotations were counted as both TP and FN, and two borderline single-span matches passed only through the inflated combined-IoU path. LOCATION recall at branch level moves from 0.247 to 0.243; old TP + FN was 609 against 604 annotations, new is 604.

Examples from the dataset

Spans below are after skip-word merging, which is why some predictions appear joined.

Fragments over one annotation: one FP per fragment, not per group (sentence 52, τ = 0.75)

card number 347415977307943 is lost, can you please send a new one to 14 Crown Street Kishiev Squares Suite 321 LONDON United Kingdom 75419 ? I am in Sutri for a business trip

Gold LOCATION covers the address and Sutri (merged across the skip words between them). Predictions: LOCATION Crown Street Kishiev Squares, LOCATION Sutri. Combined IoU is below threshold.

	Old	New
LOCATION num_predicted	1	2
LOCATION FP	1	2

Same pattern on a phone number (sentence 392, τ = 0.75). Gold CONTACT 0490 39 07 81; predictions CONTACT 0490, DATE_TIME 39 07, CONTACT 81. Old: PHONE_NUMBER num_predicted 2, FP 1. New: num_predicted 3, FP 2. The sentence really contains three PHONE_NUMBER spans (the two fragments plus the fax number).

One blob over two annotations: counted once (sentence 109, τ = 0.75)

Tomomi is from Hawkins , Richardson and Santana

Gold PERSON Tomomi, gold ORGANIZATION Hawkins , Richardson and Santana. The model tagged each name as PERSON, and skip-word merging joins them into a single PERSON span covering the whole sentence. It overlaps both annotations and matches neither.

	Old	New
PERSON num_predicted	2	1
PERSON FP	2	1
PERSON FN / ORGANIZATION FN	1 / 1	1 / 1

Same pattern with a correct type on one side (sentence 407, τ = 0.75)

He's injured from the waist down from Australia , but Akifumi just has to get laid.

Prediction LOCATION Australia Akifumi (merged across , but) overlaps gold LOCATION Australia and gold PERSON Akifumi. Old: LOCATION num_predicted 2, FP 2. New: num_predicted 1, FP 1. This over-count is why old PERSON and LOCATION precision were slightly too low.

Annotation counted as both TP and FN (sentence 669, τ = 0.5)

The bus drops you off at 13813 3784 Archwood Avenue St.

Gold LOCATION 13813 3784 Archwood Avenue; predictions DATE_TIME 13813, LOCATION Archwood Avenue St.. The old code processed the DATE_TIME group first and recorded an FN, then processed the LOCATION group and recorded a TP for the same annotation.

	Old	New
LOCATION TP	1	1
LOCATION FN	1	0
LOCATION num_annotated	1	1

Four more sentences (302, 346, 836, 1481) show the same double count.

Borderline single-span match (sentences 292 and 1255, τ = 0.5)

We moved here from Cyprus ( Greek )

Gold LOCATION Cyprus ( Greek ); predictions LOCATION Cyprus, DEMOGRAPHIC Greek. Exact pairwise IoU is 0.4615. The old combined-IoU path computed 0.50 for the same pair and passed the threshold. New: FN plus FP instead of TP.

Verification

• New tests: test_two_sided_counting_semantics (the 6-scenario contract table, incl. the P = (np−fp)/np ≠ tp/np regression case and both ledger invariants), test_single_prediction_overlapping_multiple_annotations_counted_once (the original double-counting repro), test_level_both_records_each_error_once (duplicate records with level="both"), test_global_pass_leaves_no_pii_traces_in_per_type_results, test_hierarchical_levels_share_one_ledger (binary/branch/detailed invariants at τ ∈ {0.5, 0.75, 0.9}), test_annotation_row_is_claimed_by_strongest_match_at_tie, and Fix confusion matrix consistency for overlapping entity predictions #198's TestConfusionMatrixConsistency (3 tests, unchanged). 732 unit tests pass on the merged tree; 9 pre-existing expectations updated (8 in test_span_evaluator.py, 1 in test_hierarchical_evaluation.py), each a direct consequence of the rules above.
• Differential run vs old code (300 real sentences × 18 types × τ ∈ {0.5, 0.75, 0.9}, seeded perturbed predictions): recall side bit-for-bit identical except 3 verdicts, each traced to an old-code defect (1 annotation counted FN+TP; 2 borderline matches passing only via the combined-IoU off-by-one); num_predicted now equals the independently-counted actual span total everywhere (old code was wrong on 6 of 18 types, in both directions).
• Notebook 5 run on synth_dataset_v2.json as described above, including the per-level ledger checks.

Downstream impact

• Anyone computing precision as tp/predicted from per_type counts (or pii_true_positives / pii_predicted) must switch to (predicted − fp)/predicted — tp/predicted can now exceed 1.
• Reported numbers shift on datasets with fragmented or over-wide predictions; that's the correction, not a regression. On synth_dataset_v2.json with the default Presidio analyzer, expect PII precision to step down by about 0.06 and PERSON precision to step up by about 0.01, with recall unchanged.
• Error-analysis helpers that count FP records without an entity filter (e.g. ModelError.most_common_fp_tokens) no longer double count, since the PII pass no longer emits its own FP records.
• Remaining known issues, deliberately out of scope:
  • The combined-IoU char math has an off-by-one (end+1 phantom character, shifted ranges for subsequent spans) — now isolated to the genuine multi-span path and a candidate for a follow-up fix.
  • Same-type coverage pools every overlapping span of that type rather than the best subset, so a good single-span match can be dragged below the threshold by a neighbouring same-type span that only brushes the annotation (one TP becomes one FN plus two FPs). Taking the maximum of the best pairwise IoU and the combined IoU, and crediting only the winning selection, would remove this.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

Wrong-type predictions can still be duplicated in the confusion matrix when they overlap multiple annotations.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Introduces two-sided span counting so recall is annotation-based while precision reflects actual prediction spans.

Changes:

• Reworks matching, precision calculation, and confusion-matrix accounting.
• Adds regression tests for fragmented and over-wide predictions.
• Updates documentation and changelog guidance.

File summaries

File	Description
presidio_evaluator/evaluation/span_evaluator.py	Implements two-sided matching and metrics.
tests/evaluation/test_span_evaluator.py	Tests counting and overlap scenarios.
tests/entity_mapping/test_hierarchical_evaluation.py	Updates projected-label expectations.
docs/span_evaluation.md	Documents metric semantics.
docs/span_matching_strategies.md	Documents overlap handling.
CHANGELOG.md	Records behavioral changes.

Review details

• Files reviewed: 6/6 changed files
• Comments generated: 1
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.