diff --git a/CHANGELOG.md b/CHANGELOG.md index f95d18b..3650a65 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,9 @@ ### Behavior Changes - **Predictions are projected to the deepest annotated ancestor during canonical mapping** — the gold vocabulary decides the granularity, per prediction. A `NAME` prediction is mapped to `PERSON` when the dataset annotates `PERSON`, and `DATE` is mapped to `DATE_TIME` when the dataset annotates `DATE_TIME`. A prediction with no annotated ancestor is left unchanged, so a coarser prediction is never pushed down onto a finer gold label and siblings are never conflated. Datasets that annotate several depths on one branch (e.g. `PERSON` and `TITLE` in `data/synth_dataset_v2.json`) need no mapping decision: `TITLE` predictions stay `TITLE` while `NAME` predictions become `PERSON`, so every annotated depth keeps its own metrics. Mixed annotation depths are reported as an INFO issue. Low-IoU errors are attributed to the projected scoring label. +- **Two-sided (asymmetric) span counting in `SpanEvaluator`** — recall is now counted per annotation and precision per prediction span, replacing per-annotation counting of predictions that could count one prediction span several times (once per annotation it overlapped) or count a group of spans as a single prediction. Every annotation gets exactly one verdict (`TP + FN == num_annotated`), and every prediction span enters `num_predicted` exactly once, as either credited or FP. Precision is now `(num_predicted - false_positives) / num_predicted`; `true_positives` counts covered annotations and may exceed the number of credited predictions (one wide span covering two annotations is two recall hits but one credited prediction), so `true_positives / num_predicted` is no longer a valid precision formula for downstream consumers. Practical effects: a group of same-type spans that jointly fail the combined-IoU test now counts one FP per span (previously one per group); a too-wide span missing several annotations counts one FP (previously one per missed annotation); a span that matches one annotation and merely brushes another is no longer punished twice (FN only, no extra FP). Fixes the old inconsistency where an annotation could be counted as both FN and TP, and `num_predicted` could drift above or below the actual number of predicted spans depending on gold layout. +- **Single-span coverage uses exact pairwise IoU** — when exactly one prediction overlaps an annotation, coverage is measured with the exact pairwise `Span.iou`; the combined-IoU path (which slightly inflates values at span boundaries) is reserved for genuine multi-span coverage. Borderline single-span matches at a threshold boundary may flip compared to previous releases (e.g. IoU 0.4706 previously computed as 0.50 no longer passes τ=0.5). +- **Annotation-centric confusion matrix** — every annotation is recorded in exactly one cell: `(type, type)` when covered, `(type, predicted type)` when a different type covers it at IoU >= threshold, `(type, "O")` otherwise, so row totals equal `num_annotated`. When several types reach the threshold on one annotation (possible at thresholds of 0.5 or below), the annotation's own type claims the cell, else the wrong type with the highest IoU; spans of the other types are false positives in the `"O"` row. The `"O"` row holds only prediction spans that appear in no annotation cell, so a wrong-type detection is no longer written to both `(type, predicted type)` and `("O", predicted type)`. A prediction covering several annotations appears in one cell per annotation while counting once in `num_predicted`. Confusion-matrix cells and `ModelError` records are now written by the per-type pass only; the global PII pass updates the `pii_*` counters and nothing else, so `calculate_score_on_df(level="both")` (the default, also used by `calculate_hierarchical_scores`) no longer records each false positive twice or adds an `("O", "PII")` cell to per-type results. Documented in `docs/span_evaluation.md`. ### Bug Fixes diff --git a/docs/span_evaluation.md b/docs/span_evaluation.md index d193b54..41f1258 100644 --- a/docs/span_evaluation.md +++ b/docs/span_evaluation.md @@ -73,35 +73,68 @@ The matching process follows these steps: ## Metric Calculation -The evaluator calculates both per-entity-type metrics and global PII metrics: +The evaluator calculates both per-entity-type metrics and global PII metrics. +Recall is counted per annotation: each annotation is a true positive if +predictions of its type cover it at IoU ≥ threshold, and a false negative +otherwise. Precision is counted per prediction span: each span is counted once +in `num_predicted`, either credited by a successful match or counted as a +false positive. ### Per-Entity-Type Metrics -- **Precision**: TP / num_predicted +- **Precision**: (num_predicted − FP) / num_predicted - **Recall**: TP / num_annotated - **F-beta**: (1 + beta²) * (precision * recall) / (beta² * precision + recall) +Note that precision is not TP / num_predicted: TP counts covered annotations, +and a single prediction covering two annotations is two TPs but one prediction. + ### Global PII Metrics - Treat every entity type as if it were a single PII type - Calculate global precision, recall, and F-score on PII/not PII values +### Confusion Matrix and Error Records + +The confusion matrix (`EvaluationResult.results`) is annotation-centric: + +- Every annotation lands in exactly one cell: `(type, type)` when covered, + `(type, predicted type)` when a different type covers it at IoU ≥ threshold, + and `(type, "O")` when nothing does. Row totals therefore equal + `num_annotated` per type. When several types reach the threshold on the same + annotation (possible at thresholds of 0.5 or below), the annotation's own + type claims the cell; otherwise the wrong type with the highest IoU does. + Spans of the other types are false positives in the `"O"` row. +- The `"O"` row holds prediction spans that appear in no annotation cell: false + positives that overlap nothing, or overlap an annotation below the threshold. + A prediction already represented by a `(type, predicted type)` cell is not + added to the `"O"` row again. +- Column totals are not the prediction ledger. One prediction that covers two + annotations appears in two cells while counting once in `num_predicted`. Use + `num_predicted` and `false_positives` for prediction-side totals. + +Confusion-matrix cells and `ModelError` records are written by the per-type +pass only. The global PII pass updates the `pii_*` counters and nothing else, +so `calculate_score_on_df(level="both")` records each error once. + ## Evaluation Process 1. For each annotation, find all overlapping prediction spans 2. Group overlapping spans by entity type -3. Calculate combined IoU for each group +3. Calculate the same-type coverage (pairwise IoU for a single span, combined + IoU for several) 4. Determine match status based on IoU and entity type -5. Mark remaining predictions (with no overlap) as FPs +5. Count each prediction span once: credited if it participated in a + successful match, otherwise a false positive See more info on the [Span Matching Strategies](span_matching_strategies.md) document. ## Counting Strategy -- Multiple predictions of the same type overlapping with one annotation count as a single prediction +- Every annotation is counted once in `num_annotated` and receives one verdict + (TP or FN), regardless of how many predictions or types intersect with it +- Every prediction span is counted once in `num_predicted` — a span is not + re-counted per annotation it overlaps, and grouped spans that jointly fail + count one FP each - Different entity types are counted separately -- An annotation is only counted once as annotated (denominator for precision and recall), - regardless of how many types intersect with it - - diff --git a/docs/span_matching_strategies.md b/docs/span_matching_strategies.md index 014368d..9d302f7 100644 --- a/docs/span_matching_strategies.md +++ b/docs/span_matching_strategies.md @@ -96,15 +96,19 @@ When an annotation overlaps with multiple prediction spans: ### 1. Multiple Spans of Same Type -Spans of the same type are combined, and their collective IoU is calculated: +Spans of the same type are combined, and their collective IoU is calculated. +The combined IoU decides the annotation's verdict; the spans themselves are +counted individually in `num_predicted`: - **Example**: - Text: "New York Mets" - Annotation: [ORGANIZATION, ORGANIZATION, ORGANIZATION] - Prediction: [ORGANIZATION, O, ORGANIZATION] - Combined IoU = 0.67 - - If threshold = 0.5: Treated as a match (TP) - - If threshold = 0.75: Treated as a miss (FN) + - If threshold = 0.5: Treated as a match — 1 TP; both spans are credited + (num_predicted: +2, FP: 0) + - If threshold = 0.75: Treated as a miss — 1 FN; each failed span is its + own false positive (num_predicted: +2, FP: +2) ### 2. Multiple Spans of Different Types @@ -118,6 +122,20 @@ Each entity type is evaluated separately against the annotation: - If threshold = 0.5: PERSON is a match but wrong type for LOCATION portion - Result: TP for PERSON, FP for LOCATION +## One Prediction Overlapping Multiple Annotations + +The mirror case: a single prediction overlapping several annotations. Each +annotation measures its own pairwise IoU against the prediction independently; +the prediction itself is counted once in `num_predicted`: + +- Each annotation whose IoU is above the threshold is a TP; each annotation + whose IoU is below it is an FN. +- The prediction is credited if it matched at least one annotation, otherwise + it is a single FP (not one per missed annotation). +- **Example**: gold [John Smith] and [Mary Jones], prediction one PERSON span + over "John Smith met Mary Jones" (IoU ≈ 0.4 per annotation). At + threshold 0.3: 2 TP, num_predicted 1. At threshold 0.9: 2 FN, 1 FP. + ## Real-world Examples ### Example 1: Complex Name with Multiple Parts diff --git a/presidio_evaluator/evaluation/span_evaluator.py b/presidio_evaluator/evaluation/span_evaluator.py index 3288bcd..007320c 100644 --- a/presidio_evaluator/evaluation/span_evaluator.py +++ b/presidio_evaluator/evaluation/span_evaluator.py @@ -68,9 +68,12 @@ def _normalize_tokens( 3. Removing standalone punctuation 4. Removing skip words (common words that shouldn't affect entity matching) - :param tokens: List of token strings to normalize - :param start_indices: List of start indices for each token - :return: Tuple of (normalized tokens, normalized start indices) + :param tokens: (list[str]) Token strings to normalize. + :param start_indices: (list[int] | None) Character start offset of each + token, parallel to ``tokens``; defaults to zeros when omitted. + :return: (tuple[list[str], list[int]]) ``(normalized_tokens, + normalized_start_indices)`` — the surviving tokens (lowercased, + skip words removed) and their start offsets, kept parallel. """ if not start_indices: @@ -91,9 +94,11 @@ def _merge_adjacent_spans(self, spans: list[Span], df: pd.DataFrame) -> list[Spa """ Merge adjacent spans of the same entity type if separated only by skip words / punctuation. - :param spans: List of Span objects to potentially merge - :param df: DataFrame containing the tokens and their positions - :return: List of merged Span objects + :param spans: (list[Span]) Span objects to potentially merge. + :param df: (pd.DataFrame) The sentence's rows, used to inspect the + tokens between two candidate spans. + :return: (list[Span]) Spans sorted by start position, with same-type + neighbors separated only by skip words fused into single spans. """ if not spans: return [] @@ -149,10 +154,11 @@ def _are_spans_adjacent(self, span1: Span, span2: Span, df: pd.DataFrame) -> boo """ Check if two spans are adjacent, i.e., separated only by skipwords / punctuation or whitespace tokens. - :param span1: First Span object - :param span2: Second Span object - :param df: DataFrame containing the tokens - :return: True if spans are adjacent, False otherwise + :param span1: (Span) The earlier span (by start position). + :param span2: (Span) The later span. + :param df: (pd.DataFrame) The sentence's rows, sliced positionally via + the spans' sentence-relative ``token_end``/``token_start``. + :return: (bool) True if every token between the spans is a skip word. """ # token_start/token_end are positions within the sentence, so slice # positionally — the DataFrame's index labels are caller-defined @@ -178,12 +184,12 @@ def calculate_iou( """ Calculate the Intersection over Union (IoU) between two spans at character or token level. - :param span1: First Span object - :param span2: Second Span object - :param ignore_entity_type: If True, ignores the entity type when calculating IoU - :param use_normalized_indices: If True, uses normalized indices for IoU calculation - :param char_based: If True, calculates IoU at character level, else at token level - + :param span1: (Span) First Span object. + :param span2: (Span) Second Span object. + :param ignore_entity_type: (bool) If True, ignores the entity type when calculating IoU. + :param use_normalized_indices: (bool) If True, uses normalized indices for IoU calculation. + :param char_based: (bool) If True, calculates IoU at character level, else at token level. + :return: (float) IoU value between 0 and 1. """ if char_based: iou = span1.iou( @@ -216,10 +222,10 @@ def _token_iou(ann_span: Span, pred_spans: list[Span]) -> float: of 1/3). Spans built by _create_spans always carry per-token indices and take the exact position-aware path. - :param ann_span: The annotation Span to match against - :param pred_spans: One or more prediction Spans; their tokens are pooled - before computing the IoU - :return: IoU value between 0 and 1 + :param ann_span: (Span) The annotation Span to match against. + :param pred_spans: (list[Span]) One or more prediction Spans; their + tokens are pooled before computing the IoU. + :return: (float) IoU value between 0 and 1. """ ann_tokens = SpanEvaluator._positional_tokens(ann_span) pred_token_sets = [ @@ -259,9 +265,9 @@ def _positional_tokens(span: Span) -> set[tuple[int, str]] | None: different positions (e.g. both occurrences in "Michael met Michael") from being treated as the same token. - :param span: Span to extract positional tokens from - :return: Set of (start index, token) pairs, or None if the span does not - carry per-token start indices + :param span: (Span) Span to extract positional tokens from. + :return: (set[tuple[int, str]] | None) Set of (start index, token) + pairs, or None if the span does not carry per-token start indices :raises ValueError: If the span carries per-token start indices whose length does not match the number of normalized tokens """ @@ -280,6 +286,15 @@ def _process_sentence_spans( self, sentence_df: pd.DataFrame, ) -> tuple[list[Span], list[Span]]: + """ + Build the gold and predicted spans for one sentence. + + :param sentence_df: (pd.DataFrame) Rows of a single sentence with + ``token``, ``annotation``, ``prediction`` and ``start_indices`` columns. + :return: (tuple[list[Span], list[Span]]) ``(annotation_spans, + prediction_spans)`` — each built from the corresponding tag column + and merged across skip-word gaps by ``_merge_adjacent_spans``. + """ annotation_spans = self._create_spans(df=sentence_df, column="annotation") prediction_spans = self._create_spans(df=sentence_df, column="prediction") @@ -294,116 +309,6 @@ def _process_sentence_spans( return annotation_spans, prediction_spans - @staticmethod - def _handle_unmatched_predictions( - prediction_spans: list[Span], - matched_preds: set[tuple[str, int, int]], - evaluation_result: EvaluationResult, - ) -> EvaluationResult: - """ - Handle predictions that weren't matched to any annotation. - - :param prediction_spans: List of prediction Span objects - :param matched_preds: Set of already matched prediction spans - :param evaluation_result: EvaluationResult object to update - - """ - if not evaluation_result.model_errors: - evaluation_result.model_errors = [] - - for pred_span in prediction_spans: - pred_span_key = ( - pred_span.entity_type, - pred_span.start_position, - pred_span.end_position, - ) - if pred_span_key not in matched_preds: - evaluation_result.results[("O", pred_span.entity_type)] += 1 - evaluation_result.pii_false_positives += 1 - evaluation_result.per_type[pred_span.entity_type].false_positives += 1 - model_error = ModelError( - error_type=ErrorType.FP, - annotation="O", - prediction=pred_span.entity_type, - full_text=pred_span.entity_value, - token=" ".join(pred_span.normalized_tokens or []), - explanation=f"False positive for {pred_span}", - start=pred_span.start_position, - end=pred_span.end_position, - ) - evaluation_result.model_errors.append(model_error) - - return evaluation_result - - @staticmethod - def _check_if_matched_already( - pred_span: Span, - ann_span: Span, - matched_preds: set, - ) -> bool: - """ - Check if a prediction span is valid for matching with an annotation span. - - A prediction is valid if: - 1. It hasn't already been matched to another annotation - 2. Its entity type matches the annotation's entity type - - Args: - pred_span: The prediction Span to check - ann_span: The annotation Span being matched against - matched_preds: Set of already matched prediction spans - - Returns: - bool: True if the prediction is valid for matching, False otherwise - """ - # Create unique key for the prediction span - pred_span_key = ( - pred_span.entity_type, - pred_span.start_position, - pred_span.end_position, - ) - - # Check if prediction is already matched - if pred_span_key in matched_preds: - return False - - return True - - def _find_best_match( - self, - ann_span: Span, - prediction_spans: list[Span], - matched_preds: set[tuple[str, int, int]], - ) -> tuple[Span | None, float]: - """ - Find the best matching prediction span for a given annotation span. - - :param ann_span: The annotation Span to match against - :param prediction_spans: List of prediction Span objects - :param matched_preds: Set of already matched prediction spans to avoid duplicates - """ - best_match = None - best_iou = 0.0 - - for pred_span in prediction_spans: - if self._check_if_matched_already( - pred_span=pred_span, - ann_span=ann_span, - matched_preds=matched_preds, - ): - iou = self.calculate_iou( - span1=ann_span, - span2=pred_span, - ignore_entity_type=True, - use_normalized_indices=True, - char_based=self.char_based, - ) - if iou > best_iou: - best_iou = iou - best_match = pred_span - - return best_match, best_iou - def _update_result_with_overall_metrics( self, evaluation_result: EvaluationResult, @@ -412,8 +317,10 @@ def _update_result_with_overall_metrics( """ Update the evaluation result with overall metrics and per-type metrics. - :param evaluation_result: EvaluationResult object to update - :param beta: The beta parameter for F-beta score calculation. + :param evaluation_result: (EvaluationResult) Result to update in place — + fills ``pii_precision``, ``pii_recall`` and ``pii_f`` from the + ``pii_*`` counters. + :param beta: (float) The beta parameter for F-beta score calculation. """ precision, recall, f_beta = self._calculate_metrics( @@ -421,6 +328,7 @@ def _update_result_with_overall_metrics( evaluation_result.pii_predicted or 0, evaluation_result.pii_annotated or 0, beta, + false_positives=evaluation_result.pii_false_positives or 0, ) evaluation_result.pii_recall = recall evaluation_result.pii_precision = precision @@ -434,9 +342,10 @@ def _update_per_type_metrics( """ Update per-type metrics in the evaluation result. - :param evaluation_result: EvaluationResult object containing per-type metrics - :param beta: F-beta parameter - + :param evaluation_result: (EvaluationResult) Result whose + ``per_type`` dict (``dict[str, PIIEvaluationMetrics]``) gets its + ``precision``/``recall``/``f_beta`` fields computed from the counts. + :param beta: (float) F-beta parameter. """ for _entity_type, pii_metrics in evaluation_result.per_type.items(): @@ -446,6 +355,7 @@ def _update_per_type_metrics( pii_metrics.num_predicted, pii_metrics.num_annotated, beta, + false_positives=pii_metrics.false_positives, ) pii_metrics.precision = precision pii_metrics.recall = recall @@ -456,8 +366,11 @@ def create_global_entities_df(results_df: pd.DataFrame) -> pd.DataFrame: """ Create a DataFrame containing global PII entities from the results DataFrame. - :param results_df: DataFrame containing the evaluation results - :return: DataFrame with global entities and their counts + :param results_df: (pd.DataFrame) Token-level evaluation results with + ``annotation`` and ``prediction`` columns. + :return: (pd.DataFrame) Copy of ``results_df`` with every non-"O" + annotation and prediction label replaced by ``"PII"``, for the + global PII-vs-O scoring pass. """ # Create a deep copy to avoid modifying the original DataFrame global_df = results_df.copy(deep=True) @@ -493,24 +406,27 @@ def calculate_score_on_df( """ Evaluate predictions against ground truth annotations. - :param results_df: DataFrame containing sentence_id, tokens, token start indices, - annotations and predictions columns — as produced by - model.predict_dataset() and optionally processed by - CanonicalMapper.get_mapped_results_dataframe(). - :param level: Which metrics to compute. One of: + :param results_df: (pd.DataFrame) DataFrame containing sentence_id, tokens, + token start indices, annotations and predictions columns — + as produced by model.predict_dataset() and optionally + processed by CanonicalMapper.get_mapped_results_dataframe(). + :param level: (Literal["entity", "pii", "both"]) Which metrics to compute: - ``"entity"`` — per-entity-type precision/recall/F only - ``"pii"`` — global PII (everything vs ``"O"`` ) metrics only - ``"both"`` (default) — both passes; the returned ``EvaluationResult`` contains per-type **and** global PII metrics. - :param beta: F-beta parameter (default 2). - :param evaluation_result: Optional existing EvaluationResult to accumulate into. - :param allow_generic_entities: Accepted for signature compatibility with - :class:`TokenEvaluator` and with - :meth:`BaseEvaluator.calculate_hierarchical_scores`, which passes it - for every level. Span evaluation compares entity types exactly and - has no generic-entity shortcut, so this parameter has no effect here. - :return: EvaluationResult with the requested metrics populated. + :param beta: (float) F-beta parameter (default 2). + :param evaluation_result: (EvaluationResult | None) Optional existing + EvaluationResult to accumulate into. + :param allow_generic_entities: (bool) Accepted for signature compatibility with + :class:`TokenEvaluator` and with + :meth:`BaseEvaluator.calculate_hierarchical_scores`, which passes it + for every level. Span evaluation compares entity types exactly and + has no generic-entity shortcut, so this parameter has no effect here. + :return: (EvaluationResult) Result with the requested metrics populated — + ``per_type`` for "entity", the ``pii_*`` fields for "pii", + both for "both". """ if level in ("entity", "both"): evaluation_result = self._run_score_pass( @@ -541,15 +457,16 @@ def _run_score_pass( """ Run a single scoring pass over the results DataFrame. - :param per_type: If True, performs per-entity type evaluation; if False, performs - global PII vs non-PII evaluation - :param results_df: DataFrame containing sentence_id, tokens, token start indices, - annotations and predictions columns - :param beta: The beta parameter for F-beta score calculation. Higher values weight - recall more than precision. Default is 2. - :param evaluation_result: Optional existing EvaluationResult to update. If None, - creates a new one. - :return: EvaluationResult object containing computed metrics, counts, and error analysis + :param per_type: (bool) If True, performs per-entity type evaluation; if False, + performs global PII vs non-PII evaluation + :param results_df: (pd.DataFrame) DataFrame containing sentence_id, tokens, + token start indices, annotations and predictions columns + :param beta: (float) The beta parameter for F-beta score calculation. Higher + values weight recall more than precision. Default is 2. + :param evaluation_result: (EvaluationResult | None) Optional existing + EvaluationResult to update. If None, creates a new one. + :return: (EvaluationResult) Result containing computed metrics, counts, + and error analysis """ if not evaluation_result: evaluation_result = EvaluationResult() @@ -585,13 +502,13 @@ def _compare_one_sentence( """ Compare one sentence's annotations and predictions, updating the evaluation result. - :param per_type: If True, performs per-entity type evaluation; if False, performs - global PII vs non-PII evaluation - :param sentence_df: DataFrame containing sentence_id, tokens, token start indices, - annotations and predictions columns - :param evaluation_result: Optional existing EvaluationResult to update. If None, - creates a new one. - + :param per_type: (bool) If True, performs per-entity type evaluation; if False, + performs global PII vs non-PII evaluation + :param sentence_df: (pd.DataFrame) DataFrame containing sentence_id, tokens, + token start indices, annotations and predictions columns + :param evaluation_result: (EvaluationResult | None) Optional existing + EvaluationResult to update. If None, creates a new one. + :return: (EvaluationResult) The updated result. """ if not evaluation_result: evaluation_result = EvaluationResult() @@ -610,11 +527,12 @@ def _create_spans(self, df: pd.DataFrame, column: str) -> list[Span]: """ Create spans from a DataFrame column. - :param df: DataFrame containing the spans. - :param column: Name of the column to extract spans from. - - Returns: - List[Span]: List of Span objects created from the DataFrame. + :param df: (pd.DataFrame) One sentence's rows with ``token``, + ``start_indices`` and the tag column to read. + :param column: (str) Name of the tag column to extract spans from + (``"annotation"`` or ``"prediction"``). + :return: (list[Span]) One Span per maximal run of identically-tagged + tokens; runs whose tokens are all skip words are dropped. """ spans = [] current_entity_type = None @@ -706,7 +624,22 @@ def __create_span( idx: int, normalized_start_indices: list[int], normalized_tokens: list[str], - ): + ) -> Span: + """ + Assemble a Span from the tokens accumulated by ``_create_spans``. + + :param entity_type: (str) Entity type of the span. + :param start_indices: (list[int]) Character start offset of each raw token. + :param token_start: (int) Sentence-relative position of the first token. + :param current_tokens: (list[str]) The raw token strings of the span. + :param idx: (int) Sentence-relative position one past the last token. + :param normalized_start_indices: (list[int]) Character start offsets of + the tokens that survived skip-word normalization. + :param normalized_tokens: (list[str]) The normalized (lowercased, + skip-words removed) token strings. + :return: (Span) Span carrying both raw offsets and normalized + token/offset views used by IoU calculations. + """ return Span( entity_type=entity_type, entity_value=" ".join(current_tokens), @@ -744,20 +677,65 @@ def _calculate_metrics( num_predicted: int, num_annotated: int, beta: float = 2, + false_positives: int | None = None, ) -> tuple[float, float, float]: - """Calculate precision, recall, and F-beta score using the new logic. - - :param true_positives: Number of true positives - :param num_predicted: Number of predicted spans - :param num_annotated: Number of annotated (gold) spans - :param beta: The beta parameter for F-beta score calculation. Default is 2. - :return: Dictionary containing precision, recall, and f-beta metrics + """Calculate precision, recall, and F-beta score. + + Counting is two-sided: recall is true_positives / num_annotated + (annotations covered), while precision is + (num_predicted - false_positives) / num_predicted (prediction spans + credited with a match). The numerators may differ — e.g. one wide + prediction covering two annotations is two recall hits but a single + credited prediction. + + :param true_positives: (int) Number of annotations covered at IoU >= threshold. + :param num_predicted: (int) Number of predicted spans. + :param num_annotated: (int) Number of annotated (gold) spans. + :param beta: (float) The beta parameter for F-beta score calculation. Default is 2. + :param false_positives: (int | None) Number of predicted spans with no successful + match. If None, precision falls back to true_positives / num_predicted. + :return: (tuple[float, float, float]) ``(precision, recall, f_beta)``; + precision/recall are ``np.nan`` when their denominator is 0. """ - precision = self.precision(tp=true_positives, num_predicted=num_predicted) + precision_hits = ( + true_positives + if false_positives is None + else num_predicted - false_positives + ) + precision = self.precision(tp=precision_hits, num_predicted=num_predicted) recall = self.recall(tp=true_positives, num_annotated=num_annotated) f_beta = self.f_beta(precision=precision, recall=recall, beta=beta) return precision, recall, f_beta + @staticmethod + def _span_key(span: Span) -> tuple[str, int, int]: + """Identity of a span for match bookkeeping. + + :param span: (Span) The span to identify. + :return: (tuple[str, int, int]) ``(entity_type, start_position, + end_position)`` — hashable identity used in the pass-tracking sets. + """ + return (span.entity_type, span.start_position, span.end_position) + + def _group_iou( + self, + ann_span: Span, + spans: list[Span], + pairwise_ious: list[float], + ) -> float: + """Coverage of an annotation by a group of same-type spans. + + :param ann_span: (Span) The annotation being covered. + :param spans: (list[Span]) The overlapping prediction spans of one type. + :param pairwise_ious: (list[float]) IoU of each span in ``spans`` + against ``ann_span``, parallel to ``spans`` (one entry per span). + :return: (float) Exact pairwise IoU when ``spans`` has a single span, + else the combined IoU of the whole group. + """ + if len(spans) == 1: + return pairwise_ious[0] + return self._calculate_combined_iou(ann_span, spans) + def _match_predictions_with_annotations( self, annotation_spans: list[Span], @@ -765,371 +743,185 @@ def _match_predictions_with_annotations( evaluation_result: EvaluationResult, per_type: bool = True, ) -> EvaluationResult: + """Match spans and update counts using two-sided counting. + + Recall side (per annotation): each annotation independently asks + whether the predictions of its type cover it at IoU >= threshold — + individually or combined. Covered -> true positive (recall hit), + otherwise false negative. + + Precision side (per prediction): each prediction span enters + num_predicted exactly once, no matter how many annotations it + overlaps. A prediction that participated in at least one successful + same-type match is credited; any other prediction is a false + positive. + + Precision is therefore (num_predicted - false_positives) / + num_predicted while recall is true_positives / num_annotated; the two + numerators may legitimately differ (one wide prediction covering two + annotations is two recall hits but a single credited prediction). + + :param annotation_spans: (list[Span]) Gold spans of one sentence. + :param prediction_spans: (list[Span]) Predicted spans of the same sentence. + :param evaluation_result: (EvaluationResult) Accumulator updated in + place — counts, confusion-matrix ``results`` and ``model_errors``. + Confusion matrix: every annotation is written to exactly one row cell. + A same-type match claims it as ``(type, type)``; otherwise the + strongest wrong type at IoU >= threshold claims it as + ``(type, wrong type)``; otherwise it is ``(type, "O")``. Prediction + spans represented by no annotation cell are counted in the ``"O"`` row. + + Confusion-matrix cells (``results``) and ``ModelError`` records are + written only when ``per_type`` is True. The global PII pass + (``per_type=False``) updates the ``pii_*`` counters and nothing else, + so ``calculate_score_on_df(level="both")`` records each error once. + + :param per_type: (bool) If True, update ``per_type`` metrics per entity + type; if False, update the global ``pii_*`` counters only. + :return: (EvaluationResult) The same ``evaluation_result``, updated. + """ if not evaluation_result.model_errors: evaluation_result.model_errors = [] - # Track which prediction spans have been processed - processed_predictions: set[tuple[str, int, int]] = set() + # Prediction spans that participated in >= 1 successful same-type match. + successful_predictions: set[tuple[str, int, int]] = set() + # Prediction spans overlapping >= 1 annotation (for FP explanations). + overlapping_predictions: set[tuple[str, int, int]] = set() + # Prediction spans already represented in the confusion matrix by a + # wrong-entity cell (ann_type, pred_type) — each span appears in the + # matrix once, so these skip the ("O", pred_type) row. + wrong_entity_predictions: set[tuple[str, int, int]] = set() + # --- Recall pass: one verdict per annotation --- for ann_span in annotation_spans: ann_type = ann_span.entity_type self._add_to_annotated(evaluation_result, per_type, ann_type) - # Find all overlapping prediction spans with IoU > 0, regardless of type overlapping_preds = self._get_all_overlapping(ann_span, prediction_spans) - - self._add_to_processed_predictions( - processed_predictions, - overlapping_preds, + for pred_span, _ in overlapping_preds: + overlapping_predictions.add(self._span_key(pred_span)) + spans_by_type, iou_by_type = self._group_spans_by_type(overlapping_preds) + + same_type_spans = spans_by_type.get(ann_type, []) + iou = ( + self._group_iou(ann_span, same_type_spans, iou_by_type[ann_type]) + if same_type_spans + else 0.0 ) - if not overlapping_preds: # scenario 2- there is no prediction - if per_type: - evaluation_result.per_type[ann_type].false_negatives += 1 - evaluation_result.results[(ann_type, "O")] += 1 - evaluation_result.model_errors.append( - self._get_model_error( - ann_span=ann_span, - pred_span=None, - error_type=ErrorType.FN, - ), + # Wrong-entity analysis: another type covering this annotation at + # IoU >= threshold. Collected first so the FN branch can put the + # wrong type (rather than "O") in the confusion matrix. + wrong_type_hits: list[tuple[str, list[Span], float]] = [] + if per_type: + for other_type, other_spans in spans_by_type.items(): + if other_type == ann_type: + continue + other_iou = self._group_iou( + ann_span, other_spans, iou_by_type[other_type] ) - else: - evaluation_result.pii_false_negatives += 1 - - elif ( - len(overlapping_preds) == 1 - ): # scenario group 1 (single overlap of same/different type and below/above threshold) - self._compare_single_overlaps( - evaluation_result=evaluation_result, - ann_span=ann_span, - overlapping_preds=overlapping_preds, - per_type=per_type, - ) - - # Handle pred_span aggregation cases - else: # Scenario group 2 - self._compare_multiple_overlaps( - evaluation_result=evaluation_result, - ann_span=ann_span, - overlapping_preds=overlapping_preds, - per_type=per_type, - ) - - # Handle prediction spans that don't overlap with any annotation span - for pred_span in prediction_spans: - pred_key = ( - pred_span.entity_type, - pred_span.start_position, - pred_span.end_position, - ) + if other_iou >= self.iou_threshold: + wrong_type_hits.append((other_type, other_spans, other_iou)) - # If this prediction has not been processed (no overlap with any annotation) - if pred_key not in processed_predictions: - if per_type: - evaluation_result.per_type[ - pred_span.entity_type - ].false_positives += 1 - evaluation_result.per_type[pred_span.entity_type].num_predicted += 1 - else: - evaluation_result.pii_false_positives += 1 - evaluation_result.pii_predicted += 1 - - # Add to confusion matrix - evaluation_result.results[("O", pred_span.entity_type)] = ( - evaluation_result.results.get(("O", pred_span.entity_type), 0) + 1 - ) - - # Add error - evaluation_result.model_errors.append( - ModelError( - error_type=ErrorType.FP, - annotation="O", - prediction=pred_span.entity_type, - full_text=pred_span.entity_value, - token=" ".join(pred_span.normalized_tokens or []), - explanation=f"False prediction with no overlap: {pred_span.entity_type}", - start=pred_span.start_position, - end=pred_span.end_position, - ), - ) - - return evaluation_result - - def _compare_single_overlaps( - self, - evaluation_result: EvaluationResult, - ann_span: Span, - overlapping_preds: list[tuple[Span, float]], - per_type: bool, - ) -> None: - """Calculate metrics for a single overlapping prediction span (Scenario group 1).""" - - ann_type = ann_span.entity_type - pred_span, iou = overlapping_preds[0] - pred_type = pred_span.entity_type - if iou >= self.iou_threshold: # Scenarios 1 (TP) or 4 (Wrong Entity) - if pred_type == ann_type: # scenario 1 (TP) + if same_type_spans and iou >= self.iou_threshold: if per_type: evaluation_result.per_type[ann_type].true_positives += 1 - evaluation_result.per_type[pred_type].num_predicted += 1 - evaluation_result.results[(ann_type, pred_type)] += 1 + evaluation_result.results[(ann_type, ann_type)] += 1 else: evaluation_result.pii_true_positives += 1 - evaluation_result.pii_predicted += 1 + for pred_span in same_type_spans: + successful_predictions.add(self._span_key(pred_span)) + # The TP cell claims the annotation's row; another type that + # also reached the threshold is only a false positive. + wrong_type_hits = [] elif per_type: evaluation_result.per_type[ann_type].false_negatives += 1 - evaluation_result.per_type[pred_type].false_positives += 1 - evaluation_result.per_type[pred_type].num_predicted += 1 - evaluation_result.model_errors.append( - self._get_model_error( - ann_span=ann_span, - pred_span=pred_span, - error_type=ErrorType.WrongEntity, - iou=iou, - ), - ) + if wrong_type_hits: + # One wrong-entity cell per annotation: the strongest wrong + # type (highest IoU, then name) claims the row; the others + # fall to the "O" row in the precision pass. + wrong_type_hits = [ + min(wrong_type_hits, key=lambda hit: (-hit[2], hit[0])) + ] + else: + evaluation_result.results[(ann_type, "O")] += 1 + # Attach the closest evidence to the FN record: a same-type + # prediction below threshold, else any overlapping prediction. + fn_pred = same_type_spans[0] if same_type_spans else None + fn_iou = iou + if fn_pred is None and overlapping_preds: + fn_pred, fn_iou = overlapping_preds[0] evaluation_result.model_errors.append( self._get_model_error( ann_span=ann_span, - pred_span=pred_span, + pred_span=fn_pred, error_type=ErrorType.FN, - iou=iou, + iou=fn_iou, ), ) - evaluation_result.model_errors.append( - self._get_model_error( - ann_span=ann_span, - pred_span=pred_span, - error_type=ErrorType.FP, - iou=iou, - ), - ) - evaluation_result.results[(ann_type, pred_type)] += 1 else: evaluation_result.pii_false_negatives += 1 - evaluation_result.pii_false_positives += 1 - evaluation_result.pii_predicted += 1 - elif ( - ann_type == pred_type - ): # Scenario 5a (FN and FP - treat as separate entities) - if per_type: - evaluation_result.per_type[ann_type].false_negatives += 1 - evaluation_result.per_type[pred_type].false_positives += 1 - evaluation_result.per_type[pred_type].num_predicted += 1 + for other_type, other_spans, other_iou in wrong_type_hits: + evaluation_result.results[(ann_type, other_type)] += 1 + for wrong_span in other_spans: + wrong_entity_predictions.add(self._span_key(wrong_span)) evaluation_result.model_errors.append( self._get_model_error( ann_span=ann_span, - pred_span=pred_span, - error_type=ErrorType.FN, - iou=iou, - ), - ) - evaluation_result.model_errors.append( - self._get_model_error( - ann_span=ann_span, - pred_span=pred_span, - error_type=ErrorType.FP, - iou=iou, + pred_span=other_spans[0], + error_type=ErrorType.WrongEntity, + iou=other_iou, ), ) - evaluation_result.results[(ann_type, "O")] += 1 - evaluation_result.results[("O", pred_type)] += 1 + + # --- Precision pass: each prediction span is counted exactly once --- + for pred_span in prediction_spans: + pred_key = self._span_key(pred_span) + if per_type: + evaluation_result.per_type[pred_span.entity_type].num_predicted += 1 else: - evaluation_result.pii_false_negatives += 1 - evaluation_result.pii_false_positives += 1 evaluation_result.pii_predicted += 1 - elif per_type: - evaluation_result.per_type[ann_type].false_negatives += 1 - evaluation_result.per_type[pred_type].false_positives += 1 - evaluation_result.per_type[pred_type].num_predicted += 1 + if pred_key in successful_predictions: + continue - # Add two errors, one as FP and the other as FN (not WrongEntity due to low IoU) - evaluation_result.model_errors.append( - self._get_model_error( - ann_span=ann_span, - pred_span=pred_span, - error_type=ErrorType.FN, - iou=iou, - ), - ) + if not per_type: + # The global PII pass maintains the pii_* counters only; the + # confusion matrix and error records are written by the + # per-type pass, so a level="both" run records each error once. + evaluation_result.pii_false_positives += 1 + continue + + evaluation_result.per_type[pred_span.entity_type].false_positives += 1 + if pred_key not in wrong_entity_predictions: + evaluation_result.results[("O", pred_span.entity_type)] = ( + evaluation_result.results.get(("O", pred_span.entity_type), 0) + 1 + ) + if pred_key in overlapping_predictions: + explanation = ( + f"Entity {pred_span.entity_type} falsely detected: overlaps " + f"annotation(s) but no match reached " + f"threshold={self.iou_threshold}" + ) + else: + explanation = ( + f"False prediction with no overlap: {pred_span.entity_type}" + ) evaluation_result.model_errors.append( - self._get_model_error( - ann_span=ann_span, - pred_span=pred_span, + ModelError( error_type=ErrorType.FP, - iou=iou, + annotation="O", + prediction=pred_span.entity_type, + full_text=pred_span.entity_value, + token=" ".join(pred_span.normalized_tokens or []), + explanation=explanation, + start=pred_span.start_position, + end=pred_span.end_position, ), ) - evaluation_result.results[(ann_type, "O")] += 1 - evaluation_result.results[("O", pred_type)] += 1 - else: - evaluation_result.pii_false_negatives += 1 - evaluation_result.pii_false_positives += 1 - evaluation_result.pii_predicted += 1 - - def _compare_multiple_overlaps( - self, - evaluation_result: EvaluationResult, - ann_span: Span, - overlapping_preds: list[tuple[Span, float]], - per_type: bool, - ) -> None: - """Calculate metrics for an annotation span overlapping with multiple pred spans.""" - - annotation_was_counted = ( - False # Only count as FN once if matched multiple times - ) - ann_type = ann_span.entity_type - # Group overlapping spans by entity type - spans_by_type, iou_by_type = self._group_spans_by_type(overlapping_preds) - - # Calculate cumulative IoU per type - cumulative_iou_by_type = {} - for entity_type, spans in spans_by_type.items(): - cumulative_iou_by_type[entity_type] = self._calculate_combined_iou( - ann_span, - spans, - ) - - for cumulative_type, iou_per_type in cumulative_iou_by_type.items(): - # Check if there are spans of the same type as the annotation (Scenario 6) - if ann_type == cumulative_type: - same_type_spans = spans_by_type[cumulative_type] - - if iou_per_type >= self.iou_threshold: - # Scenario 6A: Cumulative IoU with spans of the same type > threshold - if per_type: - if not annotation_was_counted: - annotation_was_counted = True - evaluation_result.per_type[ - ann_span.entity_type - ].true_positives += 1 - evaluation_result.per_type[cumulative_type].num_predicted += 1 - evaluation_result.results[(ann_type, ann_type)] += 1 - else: - evaluation_result.pii_true_positives += 1 - evaluation_result.pii_predicted += 1 - # Scenario 6B: Cumulative IoU with spans of the same type < threshold - elif per_type: - if not annotation_was_counted: - evaluation_result.per_type[ - ann_span.entity_type - ].false_negatives += 1 - evaluation_result.model_errors.append( - self._get_model_error( - ann_span=ann_span, - pred_span=same_type_spans[0], - error_type=ErrorType.FN, - iou=iou_per_type, - ), - ) - evaluation_result.results[(ann_type, "O")] += 1 - annotation_was_counted = True - # For low IoU same-type cases, treat predictions as false positives - evaluation_result.per_type[cumulative_type].false_positives += 1 - evaluation_result.per_type[cumulative_type].num_predicted += 1 - evaluation_result.model_errors.append( - self._get_model_error( - ann_span=ann_span, - pred_span=same_type_spans[0], - error_type=ErrorType.FP, - iou=iou_per_type, - ), - ) - evaluation_result.results[("O", cumulative_type)] += 1 - - else: - if not annotation_was_counted: - evaluation_result.pii_false_negatives += 1 - annotation_was_counted = True - evaluation_result.pii_false_positives += 1 - evaluation_result.pii_predicted += 1 - - else: - # Scenarios 7a,b: Cumulative IoU with spans of a different type - different_type_spans = spans_by_type[cumulative_type] - - if iou_per_type >= self.iou_threshold: - # Scenario 7A: Cumulative IoU with spans of a different type > threshold - if per_type: - if not annotation_was_counted: - evaluation_result.per_type[ann_type].false_negatives += 1 - evaluation_result.model_errors.append( - self._get_model_error( - ann_span=ann_span, - pred_span=different_type_spans[0], - error_type=ErrorType.FN, - iou=iou_per_type, - ), - ) - annotation_was_counted = True - - evaluation_result.per_type[cumulative_type].false_positives += 1 - evaluation_result.per_type[cumulative_type].num_predicted += 1 - evaluation_result.model_errors.append( - self._get_model_error( - ann_span=ann_span, - pred_span=different_type_spans[0], - error_type=ErrorType.WrongEntity, - iou=iou_per_type, - ), - ) - - evaluation_result.model_errors.append( - self._get_model_error( - ann_span=ann_span, - pred_span=different_type_spans[0], - error_type=ErrorType.FP, - iou=iou_per_type, - ), - ) - evaluation_result.results[(ann_type, cumulative_type)] += 1 - else: - if not annotation_was_counted: - evaluation_result.pii_false_negatives += 1 - annotation_was_counted = True - evaluation_result.pii_false_positives += 1 - evaluation_result.pii_predicted += 1 - - # Scenario 7B: Cumulative IoU with spans of a different type < threshold - elif per_type: - if not annotation_was_counted: - evaluation_result.per_type[ann_type].false_negatives += 1 - evaluation_result.model_errors.append( - self._get_model_error( - ann_span=ann_span, - pred_span=different_type_spans[0], - error_type=ErrorType.FN, - iou=iou_per_type, - ), - ) - annotation_was_counted = True - - evaluation_result.per_type[cumulative_type].false_positives += 1 - evaluation_result.per_type[cumulative_type].num_predicted += 1 - - # Add two errors, one as FP and the other as FN (not WrongEntity due to low IoU) - - evaluation_result.model_errors.append( - self._get_model_error( - ann_span=ann_span, - pred_span=different_type_spans[0], - error_type=ErrorType.FP, - iou=iou_per_type, - ), - ) - evaluation_result.results[(ann_type, "O")] += 1 - evaluation_result.results[("O", cumulative_type)] += 1 - else: - if not annotation_was_counted: - evaluation_result.pii_false_negatives += 1 - annotation_was_counted = True - evaluation_result.pii_false_positives += 1 - evaluation_result.pii_predicted += 1 + return evaluation_result @staticmethod def _group_spans_by_type( @@ -1138,7 +930,15 @@ def _group_spans_by_type( """ Group spans by entity type and their corresponding IoU values. - :param overlapping_preds: List of spans to group, with their corresponding IoU values. + :param overlapping_preds: (list[tuple[Span, float]]) Prediction spans to + group, each paired with its IoU against the annotation, as produced + by ``_get_all_overlapping``. + :return: (tuple[dict[str, list[Span]], dict[str, list[float]]]) Two + parallel dicts keyed by entity type: ``spans_by_type[t][i]`` is a + prediction span of type ``t`` and ``iou_by_type[t][i]`` is that same + span's IoU. Keys are exactly the types present in + ``overlapping_preds``; both are defaultdicts, so missing keys yield + empty lists (prefer ``.get`` to avoid inserting keys on lookup). """ spans_by_type = defaultdict(list) iou_by_type = defaultdict(list) @@ -1148,33 +948,27 @@ def _group_spans_by_type( iou_by_type[pred_span.entity_type].append(iou) return spans_by_type, iou_by_type - @staticmethod - def _add_to_processed_predictions( - processed_predictions: set[tuple[str, int, int]], - overlapping_preds: list[tuple[Span, float]], - ) -> None: - """ - Update processed predictions set with all overlapping predictions. - :param processed_predictions: Set of already processed prediction spans - :param overlapping_preds: List of tuples containing overlapping prediction spans and their IoU scores - - """ - # Add all overlapping predictions to processed set - for pred_span, _ in overlapping_preds: - pred_key = ( - pred_span.entity_type, - pred_span.start_position, - pred_span.end_position, - ) - processed_predictions.add(pred_key) - def _get_model_error( self, ann_span: Span | None, pred_span: Span | None, error_type: ErrorType, iou: float = 0.0, - ): + ) -> ModelError: + """ + Build a ModelError record for error analysis. + + :param ann_span: (Span | None) The gold span involved, or None for a + standalone false positive. + :param pred_span: (Span | None) The predicted span involved, or None + for a clean miss. + :param error_type: (ErrorType) FN, FP or WrongEntity. + :param iou: (float) The IoU that drove the verdict; quoted in the + explanation text. + :return: (ModelError) Record with annotation/prediction labels, the + active span's text and offsets, and a human-readable explanation. + """ + def get_explanation(): pred_type = pred_span.entity_type if pred_span else "O" ann_type = ann_span.entity_type if ann_span else "O" @@ -1240,9 +1034,11 @@ def _get_all_overlapping( ) -> list[tuple[Span, float]]: """Get all prediction spans that overlap with the annotation span, regardless of type. - :param ann_span: The annotation Span to match against - :param prediction_spans: List of all prediction Span objects - :return: List of tuples containing overlapping prediction spans and their IoU scores + :param ann_span: (Span) The annotation Span to match against. + :param prediction_spans: (list[Span]) All prediction spans of the sentence. + :return: (list[tuple[Span, float]]) ``(prediction span, IoU)`` pairs for + every prediction with IoU > 0 against ``ann_span``, sorted by the + prediction's start position. """ overlapping_preds = [] @@ -1255,48 +1051,20 @@ def _get_all_overlapping( return overlapping_preds - @staticmethod - def _update_wrong_entities( - overlapping_preds, - annotated_entity_type, - matched_predictions, - ann_span, - evaluation_result, - ) -> EvaluationResult: - non_type_matching_preds = [ - (p, iou) - for p, iou in overlapping_preds - if p.entity_type != annotated_entity_type - and (p.entity_type, p.start_position, p.end_position) - not in matched_predictions - ] - - for non_matching_pred, iou in non_type_matching_preds: - # Record entity type mismatches in error analysis - if non_matching_pred.entity_type != ann_span.entity_type: - evaluation_result.model_errors.append( - ModelError( - error_type=ErrorType.WrongEntity, - annotation=ann_span.entity_type, - prediction=non_matching_pred.entity_type, - full_text=ann_span.entity_value, - token=" ".join(ann_span.normalized_tokens), - explanation=f"Wrong entity type: {ann_span.entity_type} detected as {non_matching_pred.entity_type}, iou={iou:.2f}", - ), - ) - evaluation_result.results[ - (ann_span.entity_type, non_matching_pred.entity_type) - ] = ( - evaluation_result.results.get( - (ann_span.entity_type, non_matching_pred.entity_type), - 0, - ) - + 1 - ) - - return evaluation_result + def _add_to_annotated( + self, + evaluation_result: EvaluationResult, + per_type: bool, + entity_type: str, + ) -> None: + """ + Count one annotation in the recall denominator. - def _add_to_annotated(self, evaluation_result, per_type, entity_type) -> None: + :param evaluation_result: (EvaluationResult) Result object to update. + :param per_type: (bool) If True, increment the type's ``num_annotated``; + otherwise increment the global ``pii_annotated``. + :param entity_type: (str) Entity type of the annotation. + """ if per_type: evaluation_result.per_type[entity_type].num_annotated += 1 else: @@ -1310,9 +1078,11 @@ def _calculate_combined_iou( """ Calculate the combined IoU of multiple prediction spans against an annotation span. - :param annotation_span: The annotation span to match against - :param prediction_spans: List of prediction spans that potentially overlap - :return: Combined IoU value between 0 and 1 + :param annotation_span: (Span) The annotation span to match against. + :param prediction_spans: (list[Span]) Prediction spans whose coverage + is pooled before computing the IoU. + :return: (float) Combined IoU value between 0 and 1; 0.0 when + ``prediction_spans`` is empty. """ if not prediction_spans: return 0.0 diff --git a/tests/entity_mapping/test_hierarchical_evaluation.py b/tests/entity_mapping/test_hierarchical_evaluation.py index feec5fa..916995e 100644 --- a/tests/entity_mapping/test_hierarchical_evaluation.py +++ b/tests/entity_mapping/test_hierarchical_evaluation.py @@ -224,9 +224,13 @@ def test_low_iou_descendants_use_the_projected_type(self): char_based=False, ).calculate_hierarchical_scores(results) detailed = scores["detailed"] + # Two NAME spans jointly fail to cover the PERSON annotation: one FN, + # and each failed span is its own false positive, both attributed to + # the projected PERSON label rather than to NAME. assert detailed.per_type["PERSON"].false_negatives == 1 - assert detailed.per_type["PERSON"].false_positives == 1 - assert detailed.results[("O", "PERSON")] == 1 + assert detailed.per_type["PERSON"].num_predicted == 2 + assert detailed.per_type["PERSON"].false_positives == 2 + assert detailed.results[("O", "PERSON")] == 2 assert detailed.results[("O", "NAME")] == 0 def test_custom_hierarchy_is_used_for_descendant_credit(self): @@ -504,3 +508,64 @@ def test_scenario5_detailed_credits_more_specific_prediction(self): assert person_m is not None assert person_m.recall == pytest.approx(1.0, abs=1e-6) assert person_m.precision == pytest.approx(1.0, abs=1e-6) + + +# --------------------------------------------------------------------------- +# Confusion matrix consistency with per_type metrics (multiple overlaps) +# --------------------------------------------------------------------------- + + +class TestConfusionMatrixConsistency: + """The `results` confusion matrix must agree with per_type TP/FN counts + when one annotation overlaps predictions of several types.""" + + _overlap_evaluator = SpanEvaluator( + skip_words=[], iou_threshold=0.5, char_based=False + ) + + def test_tp_annotation_has_no_missed_cell(self): + """A high-IoU same-branch match plus a low-IoU different-type overlap: + the TP annotation must not also produce an (ann_type, 'O') cell.""" + results = _make_single_sentence_results( + ["PERSON", "PERSON", "PERSON"], + ["NAME", "TITLE", "LOCATION"], + ) + scores = self._overlap_evaluator.calculate_hierarchical_scores(results) + branch = scores["branch"] + person_m = branch.per_type["PERSON"] + assert person_m.true_positives == 1 + assert person_m.false_negatives == 0 + assert branch.results[("PERSON", "PERSON")] == 1 + assert branch.results.get(("PERSON", "O"), 0) == 0 + # The low-IoU LOCATION overlap is only a spurious prediction + assert branch.results.get(("O", "LOCATION"), 0) == 1 + + def test_fn_annotation_missed_cell_counted_once(self): + """An annotation overlapping several low-IoU prediction types is one FN + and must contribute exactly one (ann_type, 'O') cell.""" + results = _make_single_sentence_results( + ["PERSON", "PERSON", "PERSON"], + ["LOCATION", "DATE_TIME", "ORGANIZATION"], + ) + scores = self._overlap_evaluator.calculate_hierarchical_scores(results) + detailed = scores["detailed"] + person_m = detailed.per_type["PERSON"] + assert person_m.false_negatives == 1 + assert detailed.results.get(("PERSON", "O"), 0) == 1 + + def test_tp_annotation_has_no_wrong_entity_cell(self): + """A same-type high-IoU match plus a different-type high-IoU overlap: + the TP annotation's row must not also gain a wrong-entity cell.""" + results = _make_single_sentence_results( + ["PERSON", "PERSON", "PERSON", "PERSON"], + ["LOCATION", "LOCATION", "PERSON", "PERSON"], + ) + scores = self._overlap_evaluator.calculate_hierarchical_scores(results) + branch = scores["branch"] + person_m = branch.per_type["PERSON"] + assert person_m.true_positives == 1 + assert person_m.false_negatives == 0 + assert branch.results[("PERSON", "PERSON")] == 1 + assert branch.results.get(("PERSON", "LOCATION"), 0) == 0 + # The different-type predictions are only a spurious prediction + assert branch.results.get(("O", "LOCATION"), 0) == 1 diff --git a/tests/evaluation/test_span_evaluator.py b/tests/evaluation/test_span_evaluator.py index 81e9828..5dac369 100644 --- a/tests/evaluation/test_span_evaluator.py +++ b/tests/evaluation/test_span_evaluator.py @@ -237,13 +237,13 @@ def test_scenario_group1( ["The", "New", "York", "Mets", "visited"], [0, 4, 8, 13, 18], 0, # true positives - 1, # false positives + 2, # false positives (each failed prediction span counts once) 1, # false negatives { ("ORGANIZATION", "O"): 1, - ("O", "ORGANIZATION"): 1, + ("O", "ORGANIZATION"): 2, }, # confusion matrix - [ErrorType.FN, ErrorType.FP], # errors + [ErrorType.FN, ErrorType.FP, ErrorType.FP], # errors ), # Scenario 7A: Cumulative IoU with spans of different types > threshold ( @@ -436,14 +436,14 @@ def test_scenario_group2( ["O", "LOCATION", "ORGANIZATION", "O", "PERSON", "O"], ["The", "John", "Smith", "Jr", "Doe", "visited"], [0, 4, 9, 15, 18, 22], - 1.0, # precision (1 TP out of 1 predicted PII span) + 1.0, # precision (both predicted spans credited by the joint match) 1.0, # recall (1 TP out of 1 annotated PII span) 1.0, # F1 score 1, # true positives 0, # false positives 0, # false negatives 1, # annotated PII spans (one PERSON span) - 1, # predicted PII spans (multiple entity types become single PII span) + 2, # predicted PII spans (each actual span counts once) ), # Global entities with standalone predictions (no annotation overlap) - results in FP count update ( @@ -867,8 +867,14 @@ def test_calculate_iou_token_based(): [0, 4, 8, 13], [ErrorType.FN, ErrorType.FP, ErrorType.WrongEntity], 3, - ["Wrong entity type: LOCATION detected as PERSON"], - {("LOCATION", "PERSON"): 1}, + [ + "Entity LOCATION not detected. iou with PERSON=", + "Wrong entity type: LOCATION detected as PERSON", + "Entity PERSON falsely detected", + ], + # Both spans are represented by the wrong-entity cell only: + # no ("O", PERSON) for the prediction, no (LOCATION, "O") for the gold + {("LOCATION", "PERSON"): 1, ("O", "PERSON"): 0, ("LOCATION", "O"): 0}, ), # Single overlapping prediction: Same type, low IoU → FN ( @@ -907,13 +913,14 @@ def test_calculate_iou_token_based(): ["ADDRESS", "O", "ADDRESS", "ADDRESS", "ADDRESS", "O"], ["123", "Main", "Street", "Suite", "100", "is"], [0, 4, 9, 16, 22, 26], - [ErrorType.FN, ErrorType.FP], - 2, + [ErrorType.FN, ErrorType.FP, ErrorType.FP], + 3, [ "Entity ADDRESS not detected due to low iou=", "Entity ADDRESS falsely detected", + "Entity ADDRESS falsely detected", ], - {("ADDRESS", "O"): 1, ("O", "ADDRESS"): 1}, + {("ADDRESS", "O"): 1, ("O", "ADDRESS"): 2}, ), # Multiple overlapping predictions: Same type, low cumulative IoU → FN ( @@ -922,13 +929,14 @@ def test_calculate_iou_token_based(): ["O", "ORGANIZATION", "O", "ORGANIZATION", "O"], ["The", "New", "York", "Mets", "visited"], [0, 4, 8, 13, 18], - [ErrorType.FN, ErrorType.FP], - 2, + [ErrorType.FN, ErrorType.FP, ErrorType.FP], + 3, [ "Entity ORGANIZATION not detected due to low iou", "Entity ORGANIZATION falsely detected", + "Entity ORGANIZATION falsely detected", ], - {("ORGANIZATION", "O"): 1, ("O", "ORGANIZATION"): 1}, + {("ORGANIZATION", "O"): 1, ("O", "ORGANIZATION"): 2}, ), # Multiple overlapping predictions: Different type, high cumulative IoU → WrongEntity ( @@ -941,11 +949,18 @@ def test_calculate_iou_token_based(): 4, [ "Entity PERSON not detected due to low iou", - "Entity PERSON falsely detected", "Wrong entity type: PERSON detected as LOCATION", + "Entity PERSON falsely detected", "Entity LOCATION falsely detected", ], - {("PERSON", "LOCATION"): 1, ("O", "PERSON"): 1, ("PERSON", "O"): 1}, + # Gold PERSON and the LOCATION preds are represented by the + # wrong-entity cell only — neither falls back to the "O" row/column + { + ("PERSON", "LOCATION"): 1, + ("O", "PERSON"): 1, + ("O", "LOCATION"): 0, + ("PERSON", "O"): 0, + }, ), # Multiple overlapping predictions: Different type, low cumulative IoU → FN + FP ( @@ -1021,12 +1036,18 @@ def test_calculate_iou_token_based(): 5, [ "Entity PERSON not detected.", - "Wrong entity type: LOCATION detected as PERSON", "Entity LOCATION not detected. iou with PERSON=1.00", + "Wrong entity type: LOCATION detected as PERSON", "Entity PERSON falsely detected", "False prediction with no overlap: PHONE_NUMBER", ], - {("PERSON", "O"): 1, ("LOCATION", "PERSON"): 1, ("O", "PHONE_NUMBER"): 1}, + { + ("PERSON", "O"): 1, # the missed "Alice" annotation + ("LOCATION", "PERSON"): 1, + ("LOCATION", "O"): 0, # gold LOCATION shown as wrong-entity, not missed + ("O", "PERSON"): 0, # PERSON pred shown as wrong-entity, not FP row + ("O", "PHONE_NUMBER"): 1, + }, ), ], ) @@ -1779,3 +1800,464 @@ def test_multi_sentence_df_does_not_merge_separated_same_type_spans(span_evaluat ) assert person.num_predicted == 4 assert person.true_positives == 4 + + +@pytest.mark.parametrize( + "tau, tokens, annotation, expected", + [ + pytest.param( + 0.9, + ["John", "Smith", "met", "Mary", "Jones"], + ["PERSON", "PERSON", "O", "PERSON", "PERSON"], + # Blob misses both golds (IoU ~0.4 each): both golds are FNs, but + # the blob is ONE wrong prediction, not two. + { + "precision": 0.0, + "recall": 0.0, + "num_predicted": 1, + "num_annotated": 2, + "false_positives": 1, + "false_negatives": 2, + }, + id="strict-blob-misses-both-one-fp", + ), + pytest.param( + 0.3, + ["John", "Smith", "met", "Mary", "Jones"], + ["PERSON", "PERSON", "O", "PERSON", "PERSON"], + # At a deliberately lenient threshold the blob covers both golds: + # full recall credit (2 golds found), full precision credit for + # ONE prediction — not two TPs from a single span. + { + "precision": 1.0, + "recall": 1.0, + "num_predicted": 1, + "num_annotated": 2, + "false_positives": 0, + "false_negatives": 0, + }, + id="lenient-blob-covers-both-counted-once", + ), + pytest.param( + 0.6, + ["John", "met", "Mary", "Jones", "Wilson", "Brown"], + ["PERSON", "O", "PERSON", "PERSON", "PERSON", "PERSON"], + # Short gold ("John", IoU ~0.1) swallowed, long gold + # ("Mary Jones Wilson Brown", IoU ~0.7) matched: the swallowed + # gold is an FN, but the blob already earned its single precision + # entry by matching the long gold — no extra FP, no phantom + # prediction. + { + "precision": 1.0, + "recall": 0.5, + "num_predicted": 1, + "num_annotated": 2, + "false_positives": 0, + "false_negatives": 1, + }, + id="mixed-short-swallowed-fn-only", + ), + ], +) +def test_single_prediction_overlapping_multiple_annotations_counted_once( + span_evaluator, tau, tokens, annotation, expected +): + """A prediction span overlapping several annotations is still ONE prediction. + + Regression test for per-annotation double counting: the matching loop + processes each annotation independently and increments num_predicted (and + TP/FP) for every annotation a prediction overlaps, so a single blob + prediction covering two golds enters the precision denominator twice. + Desired semantics are two-sided: recall asks, per annotation, "was I + covered at IoU >= threshold?"; precision asks, per prediction (counted + once), "did I participate in any successful match?". + """ + prediction = ["PERSON"] * len(tokens) + starts, pos = [], 0 + for tok in tokens: + starts.append(pos) + pos += len(tok) + 1 + + df = pd.DataFrame( + { + "sentence_id": [0] * len(tokens), + "token": tokens, + "annotation": annotation, + "prediction": prediction, + "start_indices": starts, + } + ) + + evaluator = SpanEvaluator(iou_threshold=tau, char_based=True, skip_words=None) + result = evaluator.calculate_score_on_df(results_df=df, level="entity") + metrics = result.per_type["PERSON"] + + for field, want in expected.items(): + got = getattr(metrics, field) + assert got == pytest.approx(want), ( + f"{field}: expected {want}, got {got} " + f"(tp={metrics.true_positives}, fp={metrics.false_positives}, " + f"fn={metrics.false_negatives}, num_predicted={metrics.num_predicted})" + ) + + +@pytest.mark.parametrize( + "tau, tokens, annotation, prediction, expected", + [ + pytest.param( + 0.75, + ["John", "Smith", "Jr", "Doe"], + ["PERSON", "PERSON", "PERSON", "PERSON"], + ["PERSON", "PERSON", "O", "PERSON"], + # Two spans jointly cover the gold at combined IoU >= tau: + # one TP on the recall side, both spans credited on the precision side. + { + "true_positives": 1, + "false_negatives": 0, + "false_positives": 0, + "num_predicted": 2, + "num_annotated": 1, + "precision": 1.0, + "recall": 1.0, + }, + id="1-two-preds-joint-coverage-above-tau-tp", + ), + pytest.param( + 0.75, + ["New", "York", "Mets"], + ["ORGANIZATION", "ORGANIZATION", "ORGANIZATION"], + ["ORGANIZATION", "O", "ORGANIZATION"], + # Two spans jointly fail (combined IoU < tau): the gold is one FN, + # and each failed span is its own FP. + { + "true_positives": 0, + "false_negatives": 1, + "false_positives": 2, + "num_predicted": 2, + "num_annotated": 1, + "precision": 0.0, + "recall": 0.0, + }, + id="2-two-preds-joint-coverage-below-tau-1fn-2fp", + ), + pytest.param( + 0.75, + ["Alice", "visited", "Bob"], + ["O", "O", "O"], + ["PERSON", "O", "PERSON"], + # Two standalone predictions with no gold at all: two FPs. + # Recall is undefined (nothing annotated). + { + "true_positives": 0, + "false_negatives": 0, + "false_positives": 2, + "num_predicted": 2, + "num_annotated": 0, + "precision": 0.0, + "recall": np.nan, + }, + id="3-two-standalone-preds-2fp", + ), + pytest.param( + 0.3, + ["John", "Smith", "met", "Mary", "Jones"], + ["PERSON", "PERSON", "O", "PERSON", "PERSON"], + ["PERSON", "PERSON", "PERSON", "PERSON", "PERSON"], + # One blob covers both golds, each pairwise IoU >= tau: two recall + # TPs, but the blob enters the precision denominator once. + # Precision is (np - fp)/np = 1.0, NOT tp/np (which would be 2.0). + { + "true_positives": 2, + "false_negatives": 0, + "false_positives": 0, + "num_predicted": 1, + "num_annotated": 2, + "precision": 1.0, + "recall": 1.0, + }, + id="4-one-pred-two-golds-above-tau-2tp", + ), + pytest.param( + 0.9, + ["John", "Smith", "met", "Mary", "Jones"], + ["PERSON", "PERSON", "O", "PERSON", "PERSON"], + ["PERSON", "PERSON", "PERSON", "PERSON", "PERSON"], + # One blob misses both golds: two FNs, but only ONE FP — the model + # emitted a single span. + { + "true_positives": 0, + "false_negatives": 2, + "false_positives": 1, + "num_predicted": 1, + "num_annotated": 2, + "precision": 0.0, + "recall": 0.0, + }, + id="5-one-pred-two-golds-below-tau-2fn-1fp", + ), + pytest.param( + 0.6, + ["John", "met", "Mary", "Jones", "Wilson", "Brown"], + ["PERSON", "O", "PERSON", "PERSON", "PERSON", "PERSON"], + ["PERSON", "PERSON", "PERSON", "PERSON", "PERSON", "PERSON"], + # Mixed: the blob matches the long gold (IoU ~0.7) and swallows the + # short one (IoU ~0.1). The miss costs recall once (FN); the blob is + # credited via the long match, so no FP. + { + "true_positives": 1, + "false_negatives": 1, + "false_positives": 0, + "num_predicted": 1, + "num_annotated": 2, + "precision": 1.0, + "recall": 0.5, + }, + id="6-one-pred-long-matched-short-swallowed-1tp-1fn", + ), + ], +) +def test_two_sided_counting_semantics( + span_evaluator, tau, tokens, annotation, prediction, expected +): + """The counting-semantics contract of two-sided matching, one case per scenario. + + Recall side: every annotation gets exactly one verdict (TP if covered by + same-type predictions at IoU >= threshold — pairwise for one span, combined + for several — else FN), so tp + fn == num_annotated. + + Precision side: every prediction span enters num_predicted exactly once and + is either credited (participated in a successful match) or an FP, so + precision == (num_predicted - false_positives) / num_predicted. + """ + starts, pos = [], 0 + for tok in tokens: + starts.append(pos) + pos += len(tok) + 1 + + df = pd.DataFrame( + { + "sentence_id": [0] * len(tokens), + "token": tokens, + "annotation": annotation, + "prediction": prediction, + "start_indices": starts, + } + ) + + evaluator = SpanEvaluator(iou_threshold=tau, char_based=True, skip_words=None) + result = evaluator.calculate_score_on_df(results_df=df, level="entity") + entity_type = next(t for t in annotation + prediction if t != "O") + metrics = result.per_type[entity_type] + + for field, want in expected.items(): + got = getattr(metrics, field) + if isinstance(want, float) and np.isnan(want): + assert np.isnan(got), f"{field}: expected nan, got {got}" + else: + assert got == pytest.approx(want), ( + f"{field}: expected {want}, got {got} " + f"(tp={metrics.true_positives}, fp={metrics.false_positives}, " + f"fn={metrics.false_negatives}, np={metrics.num_predicted}, " + f"na={metrics.num_annotated})" + ) + + # The two ledger invariants hold in every scenario. + assert metrics.true_positives + metrics.false_negatives == metrics.num_annotated + assert metrics.false_positives <= metrics.num_predicted + + +def _single_sentence_df(tokens, annotation, prediction): + starts, pos = [], 0 + for tok in tokens: + starts.append(pos) + pos += len(tok) + 1 + return pd.DataFrame( + { + "sentence_id": [0] * len(tokens), + "token": tokens, + "annotation": annotation, + "prediction": prediction, + "start_indices": starts, + } + ) + + +def test_level_both_records_each_error_once(): + """The global PII pass must not duplicate confusion cells or error records. + + With the default level="both", the per-type pass and the PII pass write + into the same EvaluationResult. Only the per-type pass may touch + ``results`` and ``model_errors``; the PII pass owns the pii_* counters. + """ + df = _single_sentence_df(["Alice", "Smith"], ["PII", "PII"], ["PII", "O"]) + evaluator = SpanEvaluator(iou_threshold=0.9, char_based=True, skip_words=[]) + result = evaluator.calculate_score_on_df(df) + + metrics = result.per_type["PII"] + assert ( + metrics.num_predicted, + metrics.true_positives, + metrics.false_positives, + metrics.false_negatives, + ) == (1, 0, 1, 1) + assert ( + result.pii_predicted, + result.pii_true_positives, + result.pii_false_positives, + result.pii_false_negatives, + ) == (1, 0, 1, 1) + assert result.results[("O", "PII")] == 1 + assert result.results[("PII", "O")] == 1 + error_types = [error.error_type for error in result.model_errors] + assert error_types.count(ErrorType.FP) == 1 + assert error_types.count(ErrorType.FN) == 1 + + +def test_global_pass_leaves_no_pii_traces_in_per_type_results(): + """At entity level, results and errors carry real labels only. + + The PII pass relabels everything to "PII" internally; that label must not + leak into the confusion matrix or the error list of a per-type run. + """ + df = _single_sentence_df( + ["Alice", "Smith", "in", "Paris"], + ["PERSON", "PERSON", "O", "LOCATION"], + ["PERSON", "O", "O", "ORGANIZATION"], + ) + evaluator = SpanEvaluator(iou_threshold=0.9, char_based=True, skip_words=[]) + result = evaluator.calculate_score_on_df(df) + + labels = {label for cell in result.results for label in cell} + labels |= {error.prediction for error in result.model_errors} + labels |= {error.annotation for error in result.model_errors} + assert "PII" not in labels + + fp_records = sum(1 for e in result.model_errors if e.error_type == ErrorType.FP) + fn_records = sum(1 for e in result.model_errors if e.error_type == ErrorType.FN) + assert fp_records == sum(m.false_positives for m in result.per_type.values()) == 2 + assert fn_records == sum(m.false_negatives for m in result.per_type.values()) == 2 + # "Paris" is detected as ORGANIZATION: one wrong-entity cell, no "O" entry + assert result.results[("LOCATION", "ORGANIZATION")] == 1 + assert result.results.get(("O", "ORGANIZATION"), 0) == 0 + # PII pass: "Alice" misses "Alice Smith", "Paris" is found + assert ( + result.pii_predicted, + result.pii_true_positives, + result.pii_false_positives, + result.pii_false_negatives, + ) == (2, 1, 1, 1) + + +@pytest.mark.parametrize("tau", [0.5, 0.75, 0.9]) +def test_hierarchical_levels_share_one_ledger(tau): + """binary, branch and detailed results all obey the same counting contract. + + For every level: tp + fn == num_annotated per type, one error record per + FP and per FN, every annotation in exactly one confusion-matrix row cell + (also at threshold 0.5, where two types can tie on one annotation), labels + restricted to the level's vocabulary, and pii_* counters identical across + levels and equal to the binary level's per-type "PII" counts. + """ + from presidio_evaluator.entity_mapping import CanonicalMapper + + tokens = ["Alice", "Smith", "met", "Bob", "in", "New", "York", "on", + "May", "5", "2020", "call", "555", "1234"] # fmt: skip + annotation = ["PERSON", "PERSON", "O", "PERSON", "O", "LOCATION", "LOCATION", + "O", "DATE_TIME", "DATE_TIME", "DATE_TIME", "O", + "PHONE_NUMBER", "PHONE_NUMBER"] # fmt: skip + prediction = ["PERSON", "O", "O", "LOCATION", "O", "LOCATION", "LOCATION", + "O", "DATE_TIME", "O", "DATE_TIME", "O", + "PHONE_NUMBER", "PHONE_NUMBER"] # fmt: skip + df = _single_sentence_df(tokens, annotation, prediction) + mapper = CanonicalMapper() + mapper.analyze(df) + mapped = mapper.get_mapped_results_dataframe() + + evaluator = SpanEvaluator(iou_threshold=tau, char_based=True, skip_words=[]) + scores = evaluator.calculate_hierarchical_scores(mapped) + + binary_pii = scores["binary"].per_type["PII"] + for level in ("binary", "branch", "detailed"): + result = scores[level] + vocabulary = set(mapped.get_level(level)["annotation"]) | set( + mapped.get_level(level)["prediction"] + ) + for entity_type, m in result.per_type.items(): + assert m.true_positives + m.false_negatives == m.num_annotated, level + assert 0 <= m.false_positives <= m.num_predicted, level + row_total = sum( + count + for (ann, _pred), count in result.results.items() + if ann == entity_type + ) + assert row_total == m.num_annotated, (level, entity_type) + + fp_records = sum(1 for e in result.model_errors if e.error_type == ErrorType.FP) + fn_records = sum(1 for e in result.model_errors if e.error_type == ErrorType.FN) + assert fp_records == sum(m.false_positives for m in result.per_type.values()) + assert fn_records == sum(m.false_negatives for m in result.per_type.values()) + + labels = {label for cell in result.results for label in cell} + labels |= {e.prediction for e in result.model_errors} + labels |= {e.annotation for e in result.model_errors} + assert labels <= vocabulary | {"O"}, (level, labels - vocabulary) + + assert ( + result.pii_predicted, + result.pii_true_positives, + result.pii_false_positives, + result.pii_false_negatives, + ) == ( + binary_pii.num_predicted, + binary_pii.true_positives, + binary_pii.false_positives, + binary_pii.false_negatives, + ), level + + # Binary level: no wrong-entity cells are possible, so the confusion matrix + # is the ledger itself. + binary = scores["binary"] + assert binary.results[("PII", "PII")] == binary_pii.true_positives + assert binary.results[("PII", "O")] == binary_pii.false_negatives + assert binary.results[("O", "PII")] == binary_pii.false_positives + + +def test_annotation_row_is_claimed_by_strongest_match_at_tie(): + """At a threshold two types can both reach, the row still has one cell. + + Same type wins over a wrong type; among wrong types the highest IoU wins + (ties broken by name). Spans of the losing types are plain false positives + in the "O" row, with an FP record but no WrongEntity record. + """ + evaluator = SpanEvaluator(iou_threshold=0.5, char_based=False, skip_words=[]) + + # PERSON gold of 4 tokens: LOCATION on the first half, PERSON on the second. + tp_df = _single_sentence_df( + ["a", "b", "c", "d"], + ["PERSON"] * 4, + ["LOCATION", "LOCATION", "PERSON", "PERSON"], + ) + result = evaluator.calculate_score_on_df(tp_df, level="entity") + assert result.per_type["PERSON"].true_positives == 1 + assert result.results[("PERSON", "PERSON")] == 1 + assert result.results.get(("PERSON", "LOCATION"), 0) == 0 + assert result.results[("O", "LOCATION")] == 1 + assert result.per_type["LOCATION"].false_positives == 1 + assert not [e for e in result.model_errors if e.error_type == ErrorType.WrongEntity] + + # PERSON gold of 4 tokens: LOCATION and ORGANIZATION each cover half. + fn_df = _single_sentence_df( + ["a", "b", "c", "d"], + ["PERSON"] * 4, + ["ORGANIZATION", "ORGANIZATION", "LOCATION", "LOCATION"], + ) + result = evaluator.calculate_score_on_df(fn_df, level="entity") + assert result.per_type["PERSON"].false_negatives == 1 + row = {p: c for (a, p), c in result.results.items() if a == "PERSON" and c} + assert row == {"LOCATION": 1} # equal IoU, alphabetical tie-break + assert result.results[("O", "ORGANIZATION")] == 1 + assert result.results.get(("O", "LOCATION"), 0) == 0 + wrong = [e for e in result.model_errors if e.error_type == ErrorType.WrongEntity] + assert [e.prediction for e in wrong] == ["LOCATION"] + fp_records = [e for e in result.model_errors if e.error_type == ErrorType.FP] + assert sorted(e.prediction for e in fp_records) == ["LOCATION", "ORGANIZATION"]