Scorer evaluation fails when the dataset harm category differs from the harm definition category

[varunj-msft]

Summary

Scorer.evaluate_async() raises ValueError for any harm dataset whose CSV harm_category label differs from the category declared in its harm-definition YAML. Two of PyRIT's own shipped evaluation datasets hit this, so python -m build_scripts.evaluate_scorers cannot produce metrics for them.

Reproduction

Observed during the v1.1.0 release run of python -m build_scripts.evaluate_scorers:

File "pyrit/score/scorer_evaluation/scorer_evaluator.py", line 418, in evaluate_dataset_async
    score = self._select_evaluation_score(scores=scores, harm_category=harm_category)
File "pyrit/score/scorer_evaluation/scorer_evaluator.py", line 504, in _select_evaluation_score
    raise ValueError(
ValueError: Scorer evaluation requires a score for harm category 'bias',
but the single score returned is categorized as ['fairness_bias'].

and

ValueError: Scorer evaluation requires a score for harm category 'sexual_content',
but the single score returned is categorized as ['sexual'].

Root cause

Two different namespaces are being compared:

Dataset	CSV harm_category	Harm definition	YAML category
scorer_evals/harm/fairness_bias.csv	bias	harm_definition/fairness_bias.yaml	fairness_bias
scorer_evals/harm/sexual.csv	sexual_content	harm_definition/sexual.yaml	sexual

The scorer emits score_category from the YAML, while the evaluator compares it against the dataset label. The codebase already documents that these names legitimately diverge — scorer_evaluator.py:192-193:

# The CSV header is authoritative since the harm_category name may differ from
# the YAML filename (e.g., harm_category="bias" but file is "fairness_bias.yaml").

The remaining 6 of 8 harm datasets (violence, privacy, self_harm, exploits, hate_speech, information_integrity) happen to use identical names on both sides, which is why only these two fail.

Regression window

_select_evaluation_score was introduced by #2491 (0899b2144, "FEAT: Refactoring score model (phase 2)"). The previous implementation performed no category matching at all:

scores = await self.scorer.score_prompts_batch_async(...)
score_values = [score.get_value() for score in scores]

Both fairness_bias_metrics.jsonl and sexual_metrics.jsonl contain rows generated as recently as 0.14.0.dev0, confirming evaluation previously succeeded for these datasets.

Suggested fix

When the scorer returns exactly one score, take it. That is what the function's own docstring already describes:

A lone score is taken as the answer, because most scorers report one verdict and leave score_category empty.

With a single score there is no ambiguity about which score to select, so the category comparison adds no disambiguation value and only produces false failures. The multi-score branch genuinely needs category matching and should keep it.

Alternatively, reconcile the two namespaces explicitly (map dataset label → harm-definition category) so the check can stay strict.

Impact

Evaluation/benchmarking only. Normal scoring (score_async), attacks, and the rest of the runtime are unaffected. Metrics for these two harm categories remain at their 0.14.0.dev0 values.

[varunj-msft]

Follow-up analysis: this is a namespace collision, not a typo

I initially assumed the dataset labels were simply wrong and could be renamed. They cannot — both names are legitimate elsewhere in PyRIT, so this needs a maintainer decision rather than a quick data edit.

Three sources define the name, and they belong to two different namespaces:

Source	fairness_bias dataset	sexual dataset
CSV harm_category column	bias	sexual_content
LikertScalePaths.* config (self_ask_likert_scorer.py:113-120,143-149)	fairness_bias	sexual
harm_definition/*.yaml category:	fairness_bias	sexual

The scorer emits score_category from the YAML, while evaluate_dataset_async reads harm_category from the CSV row (scorer_evaluator.py:390-394). Those are compared directly, so the two namespaces collide.

The CSV values are not arbitrary — they come from PyRIT's harm taxonomy:

• HarmCategory.SEXUAL_CONTENT = "Pornography & Sexual Content" (models/harm_category.py:47)
• "bias" is a registered alias → [REPRESENTATIONAL, HATE_SPEECH] (models/harm_category.py:112)

Conversely, fairness_bias and sexual are not HarmCategory members at all — they are Likert scale identifiers.

The other 6 of 8 harm datasets (violence, privacy, self_harm, exploits, hate_speech, information_integrity) only pass because the two namespaces coincide for those words.

Why the obvious fixes are not obviously right

• Renaming the CSV column (bias → fairness_bias) discards the taxonomy name and conflates a scale identifier with a harm category. It also edits shipped human-labeled data.
• Relaxing the guard to accept any lone score contradicts a deliberate test, test_rejects_a_lone_score_that_names_a_different_harm (tests/unit/score/test_scorer_evaluator.py:974-977), so the strictness was intended.

Suggested direction

LikertScalePaths already pairs a scorer with its dataset explicitly (SEXUAL_SCALE binds sexual.yaml to harm/sexual.csv and declares harm_category="sexual"). Because that pairing is authoritative, the comparison could use the configured dataset_files.harm_category rather than the per-row CSV label — the config already agrees with the YAML, so both blocked datasets would pass while the guard keeps its ability to catch a genuinely mismatched scorer/dataset pairing.

Either way this is a taxonomy decision for the maintainers, so I have not changed data or the guard.

Release impact

None to the shipped runtime — evaluation/benchmarking only. For v1.1.0 the remaining 27 scorers were evaluated; these 2 keep their 0.14.0.dev0 metrics.

[varunj-msft]

Two corrections after a multi-model audit

1. The scope is wider than two scorers

AzureContentFilterScorer hits the identical mismatch. It emits its category from the Azure SDK enum:

# azure_content_filter_scorer.py:374
score_category=[category.value]     # TextCategory -> "Hate" | "SelfHarm" | "Sexual" | "Violence"

while its own registered evaluation datasets label the harm differently:

# azure_content_filter_scorer.py:60-65  (_CATEGORY_EVAL_FILES)
TextCategory.HATE:      (["harm/hate_speech.csv"], "harm/hate_speech_metrics.jsonl", "hate_speech")
TextCategory.SELF_HARM: (["harm/self_harm.csv"],   "harm/self_harm_metrics.jsonl",   "self_harm")
TextCategory.SEXUAL:    (["harm/sexual.csv"],      "harm/sexual_metrics.jsonl",      "sexual")
TextCategory.VIOLENCE:  (["harm/violence.csv"],    "harm/violence_metrics.jsonl",    "violence")

"Hate" vs hate_speech, "SelfHarm" vs self_harm, "Sexual" vs sexual_content (the CSV value), "Violence" vs violence. Three of the four differ, and all four differ from the CSV label in at least case.

These four did not raise during the v1.1.0 evaluation run, which is why the bug looked narrower than it is: RegistryUpdateBehavior.SKIP_IF_EXISTS short-circuits at scorer_evaluator.py:223 — before evaluation ever reaches the comparison at :418 — because each of those four registries already holds a matching entry with num_scorer_trials = 3. The skip hides the defect rather than avoiding it.

Consequence: a fix scoped to likert_fairness_bias_gpt4o and likert_sexual_gpt4o would leave four Azure Content Safety harm baselines latently broken, and they would surface the moment anyone re-runs with ALWAYS_UPDATE or bumps a dataset version.

This also argues for normalising both sides through HarmCategory.parse() rather than comparing raw strings, since that handles the Hate/hate_speech case as well as bias/fairness_bias.

2. Correcting my own wording above

I wrote that sexual_content and bias are "real names in PyRIT's harm taxonomy". That is accurate for bias but imprecise for sexual_content:

• "bias" is a registered alias key — harm_category.py:112 maps it to [REPRESENTATIONAL, HATE_SPEECH].
• "sexual_content" does not appear in harm_category.py at all. The enum member is named SEXUAL_CONTENT, but its value is "Pornography & Sexual Content" (harm_category.py:47). The literal token lives in datasets/seed_datasets/remote/wildguardmix_dataset.py:110,148, where it is a dataset label mapped onto HarmCategory.SEXUAL_CONTENT.

The substance is unchanged — this is a cross-namespace collision rather than a typo — but the original wording over-identified the snake_case token with the enum member.

3. Suggested regression test

Nothing currently cross-checks a Likert scale's YAML category: against its dataset's harm_category, which is why this reached a release. A parametrised test over every LikertScalePaths entry that has evaluation_files, asserting the two resolve to the same harm category, would be cheap and would have caught it pre-merge.