Labelled responses from remote datasets can't reach `HumanLabeledDataset`

[WatchTree-19]

There's no path from the remote dataset loaders to the labelled-data side of scoring.

What's there now. HumanLabeledDataset is only constructible via from_csv(), so scorer evaluation data has to be authored by hand. That's reflected in datasets/scorer_evals/ — refusal.csv is ~105 rows and several of the objective sets are single digits.

What's being dropped. datasets/seed_datasets/remote/ has 61 loaders, and several of the underlying HuggingFace datasets ship human labels alongside model responses. Those are discarded at load time because SeedDataset models prompts only. beaver_tails_dataset.py states it directly:

This loader extracts only the prompts (not the responses) and filters to unsafe entries by default.

aegis_ai_content_safety_dataset.py is the same shape. wildguardmix_dataset.py at least retains its classifier labels in metadata, but still produces a SeedDataset.

So labelled (response, score) pairs are already flowing through the loaders and being dropped, while the one class that needs them can only be fed by hand.

Proposal. A second output path off the existing remote loaders producing HumanLabeledDataset — reusing the fetch logic and the HARM_CATEGORY_ALIAS_OVERRIDES mapping that's already there, but retaining assistant_response and the human label.

Suggest starting narrow: Aegis 2.0 only (CC-BY-4.0, so no licence friction for a repo shipping under MIT) and one harm category that maps cleanly, with tests and a generated scorer_evals CSV so it's usable on merge.

Happy to build it if the shape is right — is HumanLabeledDataset the correct target here, or is there a reason the loaders are prompts-only that I'm missing?

[romanlutz]

The current behavior is largely intentional on the evaluation side. The datasets in scorer_evals are annotated against specific PyRIT standards/harm definitions, and directly mixing labels from external datasets can produce apples-to-oranges comparisons where scorer metrics become influenced by annotation differences.

That said, I agree there is value in preserving labels that already exist in remote datasets rather than dropping them. A useful direction may be to distinguish between:

• evaluation datasets used for benchmarking and scorer comparisons, and
• training datasets derived from external sources that retain their original labels and responses.

Starting with a cleanly licensed source such as Aegis and a harm category that maps directly would be a reasonable experiment.

[WatchTree-19]

Ran this before seeing your comment. Results are in harm-scorer-transfer. Aegis as training source, scorer_evals/harm/ as target, three cheap CPU classifiers.

The annotation gap is worse than just a benchmarking problem. In-distribution AUC is 0.79. Against scorer_evals/harm/ it falls to 0.56, severity correlation r=0.13. So external labels look like weak training signal, not just an unfair yardstick.

Which is where I'd disagree slightly with retaining original labels. Keeping the responses is clearly worth doing. Keeping the labels may not buy much. Aegis ships 16,608 rows with a prompt and a response and no response label at all, and that pool is useful whichever way you go.

Caveat before anyone leans on 0.56: Aegis labels are binary, yours are 0 to 1 severity, and it's one corpus. Construct mismatch, annotation gap and domain shift are all tangled together here.

Per category. Transfer is uneven. violence_multi_score 0.73, hate_speech_multi_score 0.68, sexual 0.67. Then privacy 0.49, self_harm 0.48, and fairness_bias at 0.22, which is inverted. Argues for per-category classifiers rather than one harm model.

Rater agreement. The three multi-rater files let you check the annotators against each other. All three agree exactly on 42% of rows in hate_speech_multi_score and 32% in violence_multi_score. Binarised at 0.5 that goes to 82% and 69%. Almost all the disagreement is in the severity number, not the harmful/not-harmful call. Worth knowing if a scorer is being graded against a single human_score column, and an argument for scoring binary.

Can take the Aegis + one-category experiment further if that's useful.

[WatchTree-19]

Tested the thing I recommended above and it doesn't hold up.

What I checked. Took 1,785 of the Aegis rows and relabelled them with PyRIT's own instrument rather than my own reimplementation: the Likert system prompt from pyrit/datasets/score/likert/likert_system_prompt.yaml, rendered against pyrit/datasets/harm_definition/harm.yaml, scored 1 to 5 by gpt-4o-mini, mapped to 0-1 the way the gold scores are and binarised at 0.5. Parse rate 100%. Rendered prompt is in the repo under prompts/.

Same rows, same classifier, only the training labels differ:

training labels	AUC on scorer_evals/harm/
Nvidia's original	0.533
PyRIT's own judge and harm definition	0.539

+0.006. So keeping the responses and relabelling them to your standard buys nothing measurable, which is the opposite of what I said above.

It isn't that the relabelling was a no-op. The two label sets disagree on 23.5% of the same rows (Nvidia calls 70.2% unsafe, your judge calls 56.4% harmful). Two genuinely different labellings of identical text, same transfer performance.

Where the gap actually comes from.

training data	rows	AUC
Aegis, Nvidia labels	1,785	0.533
Aegis, relabelled to your standard	1,785	0.539
scorer_evals/harm/ itself	100	0.611
scorer_evals/harm/ itself	800	0.732

100 in-domain rows beat 1,785 external ones under either labelling. Aegis and WildGuardMix are general assistant conversations, scorer_evals/harm/ is red-team output, and that difference dominates. The annotation standard was never the binding constraint.

I also replicated the original transfer number on WildGuardMix so it isn't resting on one corpus: 0.775 in-distribution, 0.587 against your gold set, against Aegis at 0.79 and 0.556.

What I think this means for the issue. The bridge is still worth building, pointed somewhere else. The in-domain rows aren't on HuggingFace, PyRIT is already producing them: every scored response in memory sits in exactly the distribution the scorers get measured on. What's missing is human labels on them, and those are the expensive part.

So rather than a loader, an annotation path. Pull scored responses out of memory, rank by where a human label is worth most (scorer sitting near its threshold, or two scorers disagreeing), export the top N in the scorer_evals/harm/ column format for labelling, read the completed file back as a HumanLabeledDataset. Same amount of human effort, spent on the rows that move the number.

Worth being explicit that the scorer's own labels shouldn't go into HumanLabeledDataset directly. That would put judge output into the set the judges are measured against, which breaks the only clean instrument here.

Two caveats. I used the general harm scale rather than per-category definitions, and per-category might recover some of it, though it would need to close 0.08 AUC to reach the 100-row in-domain baseline. And 1,785 rows is a subset, the Nvidia baseline on the full 5,085 is 0.556 rather than 0.533, so the subset costs a little, though both arms here use identical rows.

Everything is in harm-scorer-transfer, sections 6 to 8. I've also corrected the fairness_bias inversion claim from earlier in that repo, it doesn't replicate on WildGuardMix and n was 29.