[barkha1research-ai]

Closes #4405.

Adds ConfAIde (Mireshghallah et al., ICLR 2024), a benchmark for whether models can tell when it is appropriate to share information, based on contextual integrity. As far as I can tell this is the first privacy benchmark in the harness.

What's added

• lm_eval/tasks/confaide/: nine tasks and a confaide group, one task per tier and question type.
  • confaide_tier1, confaide_tier2a, confaide_tier2b: Pearson correlation of the model's ratings with the human ratings.
  • confaide_tier3_free_response, confaide_tier3_info_accessibility, confaide_tier3_privacy_sharing, confaide_tier3_control: the reference's string-match leakage and error rates.
  • confaide_tier4_meeting_summary, confaide_tier4_action_item: error when the public item is missing or the private one appears.
• The data is loaded from the authors' repository at a pinned commit, since it isn't on the Hub. Prompts, parsing and scoring follow the reference eval.py; the README lists where they differ (no Llama-2 judge metric, max_gen_toks, no re-querying of unparseable ratings).
• The group reports no aggregate, since it mixes correlations and error rates.
• Each prompt is sampled 10 times at temperature 1.0, as in the reference. Tier 3 and 4 metrics also have a _worst_case form (an item fails if any sample fails), which the paper uses for Tier 3.
• tests/test_confaide.py: parsing, prompt construction and scoring, using small fixtures in the benchmark's own file formats (no downloads).

Validation

I ran the reference eval.py (with its model backends stubbed out) and this task on the same responses: a deterministic fake model returned the same canned responses through simple_evaluate, and the reference scored them directly. For all 1,326 documents, with and without the privacy prompt, the prompts, gold fields and per-document metrics were identical, and the aggregates agreed to within floating-point summation order. The tests pass on Python 3.10, 3.12 and 3.14.

Results for three small open models, run through Ollama with local-chat-completions and --apply_chat_template, one sample per prompt:

Task	metric	Llama 3.2 3B	Qwen2.5 7B	Llama 3.1 8B
tier1	pearson ↑	0.408	0.578	0.230
tier2a	pearson ↑	-0.068	0.340	0.303
tier2b	pearson ↑	0.448	0.385	0.404
tier3_free_response	string_match ↓	0.893	0.881	0.356
tier3_info_accessibility	error ↓	0.826	0.652	0.856
tier3_privacy_sharing	error ↓	0.800	0.537	0.730
tier3_control	error ↓	0.052	0.000	0.467
tier4_meeting_summary	error ↓	0.600	0.600	0.400
tier4_action_item	error ↓	0.900	0.850	0.950

These are single-sample numbers, because the API backends send the same seed with every request, so Ollama returns the same response for all 10 repeats (noted in the README). Llama 3.1 8B often refuses ("I cannot provide…"): 57 of its 98 Tier 2a ratings could not be parsed and count as 0, as in the reference, and most of its Tier 3 control errors are refusals.

Notes

• The new_tasks CI job may report Duplicate task 'confaide_…' found in both 'confaide_…' and 'confaide' when a group file and its tasks are both in the changed-files list. The same happens locally with existing groups such as kormedmcqa, so it isn't specific to this change. With only the nine task files in that list, tests/test_tasks.py passes locally (120 tests).
• pre-commit passes on the changed files.