Align FailSafeQA compliance with benchmark threshold

[sylvesterkaczmarek]

Summary

• Change the default FailSafeQA compliance threshold from 6 to 4.
• Apply the same benchmark-aligned default to the core runner and both baseline/optimized convenience wrappers.
• Preserve explicit caller-provided thresholds.

Motivation

FailSafeQA uses a 1–6 judge rubric where scores 4, 5, and 6 are compliant. The script's own comment already says rating >= 4 should be treated as compliant, but COMPLIANCE_THRESHOLD and all callable defaults are currently set to 6.

That means the default evaluator only counts perfect score-6 answers as compliant and marks valid score-4/5 answers non-compliant, changing the benchmark definition and depressing the reported robustness/grounding metrics.

The defaults should match the benchmark boundary already documented by the rubric.

Validation

• rating 1–3 -> non-compliant as before
• rating 4–5 -> now compliant under the benchmark definition
• rating 6 -> compliant as before
• explicit compliance_threshold= overrides remain unchanged
• core runner, baseline wrapper, and optimized wrapper now share the same default

Self-review

• Four targeted 6 -> 4 replacements in one file.
• Final diff contains no unrelated formatting or logic changes.
• No model, dataset, prompt, API request, dependency, notebook, or registry changes.
• Searched open PRs and found no competing FailSafeQA threshold fix.

Maintainers may modify the branch if needed.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e29ecdc263

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Align the notebook with the new default threshold

When readers execute the FailSafeQA notebook, its calls to run_failsafeqa omit compliance_threshold, so they now calculate compliance and the derived robustness/grounding metrics at ≥4. However, the following summary cell still defaults to and explicitly passes 6, its saved output reports “Compliance threshold: ≥ 6,” and the narrative interprets the metrics as perfect 6/6 scores. Rerunning the notebook therefore produces CSVs under one metric while the published table and explanation report another; update the threshold-dependent notebook cells, outputs, and bundled results alongside this default, or explicitly retain 6 for those calls.

Useful? React with 👍 / 👎.