Add AITG-APP-15: Testing for LLM-as-Judge Oracle Silent Failure

[SuneetMalhotra]

Following Marco's 2026-06-17 invitation in the OWASP #project-ai-testing-guide Slack channel, this PR contributes a new test case for LLMs used as test oracles — the pattern of delegating the correctness decision on system-under-test output to an LLM, common in visual regression, semantic API-response validation, and AI-test-generation quality scoring.

What the test targets

Three concrete failure modes that let an LLM-oracle deployment appear healthy while being systematically wrong:

1. The specificity gap on defect-only corpora — a degenerate verdict=fail wrapper scores 100% accuracy on a corpus that contains only defective examples. Practitioners routinely publish accuracy without specificity and can't tell the difference.
2. Unreported measured-agreement — accuracy alone reported without inter-rater statistics (Cohen's κ, Fleiss' κ, Gwet's AC1) against a human-labelled ground truth.
3. Silent output fabrication — the wrapper coerces empty output, rate-limits, auth failures, timeouts, or parse errors into a syntactically valid fail verdict, indistinguishable from real failures. Downstream metrics count fabricated rows as real.

Structure

Follows the existing AITG-APP template used by AITG-APP-11 (Testing for Hallucinations) and neighbours:

• Summary
• Test Objectives
• How to Test/Payloads (three tables — one per failure mode)
• Real scenario (documented incident: 617 of 1200 silently fabricated rows, +10.4 pp inflated headline accuracy, detected via the control-pair canary pattern)
• Expected Output (six requirements for a trustworthy deployment)
• Remediation (five mitigations tied to the failure modes)
• Suggested Tools (two MIT-licensed open-source harnesses; standard statistical libraries)
• References (nine — three foundational statistics, three practitioner manuscripts under peer review, plus OWASP AITG-APP-11 companion and Sculley et al.)

Files changed

• Document/content/tests/AITG-APP-15_Testing_LLM_as_Judge_Oracle_Reliability.md — the new test (80 lines)
• Document/README.md — one-line TOC insertion after AITG-APP-14

Placement / numbering

Suggested placement as AITG-APP-15 under Section 3.1 AI Application Testing, sitting next to the existing hallucination and explainability tests. Happy to renumber, split, relocate to a new methodology category, or align tone/format per your and @marco / @mat maintenance direction — take whatever guidance you'd like to give.

References cited (author's own peer-reviewed work under review)

• Visual Oracle Bench (Phase 1) — EMSE-S-26-00876 — the source of the specificity discipline and the documented silent-fabrication incident. Zenodo DOI 10.5281/zenodo.20645248. OSF pre-registration 10.17605/OSF.IO/CSKUY.
• Cross-Layer Observability for LLM-Assisted Test Automation — JSSOFTWARE-D-26-01260 — background on measured-agreement in a self-healing pipeline.
• Backend and Composition Faults in an Unattended LLM-Agent System — ReSAISE 2026 workshop @ ISSRE 2026 — the source of the wrapper fail-fast pattern.

Each has an associated MIT-licensed replication package (Zenodo DOIs cited inline in the test document).

Author

Suneet Malhotra — IEEE Senior Member; ICSE 2027 SEIP Program Committee; ASE 2026 Artifact Evaluation Committee; NeurIPS 2026 Ethics Reviewer.

[SuneetMalhotra]

Housekeeping note for @marco @mat — while reviewing the open PR queue I noticed a potential numbering interaction with #79 (@rashim27us's Recursive Task Chain Manipulation contribution). #79 is titled as AITG-APP-15 in its subject line, though the actual diff extends AITG-APP-06 in place rather than creating a standalone AITG-APP-15 file. I've left review comments on #79 asking for clarification on which design choice is intended.

If the maintainers land on giving #79 the standalone AITG-APP-15 slot, I'm happy to renumber this PR to AITG-APP-16 — it's a straightforward rename of one file, one TOC entry, and internal cross-references. Just let me know which direction you'd prefer.

Content of this PR (LLM-as-Judge Oracle Silent Failure) and #79 (Recursive Task Chain Manipulation) are on distinct topics, so both should be able to land — the only interaction is the number.

Cc @rashim27us in case it helps to coordinate.

[rashim27us]

Thanks @SuneetMalhotra for pointing this out! We've decided to integrate the content of #79 into AITG-APP-06, so AITG-APP-15 is no longer being used from our side. There shouldn't be any numbering conflict now.

[avp9-nexus]

Read the AITG-APP-15 draft in full. The part I would not want lost in review is the malformed verdict class: treating rate-limits, auth failures, timeouts and parse errors as a distinct class excluded from metrics, rather than coercing them into fail. Most evaluation pipelines collapse "the check said no" and "the check could not run" into the same cell, and the second one is invisible precisely when it matters. Nothing in the testing methodology draws that distinction today.

One addition, on the control-pair canary specifically.

As written, the standing canary is one-sided. Expected Output asks for "known-passing examples whose only honest aggregate is 0-of-N fail", and Remediation says it trips when a wrapper "silently defaults to fail on error". Both point the same way. That catches an oracle drifting toward false positives, which is the direction your Real scenario documents.

The direction it cannot see is an oracle that has stopped detecting. A wrapper that defaults to pass, a prompt or model change that makes the judge lenient, or a rule quietly disabled will produce 0-of-N fail on those same control pairs. The canary reads green while the oracle no longer refuses anything. Test 1 catches this in an audit, since recall on defect examples collapses, but Test 1 is a point-in-time exercise and the canary is what ships with every scheduled run.

The fix is small: make the canary set two-sided. Ship a few known-failing examples alongside the known-passing ones, and treat any run where the known-failing set does not come back fail as a blocking condition, exactly like specificity below 1.0 on the passing side. Two numbers instead of one, same mechanism.

There is a working precedent inside OWASP. In the Agent Security Regression Harness, two scenarios are run against both a clean trace and a violation trace, and its CI carries a step named "Fail if any regression-demo result is not fail", plus a separate step that fails the build if the expected result file is missing at all. That second guard is worth noting on its own: a run that produced nothing and a run that produced a green result look identical unless something asserts the artifact exists.

If this is useful, it is one bullet in Expected Output and one in Remediation. Happy to draft the wording if you would rather not reopen the file yourself.