Record reasoning-failure receipts, evaluator-bias evidence, and mitigation outcomes

[mdheller]

Parent

SocioProphet/sociosphere#271

Purpose

Extend the model governance ledger so reasoning-failure receipts become auditable promotion, waiver, rollback, revocation, and factsheet evidence.

Scope

Add examples/validation/docs for ledger records covering:

• reasoning-failure receipt refs from prophet-platform;
• perturbation-suite pass/fail outcomes;
• evaluator-bias checks for LLM-as-judge;
• benchmark fragility and contamination risk;
• exactness failures for strings/counting/arithmetic/config/code identifiers;
• unsupported causal claim flags;
• temporal inconsistency flags;
• multimodal contradiction flags;
• multi-agent termination/verification failures;
• mitigation applied and residual risk.

Ledger decision support

Receipts should be usable by downstream promotion controls:

• pass;
• warn;
• require review;
• waiver with expiration;
• quarantine;
• rollback;
• revoke;
• retrain/reevaluate required.

Acceptance criteria

• A synthetic reasoning-failure receipt can be stored as evidence without raw prompt/customer/personal data.
• Ledger records link subject, model/service/agent/run, perturbation suite, verifier, policy decision, evidence refs, receipt digest, mitigation, and residual risk.
• Factsheet-generation docs distinguish benchmark score from perturbation robustness.
• Waiver/rollback/revocation examples can cite failed reasoning-failure gates.