Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 60 additions & 0 deletions .github/ISSUE_TEMPLATE/bug-report.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
name: Bug report
description: Report a reproducible AgentRunProof defect
title: "[Bug]: "
labels:
- bug
body:
- type: markdown
attributes:
value: |
Thanks for helping improve AgentRunProof. Never paste API keys, credentials, private traces, or unreviewed production certificates.
- type: input
id: agentrunproof-version
attributes:
label: AgentRunProof version
placeholder: 0.2.0
validations:
required: true
- type: input
id: sdk-version
attributes:
label: OpenAI Agents SDK version or source commit
description: Use N/A when the defect is limited to installation, CLI parsing, or local certificate checking.
placeholder: openai-agents 0.21.0, source commit abc123, or N/A
- type: input
id: python-version
attributes:
label: Python and operating system
placeholder: Python 3.12.6 on Ubuntu 24.04 x86_64
validations:
required: true
- type: textarea
id: reproduction
attributes:
label: Minimal reproduction
description: Include the exact command or smallest public-API script.
render: shell
validations:
required: true
- type: textarea
id: observed
attributes:
label: Observed result
description: Include exit status and the relevant invariant/reason, with secrets removed.
validations:
required: true
- type: textarea
id: expected
attributes:
label: Expected result
validations:
required: true
- type: checkboxes
id: checks
attributes:
label: Safety check
options:
- label: I removed credentials, private payloads, and unreviewed production data.
required: true
- label: I searched existing issues for the same behavior.
required: true
8 changes: 8 additions & 0 deletions .github/ISSUE_TEMPLATE/config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
blank_issues_enabled: false
contact_links:
- name: Questions and design discussions
url: https://github.com/FU-max-boop/agentrunproof/discussions
about: Ask how to use AgentRunProof or discuss a runtime contract before proposing it.
- name: OpenAI Agents SDK bug
url: https://github.com/openai/openai-agents-python/issues
about: Report SDK behavior directly upstream when it does not depend on AgentRunProof.
52 changes: 52 additions & 0 deletions .github/ISSUE_TEMPLATE/scenario-proposal.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
name: Scenario proposal
description: Propose a deterministic runtime contract or regression scenario
title: "[Scenario]: "
labels:
- enhancement
body:
- type: textarea
id: contract
attributes:
label: Runtime contract
description: What public SDK behavior should remain true?
validations:
required: true
- type: textarea
id: counterexample
attributes:
label: Minimal counterexample
description: Describe the smallest public-API run that violates or distinguishes the contract.
validations:
required: true
- type: textarea
id: observations
attributes:
label: Required observations and invariants
description: List only observations that can be recorded deterministically without provider calls.
validations:
required: true
- type: input
id: boundary
attributes:
label: Version or commit boundary
placeholder: 0.20.0 FAIL -> commit abc123 PASS
validations:
required: true
- type: textarea
id: ordinary-test
attributes:
label: Why a normal application test is not enough
description: Explain the cross-version, cross-path, or evidence requirement.
validations:
required: true
- type: checkboxes
id: constraints
attributes:
label: Scenario constraints
options:
- label: The proposed built-in makes no model-provider request and needs no API key.
required: true
- label: The proposal does not contain a benchmark answer, hidden test, credential, or private payload.
required: true
- label: I searched existing scenarios and issues for duplicates.
required: true
19 changes: 19 additions & 0 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
## Contract or behavior

Describe the runtime behavior, evidence contract, or documentation boundary changed.

## Verification

List the exact commands and environments used. Include both success and failure-path tests where relevant.

## Compatibility and evidence

- [ ] No certificate schema, canonical identifier, supported SDK range, or immutable evidence changes.
- [ ] Or: the compatibility and release impact is explained above.
- [ ] Built-in execution remains provider-free and requires no API key.
- [ ] No credential, private payload, hidden benchmark material, or generated distribution is committed.
- [ ] `pytest`, Ruff, mypy, build, and Twine checks relevant to this change pass.

## Unverified

State anything that was not or could not be verified.
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,14 @@

All notable changes to AgentRunProof will be documented here.

## Unreleased

- Add a provider-free real-`Runner` tool example, contribution and security guidance, structured
issue and pull-request templates, and a five-minute upstream `RunState` case study.
- Rework the project landing page around a 30-second PyPI check, supported SDK boundaries, honest
upstream impact, and a clear comparison with the SDK's public `ScriptedModel`.
- Improve package discovery metadata and add a reusable social-preview asset.

## 0.2.0 - 2026-08-15

- Verify released `openai-agents` 0.20.0 and 0.21.0 across Python 3.10–3.14 with an exact
Expand Down
62 changes: 62 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# Contributing to AgentRunProof

AgentRunProof welcomes focused contributions that make OpenAI Agents runtime
behavior easier to reproduce, compare, and verify without a model provider.

## Good contributions

- a deterministic scenario for a public SDK runtime contract;
- a minimized regression for streaming, sessions, tools, or resumable state;
- stronger certificate validation or tamper tests;
- compatibility fixes for a released, supported SDK version;
- documentation that makes a result easier to reproduce or interpret.

Built-in scenarios must be provider-free and deterministic. Do not add API keys,
private user data, hidden benchmark material, or task-specific workarounds.

## Before writing a scenario

Open a scenario-proposal issue first when the change introduces a new runtime
contract or certificate field. Describe:

1. the public SDK behavior being checked;
2. the smallest observable counterexample;
3. the expected invariant and failure reason;
4. the SDK versions or source commits involved; and
5. why an ordinary application test is insufficient.

Small documentation and clearly isolated bug fixes may go directly to a pull
request.

## Local setup

```bash
python -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[test,dev]"
pytest
ruff check .
ruff format --check .
mypy src
```

Before submitting, also build and inspect the distributions:

```bash
python -m build
python -m twine check --strict dist/*
```

## Pull requests

Keep each pull request narrow. Include the failing observation or contract,
tests for both success and failure paths, and the exact commands you ran. State
anything you could not verify.

Changes to certificate schemas, normalizers, invariant identifiers, canonical
evidence, supported dependency ranges, or publishing workflows require an
explicit compatibility and release-impact note. Never rewrite immutable
historical evidence in place.

By contributing, you agree that your contribution is licensed under the MIT
License used by this repository.
78 changes: 77 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,75 @@
# AgentRunProof

AgentRunProof is a deterministic runtime-conformance harness for the OpenAI Agents SDK. It drives the real `Runner` with scripted public-`Model` responses, compares observable state across execution paths, and writes a content-addressed conformance record. A failing record carries the normalized counterexample observations.
[![CI](https://github.com/FU-max-boop/agentrunproof/actions/workflows/ci.yml/badge.svg?branch=main&event=push)](https://github.com/FU-max-boop/agentrunproof/actions/workflows/ci.yml?query=branch%3Amain)
[![PyPI](https://img.shields.io/pypi/v/agentrunproof.svg)](https://pypi.org/project/agentrunproof/)
[![Python](https://img.shields.io/pypi/pyversions/agentrunproof.svg)](https://pypi.org/project/agentrunproof/)
[![License: MIT](https://img.shields.io/github/license/FU-max-boop/agentrunproof.svg)](https://github.com/FU-max-boop/agentrunproof/blob/main/LICENSE)

![AgentRunProof: runtime bugs deserve proofs, not screenshots](https://raw.githubusercontent.com/FU-max-boop/agentrunproof/main/docs/assets/social-preview.png)

**Catch OpenAI Agents SDK `Runner` regressions without an API key.**

AgentRunProof runs deterministic scenarios against the real `Runner`, compares observable behavior
across `run` and `run_streamed`, and writes content-addressed JSON records for stream, session,
tool-linkage, and `RunState` resume invariants. A failing record carries the normalized
counterexample observations.

AgentRunProof v0.2 declares the `openai-agents>=0.20,<0.22` compatibility window on Python
3.10–3.14. Its packaged CI matrix verifies the exact 0.20.0 and 0.21.0 release baselines.
Built-in scenarios make no model API call and require no API key.

> AgentRunProof-backed reports are referenced by two merged maintainer fixes,
> [#4413](https://github.com/openai/openai-agents-python/pull/4413) and
> [#4414](https://github.com/openai/openai-agents-python/pull/4414). This is upstream diagnostic
> impact—not OpenAI adoption, dependency, or endorsement.

Read the five-minute
[RunState case study](https://github.com/FU-max-boop/agentrunproof/blob/main/docs/case-study-runstate.md)
for the released failure, the recursive follow-up, and the exact before/after evidence chain.

## 30-second local check

```bash
python -m pip install "agentrunproof==0.2.0"
agentrunproof probe basic-tool-session-parity --certificate proof.json
agentrunproof check-certificate proof.json
```

Expected output:

```text
PASS basic-tool-session-parity
PASS execution_outcome: OK
PASS stream_parity: OK
PASS tool_linkage: OK
PASS exactly_once: OK
PASS model_script_consumed: OK
certificate_id: sha256:...
written: proof.json
VALID sha256:... PASS
```

Exit `0` means PASS, `1` means an observed invariant violation, and `2` means invalid or
unverifiable evidence. See the
[provider-free real Runner example](https://github.com/FU-max-boop/agentrunproof/blob/main/examples/provider_free_tool_demo.py)
and the
[OpenAI Agents integration guide](https://github.com/FU-max-boop/agentrunproof/blob/main/docs/openai-agents.md).

## Where it fits

Use the SDK's public `agents.testing.ScriptedModel` with `pytest` for a focused deterministic
application or SDK test. AgentRunProof delegates to `ScriptedModel` on SDK 0.21 and adds reusable
scenario orchestration, automatic `run`/`run_streamed` comparison, multi-phase `RunState` checks,
cross-version evidence, and content-addressed records.

| You need to… | Start with |
| --- | --- |
| Script model responses and assert one application behavior | `agents.testing.ScriptedModel` + `pytest` |
| Compare the same contract across runner modes or SDK versions | AgentRunProof |
| Check approval/rejection and JSON-restored `RunState` flows | AgentRunProof |
| Share a normalized record that can be checked without a provider call | AgentRunProof |
| Evaluate model-output quality | An eval framework, not AgentRunProof |

## What AgentRunProof checks

- declared completion, interruption, or Runner-exception outcomes for every scenario phase;
Expand Down Expand Up @@ -134,6 +198,18 @@ isolation defect. The next adoption target is reuse of the recursive regression
optional CI check, or a documentation reference—not a default SDK dependency. A community-tool
entry was [proposed on the official v0.21 testing-guide PR](https://github.com/openai/openai-agents-python/pull/4381#issuecomment-5293600461). The maintainer [kept that guide limited to SDK-maintained APIs](https://github.com/openai/openai-agents-python/pull/4381#issuecomment-5293704972) while explicitly welcoming future reproducible findings backed by the tool. AgentRunProof therefore remains an external project rather than an official SDK listing or dependency.

## Contribute a runtime contract

Found a public-API `Runner` inconsistency?
[Open a scenario request](https://github.com/FU-max-boop/agentrunproof/issues/new/choose) with the
exact SDK version and a minimal reproducer. Want to make it permanent? See the
[contribution guide](https://github.com/FU-max-boop/agentrunproof/blob/main/CONTRIBUTING.md) and add
the smallest failing scenario. For usage questions and early contract ideas, use
[Discussions](https://github.com/FU-max-boop/agentrunproof/discussions).

If AgentRunProof belongs in your regression toolbox, star the repository so other SDK maintainers
can find it.

## License

MIT
26 changes: 26 additions & 0 deletions SECURITY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Security policy

## Supported versions

Security fixes are made against the latest released AgentRunProof minor version.
Older evidence remains immutable, but users should reproduce a finding with the
latest release before reporting it when possible.

## Reporting a vulnerability

Use GitHub's private vulnerability-reporting form for vulnerabilities in
AgentRunProof itself. Please do not open a public issue for a credential leak,
path escape, certificate-verification bypass, unsafe evidence publication, or
other issue that could put users at risk.

Include the affected version, a minimal reproduction, impact, and any proposed
mitigation. Remove API keys, production traces, private certificates, and user
data from the report.

SDK vulnerabilities that do not depend on AgentRunProof should be reported to
the OpenAI Agents SDK maintainers through their
[documented security channel](https://github.com/openai/openai-agents-python/security/policy).

AgentRunProof executes user-provided Python scenarios and is not a sandbox. A
scenario intentionally running arbitrary code is not itself a vulnerability;
an undocumented privilege boundary bypass or unsafe default may be.
Binary file added docs/assets/social-preview.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading