Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,10 @@

Focused checks and exact summaries:

- Candidate commit/tree (and any uncommitted changes covered):
- Independent reviewer and outcome:
- Unresolved findings or unrun checks:

## Safety and release

- [ ] No worker-reported command or test was treated as evidence.
Expand Down
36 changes: 26 additions & 10 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,25 @@ permissions:
contents: read

jobs:
preflight:
runs-on: macos-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd
with:
fetch-depth: 0
persist-credentials: false
ref: ${{ github.event_name == 'workflow_dispatch' && inputs.head_sha || github.event.pull_request.head.sha }}

- name: committed diff hygiene
env:
AGY_WORKER_CI_EVENT_NAME: ${{ github.event_name == 'workflow_dispatch' && 'push' || github.event_name }}
AGY_WORKER_CI_BASE_SHA: ${{ github.event_name == 'workflow_dispatch' && inputs.base_sha || github.event.pull_request.base.sha }}
AGY_WORKER_CI_HEAD_SHA: ${{ github.event_name == 'workflow_dispatch' && inputs.head_sha || github.event.pull_request.head.sha }}
run: ./scripts/ci-diff-check.sh

shard:
needs: [preflight]
runs-on: macos-latest
timeout-minutes: 60
strategy:
Expand All @@ -39,13 +57,6 @@ jobs:
persist-credentials: false
ref: ${{ github.event_name == 'workflow_dispatch' && inputs.head_sha || github.event.pull_request.head.sha }}

- name: committed diff hygiene
env:
AGY_WORKER_CI_EVENT_NAME: ${{ github.event_name == 'workflow_dispatch' && 'push' || github.event_name }}
AGY_WORKER_CI_BASE_SHA: ${{ github.event_name == 'workflow_dispatch' && inputs.base_sha || github.event.pull_request.base.sha }}
AGY_WORKER_CI_HEAD_SHA: ${{ github.event_name == 'workflow_dispatch' && inputs.head_sha || github.event.pull_request.head.sha }}
run: ./scripts/ci-diff-check.sh

- name: shard offline suite
env:
AGY_WORKER_CI_HEAD_SHA: ${{ github.event_name == 'workflow_dispatch' && inputs.head_sha || github.event.pull_request.head.sha }}
Expand All @@ -66,7 +77,7 @@ jobs:
test:
name: test
if: always()
needs: [shard]
needs: [preflight, shard]
runs-on: macos-latest
timeout-minutes: 10
steps:
Expand All @@ -85,10 +96,15 @@ jobs:
- name: verify aggregate shard receipts
env:
AGY_WORKER_CI_HEAD_SHA: ${{ github.event_name == 'workflow_dispatch' && inputs.head_sha || github.event.pull_request.head.sha }}
PRODUCER_RESULT: ${{ needs.shard.result }}
PREFLIGHT_RESULT: ${{ needs.preflight.result }}
SHARD_RESULT: ${{ needs.shard.result }}
RECEIPTS_DIR: ${{ runner.temp }}/downloaded-shard-receipts
run: |
if [[ "${PREFLIGHT_RESULT}" != success ]]; then
printf '%s\n' 'CI preflight did not succeed' >&2
exit 1
fi
/usr/bin/python3 -I -S -B scripts/ci_sharding.py verify-aggregate \
--receipts-dir "${RECEIPTS_DIR}" \
--expected-head "${AGY_WORKER_CI_HEAD_SHA}" \
--producer-result "${PRODUCER_RESULT}"
--producer-result "${SHARD_RESULT}"
72 changes: 25 additions & 47 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ requesting review:

It is fail-fast, requires no network or provider call, does not intentionally inspect
account-HOME contents, externalizes temporary bytecode, and runs the static checks plus
all forty offline stages. Ambient local tools may still consult ordinary user
all registered offline stages. Ambient local tools may still consult ordinary user
configuration. In GitHub Actions, the suite is partitioned across four fail-closed shards
(`dispatcher`, `dispatcher-remediation`, `other-a`, `other-b`) and validated by the required aggregate
`test` check; lower CI wall time from parallelization does not mean lower total compute, provider usage,
Expand All @@ -46,59 +46,37 @@ observational monotonic wall time in an owner-private mode-0600 JSON report with
recording commands, logs, environment variables, or timestamps. For a quota-unavailable
private fork this is evidence to attach to review, not a replacement for the protected
GitHub `test` check; manually dispatch the exact committed range after Actions becomes
available and before publication. The expanded command inventory is retained below
for targeted diagnosis. A focused pass accelerates iteration but does not replace the
stable-candidate full gate:
available and before publication. List the current commands directly from the
canonical registry:

```bash
./tests/test-qa-gate.sh
./tests/test-evidence-receipt.sh
./tests/test-evidence-report.sh
/usr/bin/python3 -I -S -B tests/test-benchmark.py
/usr/bin/python3 -I -S -B tests/test-swebench-workflow-study.py
/usr/bin/python3 -I -S -B tests/test-job-lifecycle.py
./tests/test-agy-worker.sh
/usr/bin/python3 -I -S -B tests/test-agy-worker-remediation.py
./tests/test-update.sh
/usr/bin/python3 -I -S -B tests/test-adoption-measurement.py
/usr/bin/python3 -I -S -B tests/test-update-notifier.py
/usr/bin/python3 -I -S -B tests/test-version-attestation-runner.py
/usr/bin/python3 -I -S -B tests/test-version-bootstrap-runner.py
/usr/bin/python3 -I -S -B tests/test-version-initial-bootstrap-runner.py
/usr/bin/python3 -I -S -B tests/test-version-attestation-harness.py
/usr/bin/python3 -I -S -B tests/test-models-attestation-runner.py
/usr/bin/python3 -I -S -B tests/test-models-capture-runner.py
/usr/bin/python3 -I -S -B tests/test-models-capture-profile.py
/usr/bin/python3 -I -S -B tests/test-models-capture-1-1-22-version-evidence.py
/usr/bin/python3 -I -S -B tests/test-models-capture-1-1-22-profile.py
/usr/bin/python3 -I -S -B tests/test-models-capture-1-1-22-runner.py
/usr/bin/python3 -I -S -B tests/test-models-capture-1-1-22-reprofile.py
/usr/bin/python3 -I -S -B tests/test-models-capture-1-1-22-classifier.py
/usr/bin/python3 -I -S -B tests/test-agy-1-1-22-activation.py
./tests/test-reporting.sh
/usr/bin/python3 -I -S -B tests/test-feedback-triage.py
/usr/bin/python3 -I -S -B tests/test-model-intelligence.py
/usr/bin/python3 -I -S -B tests/test-model-evidence-campaign.py
/usr/bin/python3 -I -S -B tests/test-codex-usage-report.py
/usr/bin/python3 -I -S -B tests/test-delegation-policy.py
/usr/bin/python3 -I -S -B tests/test-workflow.py
./tests/test-packaging.sh
./tests/test-doctor.sh
/usr/bin/python3 -I -S -B tests/test-conformance.py
./tests/test-proof-demo.sh
bash -n ./*.sh conformance/*.sh scripts/*.sh tests/*.sh skills/*/scripts/*.sh skills/*/runtime/*.sh
(
AGY_WORKER_PYCACHE="$(mktemp -d -t agyworker-pycache.XXXXXX)" || exit 1
trap 'rm -rf -- "$AGY_WORKER_PYCACHE"' EXIT
PYTHONPYCACHEPREFIX="$AGY_WORKER_PYCACHE" \
python3 -m py_compile conformance/v1/*.py scripts/*.py skills/*/runtime/scripts/*.py
)
git diff --check
/usr/bin/python3 -I -S -B scripts/ci_stages.py --list
```

The listing shows stage ID, shard, and registered command without running tests.
Suite commands can be run directly; syntax stages use runner-provided environment.
Use the matching repository-map row to choose the owning suite. For remediation,
select the natural group that owns the change:

```bash
/usr/bin/python3 -I -S -B tests/test-agy-worker-remediation.py --group core
/usr/bin/python3 -I -S -B tests/test-agy-worker-remediation.py --group runtime
/usr/bin/python3 -I -S -B tests/test-agy-worker-remediation.py --group recovery
```

Omitting `--group` runs the full remediation inventory. A focused pass accelerates
iteration but does not replace the stable-candidate full gate.

Report exact summaries and any checks you could not run. Passing tests do not replace
human diff review or justify unrelated cleanup.

Classify a failed check before retrying: a reproducible code failure needs a fix,
a timing-sensitive test needs a deterministic reproduction, and an environment or
service failure needs its prerequisite restored. Do not increase timeouts or change
models as a substitute for diagnosing the failure. Reuse passing evidence while the
candidate bytes and relevant environment remain unchanged; after an edit, rerun the
owning checks and apply the stable-candidate full-gate rule above.

When a release is also claimed to be installed or collecting local measurements,
verify those machine states separately after publication. A clean tagged checkout is
not proof that the global Codex skill was recopied, that the LaunchAgent snapshot was
Expand Down
13 changes: 7 additions & 6 deletions docs/OPERATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,13 @@ For contribution and release checks, start with
## Run the repository checks

The required GitHub `test` job is a fail-closed aggregate over four shards:
`dispatcher`, `dispatcher-remediation`, `other-a`, and `other-b`. Each shard checks
out the exact pull-request head (or validated exact manual-dispatch head), enforces
committed-range diff hygiene, runs its registered stage subset, and publishes a
`dispatcher`, `dispatcher-remediation`, `other-a`, and `other-b`. A single preflight
checks committed-range diff hygiene before the shards start.
Each job checks out the exact pull-request head (or validated exact manual-dispatch
head). Each shard runs its registered stage subset and publishes a
mode-`0600` receipt bound to the head and canonical inventory. The aggregate succeeds
only when all four producers succeeded, every receipt matches the expected head and
inventory, and every canonical stage ran exactly once.
only when preflight and all four producers succeeded, every receipt matches the
expected head and inventory, and every canonical stage ran exactly once.

GitHub retains the uploaded privacy-safe receipt artifact for one day. The receipt
contains no paths, commands, environment values, logs, or credentials. It is workflow
Expand All @@ -32,7 +33,7 @@ same canonical fail-fast suite locally from the repository root:
./scripts/ci-offline.sh
```

The forty offline stages need no agy provider call, network access, API key, or
The canonical offline stages need no agy provider call, network access, API key, or
GitHub login. Ambient local tools may still consult their ordinary user
configuration. Keep the exact summary with the commit, tree, and `git diff --check`
evidence. On a clean tracked and untracked worktree, an optional timing report records
Expand Down
Loading
Loading