Repository navigation
Chore(deps): Bump actions/setup-python from 5 to 6 - #1
Closed
dependabot[bot] wants to merge 53 commits into
Closed
dependabot[bot] wants to merge 53 commits into
dependabot[bot] wants to merge 53 commits into
Conversation
Enh: Set ruff line-length to 120 and reformat
Gold validation now runs through the regular submission path: build "false" gold submissions from RevEngBench/output/github-dump/<iid>/build.zip (with build.sh renamed to compile.sh) via RevEngBench/scripts/make_gold_submissions.py and feed them to `programbench eval <gold_run_dir>`. Drops: - BUILD_SH and DEFAULT_GOLD_EVAL_DIR constants - Evaluator.task_dir / repository / commit ctor params - Gold branch in _compile_executable() (git clone + build.sh) - is_gold_mode plumbing through _evaluate_instance / run_eval_batch - 'gold' source handling and CLI examples Renames the in-container executable stash path to drop the "gold" prefix. Adds a post-port banner at the top of ai-docs/plans/initial_eval_port.md.
…allelism `docker run -d` was hardcoded to 60s and `docker cp` to 120s, which under --workers 4 (or larger images that need pulling) routinely tripped TimeoutExpired and failed entire instances before the container could even start. Bump both defaults to 300s and expose them via PROGRAMBENCH_DOCKER_RUN_TIMEOUT / PROGRAMBENCH_DOCKER_CP_TIMEOUT.
Python's zipfile.extractall() drops the executable bit even when it's recorded in the entry's external_attr. For submissions whose compile.sh calls a tracked autotools-style ./configure (sqlite, ffmpeg, htop, tinycc, dropbear, php-src, lnav, ctags, the_silver_searcher, chafa, tig, entr), this caused all of them to fail compile_failed with "./configure: Permission denied". Switch from extractall() to a per-entry loop that os.chmod()s each file to the mode stored in the zip header. The chmod +x ./compile.sh in the compile step stays as a safety net for submissions that don't store modes at all.
Three projects (jqlang/jq, lfos/calcurse, paradigmxyz/solar) fail to compile because their build scripts assume a working git tree: - jq: build.sh runs `git submodule update --init --recursive` for the vendored oniguruma - calcurse: build.sh -> autogen.sh -> autopoint --force - solar: cargo build invokes vergen, which calls `git rev-parse --is-inside-work-tree` The legacy RevEngBench gold pipeline got this for free via `git clone <upstream>`. The new submission-zip path delivers only the source tree, so the workspace had no .git and these builds hard-failed with "fatal: not a git repository". After extraction and before compile.sh, run a guarded `git init && git add -A && git commit --allow-empty -m gold` (with commit.gpgsign=false to bypass any user-global signing requirement). The `if [ ! -d .git ]` guard is a no-op for submissions that already ship their own .git, so this is safe for every codepath, not only gold. Lives here rather than in make_gold_submissions.py so it applies uniformly to every submission (model-generated submissions can hit the same vergen/autopoint pitfall) and so we don't have to regenerate existing submission.zip files when the seed logic changes.
Output like 'Branch 79b69dd3fd98: 158/158 expected tests missing' is useless when 200 instances run in parallel — there's no way to tell which instance the branch belongs to. Thread instance_id through Evaluator and into _process_branch_xml so warnings now read '[abishekvashok__cmatrix.5c082c6] branch 79b69dd3fd98: ...'. The EvalStepError continuation warning gets the same prefix. Falls back to the prior 'Branch <hash>:' form when instance_id is empty (e.g. the from_existing reprocessing path).
A single <testcase> with multiple result children of the same type (e.g. pytest emitting `pytest.internal` with two identical <error>s on internal failures) was flagged as `system_error`. Treat it as one result of that type instead. Mixed-kind duplicates still fall through to `system_error`.
Splits the previously-conflated output_dir into a source_dir (where submission.zip is read) and target_dir (where <iid>.eval.json is written). With --output <DIR>, each source S writes to <DIR>/<S.name>/<iid>/, mirroring the input layout so multi-source runs stay disambiguated.
After compile finishes, commit the compile container to a temporary image and run each test branch in its own fresh container off that image. This isolates branch test execution and lets branches run in parallel within an instance via the new -b/--branch-workers flag, alongside the existing -w/--workers flag for cross-instance parallelism. Add --docker-cpus to set the CPU allowance per docker container; the same value is exported as PYTEST_XDIST_AUTO_NUM_WORKERS inside the container so pytest-xdist `-n auto` matches the allotted cores.
`_process_branch_xml` now takes `ignored_tests` and `branch_ignored` so that tests previously marked ignored, and branches with `ignored: true` in `tests.json`, are excluded from the missing-from-JUnit and not-in-tests.json warnings. The Evaluator threads both through, and the batch / summarize-only call sites pass them via the new `get_ignored_branches` helper.
Four gold instances (monolith, doxygen, atlas, lnav) compiled cleanly but timed out at exactly 20.022s on the copy step, dropping ~4400 tests into not_run. Drop the unused `ls && ...` prelude (legacy debug aid) and bump the timeout to 120s, in line with the surrounding compile/test budgets.
Per-branch containers boot fresh from the committed post-compile image, so the wipe step before each branch had nothing stale to clear — but it did delete build artefacts that some test suites rely on. tukaani-project/xz ships tests that exec /workspace/build/xz; osgeo/proj's coverage tests hit /workspace/build_cov/bin/projinfo; both fell over with FileNotFoundError despite producing the same executable_hash as the legacy gold pipeline. Drop the wipe; copy_in and _restore_executable already handle layering.
Each branch runs in a fresh container from the committed image, so consuming the stash with mv is safe and avoids multi-GB byte copies on overlayfs (treemd's debug binary blew past the 120s cp timeout).
Two related fixes for the 8 branches across 8 instances that came back with `results_read_failed` / `restore_executable_failed` and empty error_details, dropping 1000+ tests into not_run: 1. run_tests timeout: 2400s -> 3600s. Empty error_details came from a subprocess.TimeoutExpired in container.execute(): r["output"] is "" on timeout, so EvalStepError(error_details="") gives no signal at all. 40 minutes was tight for the largest test suites. 2. _run_test_branch wrote per-step entries to a local_log that was only merged into result.log on the success path (eval.py:514-518). On EvalStepError the local_log was discarded, so the timed-out run_tests entry never made it to disk -- impossible to tell from the eval JSON whether 2400s was hit, what stderr looked like, or which step failed. Pass log_buf in by reference so steps land in it as they run, and merge it on the error path too.
Adds findings 7-10 from the post-2026-04-30 PB-vs-RB sweep: - 20s timeout on `mv` for big binaries (8 instances + 6 branch errors) - google__brotli silently absent (export-ignore strips eval/) - synthetic .git non-determinism breaks executable_hash reproducibility - pytest-xdist worker crashes (run-time flake, no code fix) Drops the gromacs/copy_executable_failed entry (gromacs now succeeds in this run) and the "synthetic .git produces different binaries" residual (now fixed by the deterministic seed).
… fixes Reruns only the 15 instances expected to move after the 2026-04-30 sweep: 8 copy_executable timeouts, 6 branch-level timeouts, and brotli (which previously had no eval json at all). Documents the brotli preflight (repopulate + rebuild gold submission) and leaves a hook for opt-in exec-hash drift verification across the determinism fix.
When pytest-xdist loses a worker (OOM, signal, etc.), the testcases assigned to that worker silently disappear from the JUnit XML — they show up as <error message="failed on setup with "worker 'gwN' crashed ...">. On the chamber/stgit-class branches that's tens to hundreds of tests dropped per run, and which side suffers the loss is essentially random. Add `count_worker_crashes(raw_xml)` that parses the JUnit XML structurally (no stdout grep — xdist output mixes across workers, so structured payload is the only reliable signal) and counts testcases whose error/failure message matches the xdist crash signature. Plumb a `--branch-retries N` CLI flag through run_eval_batch → _evaluate_instance → Evaluator. _evaluate_branch now retries the branch in a fresh container off the same post-compile image, keeping the attempt with the fewest worker crashes. Default 0 preserves prior behaviour.
Per-branch xdist crash recovery is on by default — based on the post- 2026-04-30 audit, 46 PB branches (15.6% of instances) lost tests to worker crashes in a single run. Three retries makes the loss rate effectively zero in the cases we've seen, at the cost of <=3 extra container starts per affected branch. Pass 0 to disable.
New --pytest-addopts flag (also reads PROGRAMBENCH_PYTEST_ADDOPTS env var)
sets PYTEST_ADDOPTS inside every test-branch container. pytest applies
PYTEST_ADDOPTS to every invocation, so this lets us inject xdist tunables
without modifying any branch's eval/run.sh.
Common uses:
- '--max-worker-restart=4' pytest-xdist replaces crashed workers, so
tests assigned to a dead worker can be
re-queued onto the replacement instead of
all dropping into the JUnit XML as
'worker gwN crashed'.
- '--reruns=2' if pytest-rerunfailures is installed in the
base image, reruns flaky individual tests.
Threaded through Evaluator -> _new_env -> ContainerEnvironment(env=...).
Defaults to empty so single-instance behaviour is unchanged unless
explicitly enabled.
Replace the user-facing --pytest-addopts flag with built-in defaults that no caller has to know about: - Every test-branch container starts with PYTEST_ADDOPTS=--max-worker-restart=4, so xdist can replace crashed workers and re-queue their tests instead of dropping them as "worker gwN crashed" entries. - During compile (immediately after hash_executable), best-effort pip3-install pytest-rerunfailures into the workspace. Since the post- compile container is committed to an image and per-branch containers inherit it, this is paid once per instance, not once per branch. If install fails (no pip3, no network, ...) we accept it and fall back to --max-worker-restart-only. - When rerunfailures was installed, _run_test_branch prepends --reruns=2 --reruns-delay=1 to PYTEST_ADDOPTS at exec time, retrying flaky individual tests inside a single run. The augmentation is per-exec rather than per-container so we don't need to differentiate pre- and post-install env in the committed image. Combined with --branch-retries (which retries the whole branch when crashes still bleed through), this is a three-layer recovery: xdist replaces workers → rerunfailures replays failures → eval re-runs the branch.
…-by-side output Adds the 31 instances flagged by the worker-crash audit (now benefiting from default --max-worker-restart=4, pytest-rerunfailures, and --branch-retries=3) and routes results to a separate output dir ($OUTPUT_DIR, default ~/gold-eval-after-fixes) so the original ~/gold/ stays intact for before/after diffs.
After a worker crash, the obvious retry strategy (re-run the same branch with the same parallelism) hits the same OOM/contention pattern: the audit log shows branches consuming all 3 retry attempts and still losing one worker each time. When a retry is triggered by detected worker crashes, set PYTEST_XDIST_AUTO_NUM_WORKERS=1 in the new container's env so any `pytest -n auto` in the branch's run.sh resolves to one worker. Slower per branch, but unaffected by the contention that caused the original crash. EvalStepError-driven retries (network, timeout, ...) keep the parallel mode since serial mode wouldn't help those.
After every branch that triggered at least one retry, log either:
- INFO ... recovered after N retries — crashes/tests by attempt: 5/523, 0/568
(final attempt has 0 crashes; we got back to a clean run)
- WARN ... did not fully recover after N retries — kept best-of-N+1
with 3 crashes (crashes by attempt: [5, 4, 3, 3]; tests by
attempt: [523, 561, 564, 568], min=523 max=568 spread=45)
(still crashy after the budget; the spread shows how
non-deterministic the run is, helping decide whether more
retries would help)
Also annotates each in-flight retry log line with the testcase count
on the failing attempt so it's clear how big the partial result was.
A small count_testcases(raw_xml) helper computes the total <testcase>
count from JUnit XML (returns 0 on parse error / empty input).
For deterministic test crashes (e.g. oha's test_longer_duration which runs the binary for 15s against a 5s test timeout), serial-pytest mode eliminates crashes but loses tests entirely from the JUnit XML. Concretely on hatoo__oha: parallel: 1 crash / 131 testcases (130 useful) serial: 0 crashes / 83 testcases ( 83 useful) The previous "fewest crashes" criterion picked serial → 130 → 83 useful tests, dropping 47 tests into not_run downstream. Switch to "most useful tests" (n_tests - crashes), which picks the parallel run when serial drops too many tests, and still picks the serial run when it actually delivers more usable results (the chamber- class contention case). The richer log line already includes the full crash/total/useful sequence per attempt, so the picker's choice is visible in the warning.
Every generated test branch's eval/run.sh ships with `--timeout-method=thread`. When a test exceeds its timeout, pytest-timeout in thread mode calls os._exit(1) on the whole worker process. xdist sees a dead worker and reports it as "worker 'gwN' crashed", and the queue of tests that worker had been pre-assigned vanishes. `signal` method delivers SIGALRM instead. pytest catches it cleanly, runs the test's normal cleanup, and the test gets a proper subprocess.TimeoutExpired failure entry. No worker crash, no queue loss, identical results in parallel and serial mode. Verified on hatoo__oha test_longer_duration: thread + parallel: 131 testcases, 1 worker crash (130 useful) thread + serial: 119–127 testcases, 1 crash (118–127 useful) signal + parallel: 129 testcases, 0 crashes, 1 fail (129 useful) signal + serial: 129 testcases, 0 crashes, 1 fail (129 useful) The patch is a one-line sed against the run.sh inside the per-branch container, applied between the test-tree copy_in and the run.sh exec. Files that don't have the literal string are unaffected.
…tical If two consecutive attempts produce the exact same (crashes, n_tests), the failure is deterministic (specific test, specific binary behaviour) rather than contention-driven, and further retries just burn ~100s each. Break out of the retry loop early; the kept attempt is unchanged (we still picked the best-by-useful-tests so far). Pairs nicely with the signal-method patch: most stubborn branches now succeed on attempt 0 anyway, but for any future branch that resists the signal method (e.g. binary segfaults rather than timing out), the serial retry budget caps at 2 attempts instead of 4. Log message gets a "(stopped early — last 2 attempts identical)" annotation when triggered.
Empirical analysis of three gold runs found no clear evidence that the per-branch retry loop recovered tests beyond what the first attempt's partial JUnit XML already produced. Most retries either reproduced the same pytest summary verbatim (noise) or ran on a fresh container with empty output. After the --timeout-method=signal patch the underlying worker-crash class is largely gone, so retries mostly burn budget on stuck/slow branches without upside. Keep the flag (still passable as --branch-retries N) but stop paying for it by default.
submission.zip went through populate's zipfile.extract path, which writes symlinks as regular files containing the link-target string. That broke deterministic gold parity on instances whose tests exercise symlink/sparse semantics (pls-rs__pls, byron__dua-cli, bootandy__dust, parts of nachoparker__dutree) — same executable_hash as RB legacy, but ~25-30 deterministic test failures per instance. Switch the whole submission pipeline to tar.gz: tarfile.extractall preserves both symlinks and Unix modes natively, so the manual mode-replay loop and the explicit zipfile import disappear. Rename the API surface submission_zip → submission_archive to match. eval_batch now discovers submission.tar.gz; the typer help text follows suit. Pairs with RevEngBench's make_gold_submissions which now writes submission.tar.gz directly.
Same audit class as the wipe_workspace 20s→300s bump: the cat of eval/results.xml runs over the docker exec channel and on some branches (rustowl frozen-scaffold-testgen-20260410_094012, quinn / kiro-editor / php-src frozen-scaffold-testgen-* branches in gold-eval-9) consistently hits exactly 60s and aborts with rc=-1. The XML payload is small in all cases — RB legacy reads the same files fine — so the timer fires while the docker exec is still spinning up, not while cat is producing data. 60s was too tight; 300s matches the budgets we set on copy/hash/restore_executable.
Prints scores and warnings for a programbench-evaluated run directory. Filters out branches/tests now marked ignored in tests.json before scoring, so stale errors from since-removed branches don't show up.
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 5 to 6. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@v5...v6) --- updated-dependencies: - dependency-name: actions/setup-python dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
dependabot
Bot
added
dependencies
meta-cla
Bot
added
the
CLA Signed
Contributor
Author
|
OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, let me know by commenting If you change your mind, just re-open this PR and I'll resolve any conflicts on it. |
dependabot
Bot
deleted the
dependabot/github_actions/actions/setup-python-6
branch
May 5, 2026 14:16
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bumps actions/setup-python from 5 to 6.
Release notes
Sourced from actions/setup-python's releases.
... (truncated)
Commits
a309ff8Bump urllib3 from 2.6.0 to 2.6.3 in /tests/data (#1264)bfe8cc5Upgrade@actionsdependencies to Node 24 compatible versions (#1259)4f41a90Bump urllib3 from 2.5.0 to 2.6.0 in /tests/data (#1253)83679a8Bump@types/nodefrom 24.1.0 to 24.9.1 and update macos-13 to macos-15-intel ...bfc4944Bump prettier from 3.5.3 to 3.6.2 (#1234)97aeb3eBump requests from 2.32.2 to 2.32.4 in /tests/data (#1130)443da59Bump actions/publish-action from 0.3.0 to 0.4.0 & Documentation update for pi...cfd55cagraalpy: add graalpy early-access and windows builds (#880)bba65e5Bump typescript from 5.4.2 to 5.9.3 and update docs/advanced-usage.md (#1094)18566f8Improve wording and "fix example" (remove 3.13) on testing against pre-releas...Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting
@dependabot rebase.Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)