Skip to content

Hold the run lock through clean and reset with or without a store - #1716

Merged
Muizzkolapo merged 1 commit into
mainfrom
fix/clean-holds-lock-always
Nov 8, 2026
Merged

Muizzkolapo merged 1 commit into
mainfrom
fix/clean-holds-lock-always

Conversation

@Muizzkolapo

Copy link
Copy Markdown
Owner

Follow-up to #1711 (issue #1699), which merged at its earlier head. A blind review of that head found a defect that is now on main; this PR fixes it. It also adds the missing "held while it works" test for #1713.

Problem

#1711 made clean and reset_workflow.py --full take the run lock, but only when store/ already existed, decided once before the prompt. Before #1711, agac clean --all deleted agent_io/store/ while a run was in progress. With #1696 the run lock lives in that folder, so the lock file went with it and a second agac run then passed the lock check. clean also deleted the live database and the run's staging inputs. reset_workflow.py --full deleted store/ the same way.

Change

  • Cleaner: once it has found something to remove, it takes the workflow's run lock in every mode (default, --target, --all), before it asks or deletes anything, and holds it until it ends. It takes it with or without a store folder: the lock creates the folder. It exits with RunInProgressError while a run or retry holds the lock, and that error is no longer wrapped as "Cleaning failed".
  • The lock file is never deleted. AgentManager.clean_directory takes keep=: the store is emptied around the held lock file, so clean --all leaves store/ holding only the empty <workflow>.db.lock. The next run locks that same file. Deleting it would let a run lock a new file beside the one still held, and Windows refuses to delete a file held open.
  • Whether there was a store is read before the lock creates the folder. A store holding only the lock file counts as none, so clean does not open it for the batch registry (which would create a database).
  • reset_workflow.py --full: the same rules: lock taken in every case and held through the wipe (agac's hold_run_lock where the framework imports, else the same flock from plain python3), store cleared around the lock file (clear_store), and only a store that was there before the lock is cleared. Without fcntl and without the framework (Windows, plain python3) it warns and goes on, as before. The soft reset is unchanged.
  • Docs: cli/run.md, cli/utilities.md, data-io/output-format.md, the script docstring, SKILL.md, skills/ARCHITECTURE.md and the realtime manifest say that clean refuses during a run once it has something to remove, and that the lock file stays.
  • Test for Take the run lock for agac retry --filter #1713 (retry --filter): test_retry_filter_holds_the_run_lock.py asserts the lock is held for all of _filter and not taken by --dry-run. A lock taken and released at once passed every earlier test; it fails this one.

Review history

The version of #1711 that merged took the lock only when store/ already existed, decided once before the prompt. A blind review reproduced the failure: a run that started while clean waited at its prompt created the store, took the lock, and then lost its staging and target. Three wrong versions also passed its tests: lock taken and released before the delete, lock taken after the prompt, and the script releasing the lock before the wipe. This PR fixes the first and adds tests that fail on each of the others.

Verification

  • gate.sh red recorded tests/unit/cli/test_clean_and_reset_hold_the_run_lock.py failing on the merged Refuse agac clean and reset_workflow --full during a run #1711 code (13 failed, 3 controls passed): the lock is not held while clean asks or deletes with no store, clean --all and reset --full delete the lock file, and reset --full deletes its last path with no lock file left.
  • gate.sh green: RED target passes, ruff format --check, ruff check, mypy agent_actions, full pytest 17633 passed, 37 skipped.
  • test_a_refused_clean_or_reset_reclaims_nothing.py: a clean --all and a reset --full refused for a run in progress release no batch record that the store names; the same commands with no run release it (live control).
  • Mutants (mutate.sh), each fails at least one test: clean lock released before the delete; clean with no lock; keep ignored in clean_directory; the script's with held_run_lock released at once; the script with no lock; clear_store ignoring the lock file; retry --filter lock released before _filter. A reviewer also ran: lock after the prompt; no lock when store/ is missing; script fallback with no flock; dry run taking the lock: all fail.
  • Existing tests that asserted store/ is gone after clean --all or reset --full now assert it holds exactly the lock file (== ["wf.db.lock"]), which is stricter. test_clean_all_leaves_no_store_behind_when_there_was_none keeps its intent: no database is made.
  • Blind review, HIGH tier (risk.sh, 12 files): three lenses YES. Their items (a lock-only store counted as a store; the script clearing a store that appeared after its check; docs claiming every mode refuses; four docs still saying the store is removed; no test that the batch reclaim runs under the lock) are fixed in this diff, and a fourth reviewer checked that delta: YES. The edits to existing tests replace "store/ is gone" with "store/ holds exactly the lock file", which is stricter.
  • gate.sh smoke (risk.sh: smoke_required=yes): scan-project.sh clean; ql_flow_diagram online ran 33 of 33 actions, store verified with 70 records and no failed or exhausted disposition.

clean took the run lock only when store/ already existed, and decided
that once, before its confirmation prompt. With no store, a run that
started while the prompt waited created the store, took the lock, and
then lost its staging and target to the clean. reset_workflow.py --full
had the same check-then-act gap. Both also deleted the store with the
held lock file in it, which lets a run lock a new file beside it, and
which Windows refuses because the file is open.

clean and reset --full now take the lock in every case, before asking
or deleting, and hold it to the end. They empty the store around the
lock file instead of deleting it (clean_directory keep=, clear_store),
so the next run locks the same file. Whether there was a store is read
before the lock creates the folder, and a store holding only the lock
file counts as none, so it is not opened for the batch registry.

Tests now read the lock state while clean asks and while each command
deletes, assert the lock file is kept, assert a refused clean or reset
reclaims no batch record, and assert retry --filter holds the lock for
all its work and its dry run takes none. Each fails on a version that
takes the lock and releases it early, or takes none.
@Muizzkolapo
Muizzkolapo merged commit 136f145 into main Nov 8, 2026
5 checks passed
@github-actions github-actions Bot locked and limited conversation to collaborators Nov 8, 2026
@Muizzkolapo
Muizzkolapo deleted the fix/clean-holds-lock-always branch November 8, 2026 19:30
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant