Skip to content

Answer a WAIT that looks too soon with the exit, not CONFLICT - #92

Closed
pcarrier wants to merge 2 commits into
yas-run:mainfrom
indent-com:test-waits
Closed

pcarrier wants to merge 2 commits into
yas-run:mainfrom
indent-com:test-waits

Conversation

@pcarrier

@pcarrier pcarrier commented Sep 30, 2026 •

Copy link
Copy Markdown
Collaborator

A WAIT right after a KILL of a detached process could answer CONFLICT instead of the exit. This PR fixes that server race, and one flaky process test.

The race

A WAIT on a process its session holds no route to looks at it with a temporary WATCH (wait_route → watch_process → watch_native). The native side refuses that WATCH with CONFLICT in two cases:

  • the process's exit is on its way to its watchers (terminal_queued: the record becomes final only once they all have it);
  • this session's own look is still bound: a concurrent CONTROL's temporary one, or a route that an eviction failed and has yet to detach (dispatch_outbound fails the route, then sends the Detach).

The fix: in wait_route, a CONFLICT from that look now waits for it to settle, then looks again, all within the WAIT's timeout. New Manager::wait_native_look returns once either holds:

  • the catalogue's revision has moved: an exit in flight becomes final through finish_detached / release_record, which move it;
  • nothing refuses a look any more: no exit in flight and no binding of this endpoint. A Detach notifies record.changed, which it also waits on.

Reading the revision needs no snapshot: new Server::native_catalogue_revision().

This surfaced as a flake of client_host's a_process_whose_attachment_went_is_waited_for_and_held_by_nothing on a loaded machine (the loop that KILLs and WAITs on each detached process got Process WAIT: Conflict).

New test a_wait_right_after_a_kill_of_a_detached_process_gets_its_exit, 64 rounds of spawn detachable → detach → KILL → WAIT:

  • without the change, 1 run in 10 failed (with the exit: Conflict);
  • with it, 0 in 20, then 0 in 10.

New test a_wait_that_finds_its_own_look_still_bound_answers_once_it_goes (r2, from the review of 3ec30d5). A watcher's route is failed, it WAITs, and its look is detached 200 ms later:

  • waiting for the catalogue alone, the WAIT slept until the child's second was over, then answered NOT_FOUND;
  • now it answers with the exit (15/15 runs).

Test hardening

a_watcher_whose_queue_fills_after_the_exit_fails_alone:

  • Its frame loop took only output and panicked on the stdin's StdinProgress event, which a slow machine delivers among the frames. The event now passes, as the other tests already allow.
  • It now waits for the child's reaping (Server::wait_reaped, cfg(all(test, unix))) instead of sleeping 500 ms.

Known, left for a follow-up PR

  1. an_owner_that_drops_its_stream_after_the_exit_still_waits_for_it (unchanged here). It has two problems.
    • Its premise is timing-based. With the 1 MiB default window, a reader whose reads are small reaches the 32 unacknowledged frames first. The pipe then fills and the child never exits; after the 800 ms sleep the detach comes first.
    • It exposes a real race: the owner's WAIT can miss a non-detachable process's exit. The owner's route can leave after the child exits but before its terminal is delivered. The record is then released with no replay and nothing in finals, so a WAIT that looks after the release answers NOT_FOUND. The comment in dispatch_outbound's Exit arm describes this path.
    • The same thing happens to a connection that drops and reconnects just as a process exits. The follow-up keeps such a process's final for its owner, bounded like finals. That is a design change in Pace process output by its owner; drop a lagging watcher alone #85/Report a spawned process's exit unasked: bash in one round trip #86's area, and its PR will say so.
  2. a_watcher_whose_queue_fills_after_the_exit_fails_alone still assumes 80 lines make 80 frames. A loaded reader can read several lines at once, and then the Exit comes before the 80th frame. The follow-up reworks it so it doesn't depend on read sizes.

Checks

  • cargo fmt --check: passes.
  • cargo clippy -p yas-server --all-targets -D warnings: clean.
  • The KILL → WAIT test: 10/10 runs pass (and 20/20, then 10/10, before r2).
  • The own-look test: 15/15.
  • a_watcher_whose_queue_fills_after_the_exit_fails_alone: 10/10 (r1).
  • yas-server lib: 938/938 in 3 runs. Known follow-up 1 is an_owner_that_drops_its_stream_after_the_exit_still_waits_for_it, which failed 1 lib run with NOT_FOUND.
  • client_host: 38/38.

All on a machine at load 50–110. Fork CI: see the linked indent-com draft.

A WAIT on a process its session holds no route to looks at it with a temporary
WATCH. That look is refused as CONFLICT while the process's exit is on its way to
its watchers (the record is final only once they all have it), or while the
session's own previous look, such as the one a KILL's CONTROL just made, is still
leaving. So a WAIT right after a KILL of a detached process could answer CONFLICT
instead of the exit: client_host's
a_process_whose_attachment_went_is_waited_for_and_held_by_nothing failed so on a
loaded machine. Such a WAIT now waits for the process catalogue's next change,
with which either settles, and looks again, within its timeout.

a_wait_right_after_a_kill_of_a_detached_process_gets_its_exit (64 rounds of
spawn detachable, detach, KILL, WAIT) failed 1 run in 10 without the change, and
none in 20 with it.

a_watcher_whose_queue_fills_after_the_exit_fails_alone took only output in its
frame loop and panicked on the stdin's progress event, which a slow machine
delivers among the frames; it lets it pass now, and waits for the child's reaping
(Server::wait_reaped, tests only) rather than 500 ms.
@ghost

ghost commented Sep 30, 2026 •

Copy link
Copy Markdown

Reviewed 3ec30d5 in 9m 25s: 2 issues · Session

@ghost ghost left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ultimator's review of 3ec30d5

When a WAIT's temporary look gets CONFLICT, it now waits for the catalogue's next change and looks again within its timeout. This fixes KILL → WAIT on a detached process (the terminal_queued case settles through finish_detached/release_record, and the new test passes here). It also hardens two tests. The other case the new comment names, this session's own look still bound, isn't settled by a catalogue change (#1).

2 issues: 1 bug, 1 nit

Outside the diff (1)
  • ⚪ Nit · crates/server/src/yas_process.rs:995-997 · Exit-arm comment still says the WAIT answers CONFLICT (#2): With this change, a WAIT in this session no longer answers CONFLICT while the exit is in flight. It waits for the exit to land, then answers with the final record (for a detachable process) or NOT_FOUND. The PR description points to this comment as describing the path.

No issue is open since the review of d721756.

Session · @ultimator-agent review reviews again · @ultimator-agent fix fixes the open issues

Comment thread crates/server/src/yas_process.rs
A WAIT's temporary look can be refused as CONFLICT because this session's own
look at the process is still bound: a concurrent CONTROL's, or a route that an
eviction failed and has yet to detach (dispatch_outbound fails the route, then
sends the Detach). Such a WAIT waited for the catalogue's next change, which a
Detach doesn't make, so it slept until the process ended and then found nothing
of an ordinary process: NOT_FOUND. It now waits with Manager::wait_native_look,
which also wakes when the record changes and returns once nothing refuses a look:
no exit in flight and no binding of this endpoint.

a_wait_that_finds_its_own_look_still_bound_answers_once_it_goes fails a watcher's
route, WAITs, and detaches the look 200 ms later. Without the change the WAIT
answered NOT_FOUND once the child's second had passed; with it, the exit.

The Exit arm's comment still said such a WAIT answers CONFLICT; it says what it
does now.
@ghost

ghost commented Sep 30, 2026 •

Copy link
Copy Markdown

Reviewed d721756 in 1m 44s: no new issues, 2 resolved · Session

@ghost ghost left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ultimator's review of d721756

d721756 swaps the plain catalogue wait for Manager::wait_native_look. It arms catalog_changed and the record's changed before checking. It returns once the revision moves, the handle leaves live, or neither terminal_queued nor this endpoint's own binding still refuses the look. A Detach (which notifies record.changed) now lets the WAIT attach again and get the exit. The new eviction-order test and the KILL → WAIT test pass here (3/3 runs). The Exit-arm comment is updated. Nothing new.

No issues found · 2 resolved

Session · @ultimator-agent review reviews again · @ultimator-agent fix fixes the open issues

@pcarrier

Copy link
Copy Markdown
Collaborator Author

Included in #93 (dd33f04), which Pierre squash-merged with both of #92's commits.

@pcarrier pcarrier closed this Sep 30, 2026
@pcarrier
pcarrier deleted the test-waits branch September 30, 2026 15:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant