Repair a local Ray head without serving_slot; stop early on unrecoverable baseline failures

[zoroyihan7]

• Description: what and why

  A Ray head that already exists when Hyperloom starts (for example one started by hand from the troubleshooting doc) is reused by ensure_ray_cluster as long as ray status answers. If it lacks the serving_slot custom resource, every serving lease fails _assert_cluster_feasible with existing Ray head has no serving_slot resource, each baseline attempt fails as subprocess_nonzero, and the session only ends after the retry budget, with long idle stretches in between (the first failure opens the enablement lane, which has no launch log to author against and never schedules another baseline). The installer already restarts such a head; the runtime path did not.

  1. Runtime repair (_ray_serving._ensure_cluster_feasible, _ray_runtime.local_head_restartable / restart_local_head_with_serving_slot): when the connected head lacks serving_slot, restart it the way the installer does (ray stop --force, then ray start --head ... --resources={"serving_slot": 1}), reconnect, and re-check feasibility. Because ray stop --force stops every Ray process on the host, the restart is refused (and the existing error kept, with the reason appended) unless all of these hold: no explicit RAY_ADDRESS (only unset/auto; local would start a new cluster on reconnect), not a multi-node run, exactly one node record in the cluster and it is this host's live raylet, the only gcs_server and raylet visible on the host are the connected cluster's (by --gcs_server_port / --node_id), no resource held, no actor that is not dead, no other live driver (by job id), and this process has not submitted a lease actor yet (actor creation is asynchronous). Leases now connect, check feasibility and create their actor under one lock that the repair also holds. The slot is checked before the GPU count so a hand-started head short of both still reaches the repair.
  2. Baseline stop (loop/writeback.py): infeasibility errors now carry a stable ray_cluster_infeasible marker. A baseline that fails with it stops the run as baseline_failed on the first occurrence (also in enablement) and is not stashed as an enablement launch log; whatever the runtime could repair it already has. Separately, two consecutive subprocess_nonzero failures whose whole error text matches after folding numbers/hex ids stop at two instead of three (outside enablement only; other classes carry fixed sentences and keep the three-strike rule). New state field baseline_last_failure_signature.
  3. Docs: the manual ray start --head commands in troubleshooting.md and upgrade.md now declare serving_slot, with a note on the automatic repair and when it is refused.

• Linked issue(s): none

• Tests: added/updated? commands run?

  • New orchestrator/actions/executors/tests/test_ray_serving_slot_repair.py: lone idle local head without the slot is restarted then feasible (also end to end through ServingLease.ensure, and when it is also short of GPUs); head with the slot, or a lease that needs no slot, is never restarted; explicit RAY_ADDRESS (incl. local), multi-node, a second live or dead node record, a head on another host, held resources, a resourceless live actor, another live driver (same pid, different job id), an uninspectable cluster, a second head / a foreign raylet / a non-matching head on the host, and a lease actor already submitted all keep the error; actor creation happens under the startup lock; /proc parsing; the restart runs stop then a start declaring the slot.
  • test_coordinator_async_methods_coverage_unit.py: same failure twice stops at two; different failures, a shared long traceback tail, a repeated non-subprocess_nonzero class, and a repeat inside enablement keep the old budget; an infeasible cluster stops on the first failure, also in enablement, with no launch log stashed.
  • Every new guard was mutated (removed or weakened) and at least one test failed for each; restored afterwards.
  • Ran: ruff check ., ruff format --check . (0.16.2), and pytest on test_ray_*.py, test_ray_backend_unit.py, test_coordinator_async_methods_coverage_unit.py, test_dead_holder_failure_accounting.py, test_dispatched_task_policy.py, test_bringup_round_scenario.py, test_enablement_*, test_sbd_v6_enablement_wiring.py, test_phase_state_machine.py, test_shared_state_evolution.py, test_backend_gating.py, test_gaps_field.py: all pass. Ray itself is faked in the unit tests (the private ray._private.state reads and /proc layout were checked against the Ray 2.44.1 sources).

• Size/complexity triggers crossed: the diff is mostly tests (~470 of ~850 added lines); production code is split into small helpers (_visible_ray_daemons, _cluster_activity, local_head_restartable, _ensure_cluster_feasible, _baseline_failure_signature).

• If this simplifies or refactors: no; force_restart_local_cluster gained an optional reason and a generic failure message, behaviour otherwise unchanged.

• Observable effect: a session started on a host where an idle, lone, hand-started Ray head lacks serving_slot now logs Ray head has no serving_slot resource; restarting the local head with it and proceeds. Where the restart is not safe, the existing error now says why (for example ... not restarting it automatically: RAY_ADDRESS='...' names an explicit cluster) and the session ends as baseline_failed on the first baseline instead of after three attempts and hours of idling. Two identical subprocess_nonzero baseline failures end the session one attempt earlier.

• Breaking changes: no.

• PR addresses single concern: yes (a pre-existing Ray head without serving_slot and the retries it caused).

• Root cause is upstream: no.

🤖 Generated with Claude Code