The actor died unexpectedly #123
Description
File "/data/xuelei12/RL_Factory/RL-Factory/verl/trainer/ppo/ray_trainer.py", line 1102, in fit
gen_batch_output = self.actor_rollout_wg.generate_sequences_loop(gen_batch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/xuelei12/RL_Factory/RL-Factory/verl/single_controller/ray/base.py", line 51, in call
output = ray.get(output)
^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ray.exceptions.ActorDiedError: The actor died unexpectedly before finishing this task.
class_name: create_colocated_worker_cls..WorkerDict
actor_id: 17f730ef04a3a7406b421ea101000000
pid: 16424
name: wd6rCqWorkerDict_0:5
namespace: bd542de1-90f2-46ba-8ece-bfcd459a3150
ip: 11.11.12.211
The actor is dead because its worker process has died. Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.