assert labels.shape == (n_rows,)

[z-x-x136]

System Info

(TaskRunner pid=782838) step:0 - val-core/vision_table_comparison/reward/mean@8:0.5797759237576205 - val-aux/vision_table_comparison/reward/std@8:0.04346181558967558 - val-aux/vision_table_comparison/reward/best@2/mean:0.5981306398997328 - val-aux/vision_table_comparison/reward/best@2/std:0.027649894546757196 - val-aux/vision_table_comparison/reward/worst@2/mean:0.5615872453302144 - val-aux/vision_table_comparison/reward/worst@2/std:0.0451028965670866 - val-aux/vision_table_comparison/reward/best@4/mean:0.607696205476826 - val-aux/vision_table_comparison/reward/best@4/std:0.013642463673652247 - val-aux/vision_table_comparison/reward/worst@4/mean:0.5405101236723503 - val-aux/vision_table_comparison/reward/worst@4/std:0.04120077883325907 - val-core/vision_table_comparison/reward/best@8/mean:0.6128662523748177 - val-core/vision_table_comparison/reward/best@8/std:0.0074369439234818575 - val-aux/vision_table_comparison/reward/worst@8/mean:0.5212478603658811 - val-aux/vision_table_comparison/reward/worst@8/std:0.03235882541980303 - val-aux/num_turns/min:2 - val-aux/num_turns/max:2 - val-aux/num_turns/mean:2.0
(TaskRunner pid=782838) Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::WorkerDict.ref_compute_ref_log_prob() (pid=790512, ip=192.168.0.11, actor_id=d605e2db7eb23153620ff54801000000, repr=<verl.single_controller.ray.base.WorkerDict object at 0x7f7830d00470>)
(TaskRunner pid=782838) File "/usr/lib/python3.12/concurrent/futures/_base.py", line 456, in result
(TaskRunner pid=782838) return self.__get_result()
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(TaskRunner pid=782838) raise self._exception
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/workspace/verl/verl/single_controller/ray/base.py", line 841, in func
(TaskRunner pid=782838) return getattr(self.worker_dict[key], name)(*args, **kwargs)
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/workspace/verl/verl/single_controller/base/decorator.py", line 456, in inner
(TaskRunner pid=782838) return func(*args, **kwargs)
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/workspace/verl/verl/utils/transferqueue_utils.py", line 314, in dummy_inner
(TaskRunner pid=782838) output = func(*args, **kwargs)
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/workspace/verl/verl/utils/profiler/profile.py", line 274, in wrapper
(TaskRunner pid=782838) return func(self_instance, *args, **kwargs_inner)
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/workspace/verl/verl/workers/fsdp_workers.py", line 1061, in compute_ref_log_prob
(TaskRunner pid=782838) output, _ = self.ref_policy.compute_log_prob(data=data, calculate_entropy=False)
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/workspace/verl/verl/utils/profiler/performance.py", line 105, in f
(TaskRunner pid=782838) return self.log(decorated_function, *args, **kwargs)
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/workspace/verl/verl/utils/profiler/performance.py", line 118, in log
(TaskRunner pid=782838) output = func(*args, **kwargs)
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/workspace/verl/verl/workers/actor/dp_actor.py", line 378, in compute_log_prob
(TaskRunner pid=782838) entropy, log_probs = self._forward_micro_batch(
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/workspace/verl/verl/workers/actor/dp_actor.py", line 219, in _forward_micro_batch
(TaskRunner pid=782838) log_probs = logprobs_from_logits(
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/workspace/verl/verl/utils/torch_functional.py", line 94, in logprobs_from_logits
(TaskRunner pid=782838) output = logprobs_from_logits_flash_attn(logits, labels, inplace_backward=inplace_backward)
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/workspace/verl/verl/utils/torch_functional.py", line 122, in logprobs_from_logits_flash_attn
(TaskRunner pid=782838) output = cross_entropy_loss(logits, labels, inplace_backward=inplace_backward)
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/usr/local/lib/python3.12/dist-packages/flash_attn/ops/triton/cross_entropy.py", line 320, in cross_entropy_loss
(TaskRunner pid=782838) return CrossEntropyLoss.apply(
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/usr/local/lib/python3.12/dist-packages/torch/autograd/function.py", line 576, in apply
(TaskRunner pid=782838) return super().apply(*args, **kwargs) # type: ignore[misc]
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/usr/local/lib/python3.12/dist-packages/flash_attn/ops/triton/cross_entropy.py", line 171, in forward
(TaskRunner pid=782838) assert labels.shape == (n_rows,)
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) AssertionError
(TaskRunner pid=782838) Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::WorkerDict.ref_compute_ref_log_prob() (pid=790557, ip=192.168.0.11, actor_id=124c3751660a438e2946d09201000000, repr=<verl.single_controller.ray.base.WorkerDict object at 0x7fa509b985f0>)
(TaskRunner pid=782838) File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
(TaskRunner pid=782838) return self.__get_result()
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(TaskRunner pid=782838) raise self._exception
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/workspace/verl/verl/single_controller/ray/base.py", line 841, in func
(TaskRunner pid=782838) return getattr(self.worker_dict[key], name)(*args, **kwargs)
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/workspace/verl/verl/single_controller/base/decorator.py", line 456, in inner
(TaskRunner pid=782838) return func(*args, **kwargs)
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/workspace/verl/verl/utils/transferqueue_utils.py", line 314, in dummy_inner
(TaskRunner pid=782838) output = func(*args, **kwargs)
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/workspace/verl/verl/utils/profiler/profile.py", line 274, in wrapper
(TaskRunner pid=782838) return func(self_instance, *args, **kwargs_inner)
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/workspace/verl/verl/workers/fsdp_workers.py", line 1061, in compute_ref_log_prob
(TaskRunner pid=782838) output, _ = self.ref_policy.compute_log_prob(data=data, calculate_entropy=False)
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/workspace/verl/verl/utils/profiler/performance.py", line 105, in f
(TaskRunner pid=782838) return self.log(decorated_function, *args, **kwargs)
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/workspace/verl/verl/utils/profiler/performance.py", line 118, in log
(TaskRunner pid=782838) output = func(*args, **kwargs)
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/workspace/verl/verl/workers/actor/dp_actor.py", line 378, in compute_log_prob
(TaskRunner pid=782838) entropy, log_probs = self._forward_micro_batch(
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/workspace/verl/verl/workers/actor/dp_actor.py", line 219, in _forward_micro_batch
(TaskRunner pid=782838) log_probs = logprobs_from_logits(
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/workspace/verl/verl/utils/torch_functional.py", line 94, in logprobs_from_logits
(TaskRunner pid=782838) output = logprobs_from_logits_flash_attn(logits, labels, inplace_backward=inplace_backward)
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/workspace/verl/verl/utils/torch_functional.py", line 122, in logprobs_from_logits_flash_attn
(TaskRunner pid=782838) output = cross_entropy_loss(logits, labels, inplace_backward=inplace_backward)
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/usr/local/lib/python3.12/dist-packages/flash_attn/ops/triton/cross_entropy.py", line 320, in cross_entropy_loss
(TaskRunner pid=782838) return CrossEntropyLoss.apply(
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/usr/local/lib/python3.12/dist-packages/torch/autograd/function.py", line 576, in apply
(TaskRunner pid=782838) return super().apply(*args, **kwargs) # type: ignore[misc]
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/usr/local/lib/python3.12/dist-packages/flash_attn/ops/triton/cross_entropy.py", line 171, in forward
(TaskRunner pid=782838) assert labels.shape == (n_rows,)
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) AssertionError
(TaskRunner pid=782838) Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::WorkerDict.ref_compute_ref_log_prob() (pid=790562, ip=192.168.0.11, actor_id=c122ced1511239abde824b3f01000000, repr=<verl.single_controller.ray.base.WorkerDict object at 0x7fcf2dc44740>)
(TaskRunner pid=782838) File "/usr/lib/python3.12/concurrent/futures/_base.py", line 456, in result
(TaskRunner pid=782838) return self.__get_result()
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(TaskRunner pid=782838) raise self._exception
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/workspace/verl/verl/single_controller/ray/base.py", line 841, in func
(TaskRunner pid=782838) return getattr(self.worker_dict[key], name)(*args, **kwargs)
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/workspace/verl/verl/single_controller/base/decorator.py", line 456, in inner
(TaskRunner pid=782838) return func(*args, **kwargs)
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/workspace/verl/verl/utils/transferqueue_utils.py", line 314, in dummy_inner
(TaskRunner pid=782838) output = func(*args, **kwargs)
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/workspace/verl/verl/utils/profiler/profile.py", line 274, in wrapper
(TaskRunner pid=782838) return func(self_instance, *args, **kwargs_inner)
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/workspace/verl/verl/workers/fsdp_workers.py", line 1061, in compute_ref_log_prob
(TaskRunner pid=782838) output, _ = self.ref_policy.compute_log_prob(data=data, calculate_entropy=False)
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/workspace/verl/verl/utils/profiler/performance.py", line 105, in f
(TaskRunner pid=782838) return self.log(decorated_function, *args, **kwargs)
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/workspace/verl/verl/utils/profiler/performance.py", line 118, in log
(TaskRunner pid=782838) output = func(*args, **kwargs)
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/workspace/verl/verl/workers/actor/dp_actor.py", line 378, in compute_log_prob
(TaskRunner pid=782838) entropy, log_probs = self._forward_micro_batch(
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/workspace/verl/verl/workers/actor/dp_actor.py", line 219, in _forward_micro_batch
(TaskRunner pid=782838) log_probs = logprobs_from_logits(
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/workspace/verl/verl/utils/torch_functional.py", line 94, in logprobs_from_logits
(TaskRunner pid=782838) output = logprobs_from_logits_flash_attn(logits, labels, inplace_backward=inplace_backward)
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/workspace/verl/verl/utils/torch_functional.py", line 122, in logprobs_from_logits_flash_attn
(TaskRunner pid=782838) output = cross_entropy_loss(logits, labels, inplace_backward=inplace_backward)
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/usr/local/lib/python3.12/dist-packages/flash_attn/ops/triton/cross_entropy.py", line 320, in cross_entropy_loss
(TaskRunner pid=782838) return CrossEntropyLoss.apply(
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/usr/local/lib/python3.12/dist-packages/torch/autograd/function.py", line 576, in apply
(TaskRunner pid=782838) return super().apply(*args, **kwargs) # type: ignore[misc]
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) File "/usr/local/lib/python3.12/dist-packages/flash_attn/ops/triton/cross_entropy.py", line 171, in forward
(TaskRunner pid=782838) assert labels.shape == (n_rows,)
(TaskRunner pid=782838) ^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=782838) AssertionError
(TaskRunner pid=782838) swanlab: 🏠 View project at https://swanlab.cn/@zxx2000/qwen3vl_table_ocr
(TaskRunner pid=782838) swanlab: 🚀 View run at https://swanlab.cn/@zxx2000/qwen3vl_table_ocr/runs/jyqb6wz79pxyj1d93isth
Training Progress: 0%| | 0/500 [02:34<?, ?it/s]
Error executing job with overrides: ['algorithm.adv_estimator=grpo', 'data.train_files=/workspace/verl_grpo/verl_grpo_data/train.parquet', 'data.val_files=/workspace/verl_grpo/verl_grpo_data/test.parquet', 'critic.enable=False']
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/workspace/verl/verl/trainer/main_ppo.py", line 447, in
main()
File "/usr/local/lib/python3.12/dist-packages/hydra/main.py", line 94, in decorated_main
_run_hydra(
File "/usr/local/lib/python3.12/dist-packages/hydra/_internal/utils.py", line 394, in _run_hydra
_run_app(
File "/usr/local/lib/python3.12/dist-packages/hydra/_internal/utils.py", line 457, in _run_app
run_and_report(
File "/usr/local/lib/python3.12/dist-packages/hydra/_internal/utils.py", line 223, in run_and_report
raise ex
File "/usr/local/lib/python3.12/dist-packages/hydra/_internal/utils.py", line 220, in run_and_report
return func()
^^^^^^
File "/usr/local/lib/python3.12/dist-packages/hydra/_internal/utils.py", line 458, in
lambda: hydra.run(
^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/hydra/_internal/hydra.py", line 132, in run
_ = ret.return_value
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/hydra/core/utils.py", line 260, in return_value
raise self._return_value
File "/usr/local/lib/python3.12/dist-packages/hydra/core/utils.py", line 186, in run_job
ret.return_value = task_function(task_cfg)
^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/verl/verl/trainer/main_ppo.py", line 45, in main
run_ppo(config)
File "/workspace/verl/verl/trainer/main_ppo.py", line 99, in run_ppo
ray.get(runner.run.remote(config))
File "/usr/local/lib/python3.12/dist-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 2882, in get
values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 968, in get_objects
raise value.as_instanceof_cause()

5bdabf01000000, repr=<main_ppo.TaskRunner object at 0x7f7e30145e20>)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/verl/verl/trainer/main_ppo.py", line 366, in run
trainer.fit()
File "/workspace/verl/verl/trainer/ppo/ray_trainer.py", line 1483, in fit
ref_log_prob = self._compute_ref_log_prob(batch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/verl/verl/trainer/ppo/ray_trainer.py", line 1177, in _compute_ref_log_prob
ref_log_prob = self.ref_policy_wg.compute_ref_log_prob(batch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/verl/verl/single_controller/ray/base.py", line 53, in call
output = ray.get(output)
^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ray.exceptions.RayTaskError(AssertionError): ray::WorkerDict.ref_compute_ref_log_prob() (pid=790530, ip=192.168.0.11, actor_id=7577f135929f6e4afe85fe4b01000000, repr=<verl.single_controller.ray.base.WorkerDict object at 0x7f979cf144a0>)
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/verl/verl/single_controller/ray/base.py", line 841, in func
return getattr(self.worker_dict[key], name)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/verl/verl/single_controller/base/decorator.py", line 456, in inner
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/workspace/verl/verl/utils/transferqueue_utils.py", line 314, in dummy_inner
output = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/workspace/verl/verl/utils/profiler/profile.py", line 274, in wrapper
return func(self_instance, *args, **kwargs_inner)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/verl/verl/workers/fsdp_workers.py", line 1061, in compute_ref_log_prob
output, _ = self.ref_policy.compute_log_prob(data=data, calculate_entropy=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/verl/verl/utils/profiler/performance.py", line 105, in f
return self.log(decorated_function, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/verl/verl/utils/profiler/performance.py", line 118, in log
output = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/workspace/verl/verl/workers/actor/dp_actor.py", line 378, in compute_log_prob
entropy, log_probs = self._forward_micro_batch(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/verl/verl/workers/actor/dp_actor.py", line 219, in _forward_micro_batch
log_probs = logprobs_from_logits(
^^^^^^^^^^^^^^^^^^^^^
File "/workspace/verl/verl/utils/torch_functional.py", line 94, in logprobs_from_logits
output = logprobs_from_logits_flash_attn(logits, labels, inplace_backward=inplace_backward)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/verl/verl/utils/torch_functional.py", line 122, in logprobs_from_logits_flash_attn
output = cross_entropy_loss(logits, labels, inplace_backward=inplace_backward)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/flash_attn/ops/triton/cross_entropy.py", line 320, in cross_entropy_loss
return CrossEntropyLoss.apply(
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/autograd/function.py", line 576, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/flash_attn/ops/triton/cross_entropy.py", line 171, in forward
assert labels.shape == (n_rows,)
^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

GRPO训练qwen3-vl-thinking-4b

ubuntu@zhizhehui:~/verl_grpo$ pip show verl
Name: verl
Version: 0.7.0.dev0
Summary: verl: Volcano Engine Reinforcement Learning for LLM
Home-page: https://github.com/volcengine/verl
Author: Bytedance - Seed - MLSys
Author-email: zhangchi.usc1992@bytedance.com, gmsheng@connect.hku.hk
License: Apache-2.0
Location: /workspace/.local/lib/python3.12/site-packages
Editable project location: /workspace/verl
Requires: accelerate, codetiming, datasets, dill, hydra-core, numpy, packaging, pandas, peft, pyarrow, pybind11, pylatexenc, ray, tensorboard, tensordict, torchdata, transformers, wandb
Required-by:

Expected behavior

应该是正常跑到step1

[z-x-x136]

ubuntu@zhizhehui:~/verl_grpo$ pip list
Package Version Editable project location

---

absl-py 2.2.2
accelerate 1.10.1
aiohappyeyeballs 2.6.1
aiohttp 3.11.16
aiohttp-cors 0.8.1
aiosignal 1.3.2
annotated-types 0.7.0
antlr4-python3-runtime 4.9.3
anyio 4.9.0
argon2-cffi 23.1.0
argon2-cffi-bindings 21.2.0
arrow 1.3.0
asciitree 0.3.3
astor 0.8.1
asttokens 3.0.0
astunparse 1.6.3
async-lru 2.0.5
attrs 25.3.0
audioread 3.0.1
autocommand 2.2.2
av 15.1.0
babel 2.17.0
backports.tarfile 1.2.0
beautifulsoup4 4.13.4
black 25.1.0
blake3 1.0.7
bleach 6.2.0
blis 0.7.11
cachetools 5.5.2
catalogue 2.0.10
cbor2 5.7.0
certifi 2025.4.26
cffi 1.17.1
cfgv 3.4.0
charset-normalizer 3.4.1
click 8.1.8
cloudpathlib 0.21.1
cloudpickle 3.1.1
cmake 3.31.6
codetiming 1.4.0
colorful 0.5.7
comm 0.2.2
compressed-tensors 0.11.0
confection 0.1.5
contourpy 1.3.2
cuda-bindings 12.8.0
cuda-python 12.8.0
cudf 25.4.0
cudf-polars 25.4.0
cugraph 25.4.0
cugraph-service-client 25.4.0
cugraph-service-server 25.4.0
cuml 25.4.0
cupy-cuda12x 13.3.0
cuvs 25.4.0
cycler 0.12.1
cymem 2.0.11
Cython 3.0.12
dask 2025.2.0
dask-cuda 25.4.0
dask-cudf 25.4.0
datasets 4.1.1
dbus-python 1.3.2
debugpy 1.8.14
decorator 5.2.1
defusedxml 0.7.1
depyf 0.19.0
dill 0.4.0
diskcache 5.6.3
distlib 0.4.0
distributed 2025.2.0
distributed-ucxx 0.43.0
distro 1.9.0
dm-tree 0.1.9
dnspython 2.8.0
einops 0.8.1
email-validator 2.3.0
execnet 2.1.1
executing 2.2.0
expecttest 0.3.0
fastapi 0.118.0
fastapi-cli 0.0.13
fastapi-cloud-cli 0.3.0
fasteners 0.19
fastjsonschema 2.21.1
fastrlock 0.8.3
filelock 3.18.0
flash_attn 2.8.3
fonttools 4.58.0
fqdn 1.5.1
frozendict 2.4.6
frozenlist 1.5.0
fsspec 2025.3.2
gast 0.6.0
gguf 0.17.1
gitdb 4.0.12
GitPython 3.1.45
google-api-core 2.25.2
google-auth 2.41.1
googleapis-common-protos 1.70.0
grpcio 1.75.1
h11 0.16.0
hf_transfer 0.1.9
hf-xet 1.1.10
httpcore 1.0.9
httptools 0.6.4
httpx 0.28.1
huggingface-hub 0.35.3
hydra-core 1.3.2
hypothesis 6.130.8
identify 2.6.15
idna 3.10
importlib_metadata 8.0.0
inflect 7.3.1
iniconfig 2.1.0
intel-openmp 2021.4.0
interegular 0.3.3
ipykernel 6.29.5
ipython 9.2.0
ipython_pygments_lexers 1.1.1
isoduration 20.11.0
isort 6.0.1
jaraco.collections 5.1.0
jaraco.context 5.3.0
jaraco.functools 4.0.1
jaraco.text 3.12.1
jedi 0.19.2
Jinja2 3.1.6
jiter 0.11.0
joblib 1.4.2
json5 0.12.0
jsonpointer 3.0.0
jsonschema 4.23.0
jsonschema-specifications 2025.4.1
jupyter_client 8.6.3
jupyter_core 5.7.2
jupyter-events 0.12.0
jupyter-lsp 2.2.5
jupyter_server 2.16.0
jupyter_server_terminals 0.5.3
jupyterlab 4.4.2
jupyterlab_code_formatter 3.0.2
jupyterlab_pygments 0.3.0
jupyterlab_server 2.27.3
jupyterlab_tensorboard_pro 4.0.0
jupytext 1.17.1
kiwisolver 1.4.8
kvikio 25.4.0
langcodes 3.5.0
language_data 1.3.0
lark 1.2.2
lazy_loader 0.4
libcudf 25.4.0
libcugraph 25.4.0
libcuml 25.4.0
libcuvs 25.4.0
libkvikio 25.4.0
libraft 25.4.0
librmm 25.4.0
librmm-cu12 25.4.0
librosa 0.11.0
libucxx 0.43.0
liger_kernel 0.6.2
lightning-thunder 0.2.3.dev0
lightning-utilities 0.14.3
lintrunner 0.12.7
llguidance 0.7.30
llvmlite 0.44.0
lm-format-enforcer 0.11.3
locket 1.0.0
looseversion 1.3.0
marisa-trie 1.2.1
Markdown 3.8
markdown-it-py 3.0.0
MarkupSafe 3.0.2
mathruler 0.1.0
matplotlib 3.10.3
matplotlib-inline 0.1.7
mdit-py-plugins 0.4.2
mdurl 0.1.2
mistral_common 1.8.5
mistune 3.1.3
mkl 2021.1.1
mkl-devel 2021.1.1
mkl-include 2021.1.1
mock 5.2.0
more-itertools 10.3.0
mpmath 1.3.0
msgpack 1.1.0
msgspec 0.19.0
multidict 6.4.3
multiprocess 0.70.16
murmurhash 1.0.12
mypy_extensions 1.1.0
nbclient 0.10.2
nbconvert 7.16.6
nbformat 5.10.4
nest-asyncio 1.6.0
networkx 3.4.2
ninja 1.11.1.4
nodeenv 1.9.1
notebook 7.4.2
notebook_shim 0.2.4
numba 0.61.2
numba-cuda 0.4.0
numcodecs 0.13.1
numpy 1.26.4
nvdlfw_inspect 0.1.0
nvfuser 0.2.27a0+9bf5aca
nvidia-cublas-cu12 12.8.4.1
nvidia-cuda-cupti-cu12 12.8.90
nvidia-cuda-nvrtc-cu12 12.8.93
nvidia-cuda-runtime-cu12 12.8.90
nvidia-cudnn-cu12 9.10.2.21
nvidia-cudnn-frontend 1.11.0
nvidia-cufft-cu12 11.3.3.83
nvidia-cufile-cu12 1.13.1.3
nvidia-curand-cu12 10.3.9.90
nvidia-cusolver-cu12 11.7.3.90
nvidia-cusparse-cu12 12.5.8.93
nvidia-cusparselt-cu12 0.7.1
nvidia-dali-cuda120 1.49.0
nvidia-ml-py 12.570.86
nvidia-modelopt 0.27.1
nvidia-modelopt-core 0.27.1
nvidia-nccl-cu12 2.27.3
nvidia-nvcomp-cu12 4.2.0.14
nvidia-nvimgcodec-cu12 0.5.0.13
nvidia-nvjitlink-cu12 12.8.93
nvidia-nvjpeg-cu12 12.4.0.16
nvidia-nvjpeg2k-cu12 0.8.1.40
nvidia-nvtiff-cu12 0.5.0.67
nvidia-nvtx-cu12 12.8.90
nvidia-resiliency-ext 0.3.0
nvtx 0.2.11
nx-cugraph 25.4.0
omegaconf 2.3.0
onnx 1.17.0
openai 2.1.0
openai-harmony 0.0.4
opencensus 0.11.4
opencensus-context 0.1.3
opencv-python-headless 4.11.0.86
opentelemetry-api 1.37.0
opentelemetry-exporter-prometheus 0.58b0
opentelemetry-proto 1.27.0
opentelemetry-sdk 1.37.0
opentelemetry-semantic-conventions 0.58b0
opt_einsum 3.4.0
optree 0.15.0
orjson 3.11.3
outlines_core 0.2.11
overrides 7.7.0
packaging 25.0
pandas 2.2.3
pandocfilters 1.5.1
parso 0.8.4
partd 1.4.2
partial-json-parser 0.2.1.1.post6
pathspec 0.12.1
peft 0.17.1
pexpect 4.9.0
pillow 11.2.1
pip 25.2
platformdirs 4.3.8
pluggy 1.6.0
ply 3.11
polars 1.25.2
polygraphy 0.49.20
pooch 1.8.2
pre_commit 4.3.0
preshed 3.0.9
prettytable 3.17.0
prometheus_client 0.22.0
prometheus-fastapi-instrumentator 7.1.0
prompt_toolkit 3.0.51
propcache 0.3.1
proto-plus 1.26.1
protobuf 4.24.4
psutil 7.0.0
ptyprocess 0.7.0
PuLP 3.1.1
pure_eval 0.2.3
py-cpuinfo 9.0.0
py-spy 0.4.1
pyarrow 21.0.0
pyasn1 0.6.1
pyasn1_modules 0.4.2
pybase64 1.4.2
pybind11 2.13.6
pybind11_global 2.13.6
pycocotools 2.0+nv0.8.1
pycountry 24.6.1
pycparser 2.22
pydantic 2.11.10
pydantic_core 2.33.2
pydantic-extra-types 2.10.5
pyecharts 2.0.9
Pygments 2.19.1
PyGObject 3.48.2
pylatexenc 2.10
pylibcudf 25.4.0
pylibcugraph 25.4.0
pylibraft 25.4.0
pylibwholegraph 25.4.0
pynvjitlink 0.3.0
pynvml 12.0.0
pyparsing 3.2.3
pytest 8.1.1
pytest-flakefinder 1.1.0
pytest-rerunfailures 15.1
pytest-shard 0.1.2
pytest-xdist 3.6.1
python-dateutil 2.9.0.post0
python-dotenv 1.1.1
python-hostlist 2.2.1
python-json-logger 3.3.0
python-multipart 0.0.20
pytz 2023.4
pyvers 0.1.0
PyYAML 6.0.2
pyzmq 26.4.0
qwen-vl-utils 0.0.14
raft-dask 25.4.0
rapids-dask-dependency 25.4.0a0
rapids-logger 0.1.11
ray 2.49.2
referencing 0.36.2
regex 2024.11.6
requests 2.32.3
rfc3339-validator 0.1.4
rfc3986-validator 0.1.1
rich 13.9.4
rich-toolkit 0.15.1
rignore 0.7.0
rmm 25.4.0
rpds-py 0.25.1
rsa 4.9.1
ruff 0.13.3
safetensors 0.5.3
scikit-learn 1.6.1
scipy 1.15.2
Send2Trash 1.8.3
sentencepiece 0.2.1
sentry-sdk 2.39.0
setproctitle 1.3.7
setuptools 78.1.1
shellingham 1.5.4
simplejson 3.20.2
six 1.16.0
smart-open 7.1.0
smmap 5.0.2
sniffio 1.3.1
sortedcontainers 2.4.0
soundfile 0.13.1
soupsieve 2.7
soxr 0.5.0.post1
spacy 3.7.5
spacy-legacy 3.0.12
spacy-loggers 1.0.5
srsly 2.5.1
stack-data 0.6.3
starlette 0.48.0
swanlab 0.7.7
sympy 1.14.0
tabulate 0.9.0
tbb 2021.13.1
tblib 3.1.0
tensorboard 2.16.2
tensorboard-data-server 0.7.2
tensordict 0.10.0
tensorrt 10.10.0.31
terminado 0.18.1
thinc 8.2.5
threadpoolctl 3.6.0
thriftpy2 0.5.2
tiktoken 0.11.0
tinycss2 1.4.0
tokenizers 0.22.1
tomli 2.0.1
toolz 1.0.0
torch 2.8.0
torchaudio 2.8.0
torchdata 0.11.0
torchprofile 0.0.4
torchvision 0.23.0
tornado 6.5
tqdm 4.67.1
traitlets 5.14.3
transformers 4.57.0
treelite 4.4.1
triton 3.4.0
typeguard 4.3.0
typer 0.15.4
types-dataclasses 0.6.6
types-python-dateutil 2.9.0.20250516
typing_extensions 4.13.2
typing-inspection 0.4.1
tzdata 2025.2
ucx-py 0.43.0
ucxx 0.43.0
uri-template 1.3.0
urllib3 2.0.7
uvicorn 0.37.0
uvloop 0.21.0
verl 0.7.0.dev0 /workspace/verl
virtualenv 20.34.0
vllm 0.11.0
wandb 0.22.1
wasabi 1.1.3
watchfiles 1.1.0
wcwidth 0.2.13
weasel 0.4.1
webcolors 24.11.1
webencodings 0.5.1
websocket-client 1.8.0
websockets 15.0.1
Werkzeug 3.1.3
wheel 0.45.1
wrapt 1.17.2
xdoctest 1.0.2
xformers 0.0.32.post1
xgrammar 0.1.25
xxhash 3.6.0
yapf 0.43.0
yarl 1.19.0
zarr 2.18.7
zict 3.0.0
zipp 3.19.2

[leoriczhang]

解决了吗，我也遇到了

[Kirrito-k423]

This AssertionError: labels.shape == (n_rows,) indicates a shape mismatch in the flash attention cross-entropy loss function. The labels tensor has unexpected dimensions.

Common causes:

1. Multi-modal input mismatch: Qwen3-VL with vision tokens may have different label shapes due to image token handling. The labels tensor should be flattened to match the logits shape.

2. Packing/padding issue: If using use_remove_padding=True, the labels need to be properly packed alongside the input_ids.

Workaround:

• Check if labels is being correctly constructed in your data preprocessing
• Ensure vision token positions have labels=-100 (ignored in loss)
• Try setting actor_rollout_ref.actor.use_remove_padding=False to isolate the issue

Debugging:

# Add before the failing line in torch_functional.py
print(f"logits shape: {logits.shape}, labels shape: {labels.shape}")

This will show where the dimension mismatch originates.