num_steps_per_rollout / use-dynamic-global-batch-size semantics are unclear and unguarded in async/fully-async training

[Shi-Dong]

Summary

In async and fully-async training, --num-steps-per-rollout and --use-dynamic-global-batch-size still take full effect through the shared training path, but their interaction with off-policyness is undocumented and unguarded. Setting num_steps_per_rollout > 1 alongside staleness-based async silently stacks per-batch off-policyness on top of the inter-batch weight-version gap, with no warning.

Background

Fully-async (examples/fully_async/) is not a distinct mode — it reuses the train_async.py driver and swaps in a custom rollout function (generate_rollout_fully_async) that drains a background queue. The drained batch (rollout_batch_size × n_samples_per_prompt samples) flows through the normal actor_model.train() consume path:

• data.py:357 — num_steps_per_rollout = num_local_samples // num_local_gbs, so the drained batch is split into that many optimizer steps.
• rollout.py:573-588 — the trim / dynamic-global-batch resize block runs as usual.

So both flags behave exactly as in sync training.

The problem

1. Silent off-policyness foot-gun. With --max-weight-staleness set (async), each drained batch already carries an inter-batch staleness gap. If num_steps_per_rollout > 1, the 2nd…Nth optimizer step is applied to weights that have already moved while the data was generated by the pre-update policy — reintroducing per-batch off-policyness on top of staleness. Nothing warns the user.

2. use-dynamic-global-batch-size is redundant, not harmful, in fully-async. Its real purpose is avoiding sample-trim waste when the collected sample count is variable (dynamic sampling / oversampling recycle). In fully-async the drain count is fixed at rollout_batch_size, so it just resizes global_batch_size to force num_steps_per_rollout = 1 — a legitimate but non-obvious use. Its help text ("disable trim samples…") doesn't convey this.

3. Docs never connect the two. docs/user-guide/fully-async.md covers off-policyness only via --max-weight-staleness; the one-optimizer-step-per-drain invariant and the role of these two flags are undocumented (the num_steps_per_rollout=1 rationale lives only as a docstring in ray/rollout.py).

Proposed fix

• Warn (not error) when num_steps_per_rollout > 1 and max_weight_staleness is set, explaining the compounded off-policyness. A hard error would wrongly break legitimate overlapped-async runs where multi-step is valid.
• Add an info/log line noting use-dynamic-global-batch-size is redundant in fully-async (drain count is already fixed).
• Document the one-step-per-drain invariant and both flags' roles in fully-async.md.

Non-goals / rejected alternatives

• Disabling or hard-erroring on either flag in async — rejected: there's no first-class --fully-async flag to gate on (only the rollout_function_path heuristic), and num_steps_per_rollout > 1 is valid in overlapped async. Erroring would break real configs.

Related

• Doc PR docs(async): document dynamic global batch pinning num_steps_per_rollout=1 #1696 (adds a first note on dynamic global batch to fully-async.md; will be reconciled with this issue).