Skip to content

fix(deepswe): Configure default EOS_TOKENS and plumb into rollout launchers - #2314

Open
andytwigg wants to merge 1 commit into
atwigg/mlperffrom
atwigg/fix-deepswe-eos-tokens
Open

andytwigg wants to merge 1 commit into
atwigg/mlperffrom
atwigg/fix-deepswe-eos-tokens

Conversation

@andytwigg

Copy link
Copy Markdown
Collaborator

Summary

This PR configures default EOS_TOKENS and plumbs them to rollout nodes for DeepSWE distributed RL runs (both local and GKE k8s), aligning with the pattern established in math_gsm8k_dist.

Problem

  • In tunix/experimental/examples/deepswe_dist/launcher.sh, EOS_TOKENS defaulted to empty string (${EOS_TOKENS-}), so --eos_tokens was never passed to run_rollout_node.main unless manually set in the environment.
  • In tunix/experimental/examples/deepswe_dist/k8s_launcher.sh, EOS_TOKENS was neither declared nor passed in start_rollout().
  • Qwen models (Qwen3-1.7B, Qwen3-4B, Qwen3.5-35B-A3B) finish assistant turns with <|im_end|> (token 151645), while <|endoftext|> is token 151643. Without 151645 configured as an EOS token, vLLM fails to stop at the end of turns during rollouts, generating up to MAX_RESPONSE_LENGTH tokens (e.g. 61,440 tokens), hallucinating additional turns, and causing trajectory timeouts.

Changes

  1. tunix/experimental/examples/deepswe_dist/launcher.sh:
    • Set default EOS_TOKENS=${EOS_TOKENS-'151645,151643'}.
    • Display eos tokens in startup banner.
  2. tunix/experimental/examples/deepswe_dist/k8s_launcher.sh:
    • Define and export default export EOS_TOKENS=${EOS_TOKENS-'151645,151643'}.
    • Plumb ${EOS_TOKENS:+--eos_tokens=\"${EOS_TOKENS}\"} \ into run_rollout_node.main startup command.

Verification

  • Validated bash syntax (bash -n).
  • Verified generated rollout JobSet manifests via --dry-run:
    • Defaults to --eos_tokens="151645,151643".
    • Setting EOS_TOKENS="" omits --eos_tokens.
    • Setting custom EOS_TOKENS is correctly passed.

…nchers

- Set default EOS_TOKENS='151645,151643' in launcher.sh (previously defaulted to empty)
- Define and export default EOS_TOKENS in k8s_launcher.sh
- Plumb --eos_tokens to run_rollout_node in k8s_launcher.sh
- Display eos tokens in launcher.sh startup banner
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants