WIP: fix two DDP wandb logging bugs (duplicate runs + dropped eval metrics)

[eugenevinitsky]

WIP. Two independent DDP-only logging bugs surfaced while running the multi-agent nightly on 4 GPUs.

1. Duplicate wandb runs (one per rank)

Logger creation in train() was ungated, so under torchrun every rank called wandb.init() → world_size duplicate runs in the same group. Now only rank 0 builds the logger; other ranks keep logger=None (PuffeRL wraps that in NoLogger). Eval was already rank-0-only, so nothing else changes.

2. Eval metrics silently dropped by wandb

Training logs at agent_steps = dist_sum(global_step) (summed across ranks); eval logged at the raw per-rank global_step, which is world_size× smaller. wandb rejects every eval log as non-monotonic

[Copilot]

Pull request overview

Fixes two DDP-only logging bugs: (1) every rank was creating its own wandb/Neptune/TB run, producing world_size duplicate runs per training job, and (2) eval metrics were being silently rejected by wandb because eval logged at the raw per-rank global_step while training logged at the rank-summed agent_steps, making the eval step non-monotonic.

Changes:

• Gate logger construction in train() to rank 0 only; non-rank-0 ranks pass logger=None and are wrapped in NoLogger by PuffeRL.
• Cache the agent_steps (rank-summed global_step) computed inside mean_and_log on self.agent_steps, and use it as the step for both the in-loop and final force=True _eval_manager.maybe_run calls.
• Initialize self.agent_steps = 0 in PuffeRL.__init__.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.