fix(monitor): fix wandb internal RSS growth at high sample log frequency

[mikasenghaas]

Summary

Following the glibc heap fix (#2527), a secondary RSS accumulation was identified in the orchestrator's wandb sample logging path. This PR tracks investigation and fix.

Bug: WandbMonitor.log_samples() causes ~2.5 MB/step RSS growth when log_extras.interval=1. At the default interval=10 this is ~0.3 MB/step — slow but unbounded over long runs.

Root cause (partially understood): The accumulation is inside wandb's internals — not the Python wandb.Table object itself. Resetting self.samples_table after each wandb.log() call (which clears table.data) had no measurable effect on RSS, ruling out the Python table object as the source. The leak is somewhere in wandb's serialization/upload layer.

Reproduction

# With sample logging every step (exaggerated to make leak visible faster)
uv run rl @ examples/alphabet_sort/rl.toml --max-steps 20 --clean-output-dir \
  --orchestrator.wandb.log-extras.interval 1

# Monitor post-trim RssAnon (replace PID)
watch -n3 "grep -m1 RssAnon /proc/<PID>/status"

Data

Post-trim RssAnon per completed step (alphabet-sort, 512 rollouts/step):

Step	No wandb (baseline)	wandb interval=1
2	941 MB	941 MB
3	941 MB (=)	941 MB (=)
4	941 MB (=)	944 MB (+3)
5	942 MB (=)	946 MB (+5)
6	—	950 MB (+9)
7	—	954 MB (+13)
8	—	958 MB (+17)

No-wandb baseline is flat. With wandb at interval=1: ~2.5 MB/step monotonic drift that malloc_trim cannot reclaim (Python/wandb heap, not glibc).

At interval=10 (default) the drift is ~0.3 MB/step — approximately 18 MB per hour at a 60s/step pace.

What was ruled out

• wandb.Table Python object accumulation: resetting self.samples_table after each wandb.log() call made no difference to the observed RSS drift — ruled out as the primary cause
• glibc heap fragmentation: fixed in fix(orchestrator): trim glibc heap after each step to prevent RSS growth #2527 via malloc_trim(0); confirmed not the cause of this residual drift