feat: enable GRPO training with logprobs from offline trajectory data

[JRMeyer]

Summary

This PR enables proper GRPO training with importance sampling when using offline trajectory data (e.g., from vLLM traces). It includes four complementary changes:

1. Extract logprobs from dict messages

Problem: ART's tokenizer only extracted logprobs from OpenAI Choice objects, but offline trajectory data often stores logprobs in plain Python dicts. This caused all dict message logprobs to be set to NaN, making the importance ratio = 1.0 always (effectively REINFORCE instead of GRPO).

Solution: Modified tokenize.py to also extract logprobs from dict messages that have the format {"logprobs": {"content": [{"logprob": -0.5}, ...]}}.

2. Strip logprobs before RULER scoring

Problem: When trajectories contain verbose logprobs data, sending them to the RULER judge causes context length errors.

Solution: Strip logprobs from trajectories before sending to RULER using strip_logprobs().

3. Preserve _internal_config.engine_args

Problem: When using TrainableModel._internal_config.engine_args to configure vLLM engine settings (like max_logprobs), the configuration was silently lost when using the SkyPilot backend.

Solution: Add a model_validator(mode="wrap") to preserve _internal_config during Pydantic deserialization.

4. Add importance sampling observability metrics

Problem: ART computes importance sampling ratios internally but doesn't expose them, making it impossible to verify if importance sampling is actually working.

Solution: Add three new metrics logged during training:

• frac_old_logprobs_valid: Fraction of old logprobs that are not NaN (0 = no importance sampling)
• mean_importance_ratio: Mean π_new/π_old across assistant tokens (should vary around 1.0)
• clip_fraction: Fraction of tokens where PPO clipping was triggered (>0 means off-policy correction active)

Impact

Aspect	Before	After
Importance ratio	1.0 always (for dict messages)	π_new / π_old
PPO clipping	Never activates	Activates when ratio outside [0.8, 1.2]
Algorithm	REINFORCE	GRPO with importance sampling
Observability	None	frac_old_logprobs_valid, mean_importance_ratio, clip_fraction

New Metrics Interpretation

Metric	Working (GRPO)	Not Working (REINFORCE)
frac_old_logprobs_valid	> 0 (close to 1.0)	= 0 (all NaN)
mean_importance_ratio	varies around 1.0	exactly 1.0
clip_fraction	> 0	= 0

Test plan

• Verified max_logprobs setting works with SkyPilot backend
• Ran ./scripts/run_checks.sh - all checks pass
• Test with training that uses offline trajectory data with logprobs
• Verify new metrics appear in training logs/wandb