feat(dpo): support SimPO reference-free preference optimization with target margin

[hsusul]

Description

Direct Preference Optimization (DPO) and its variants are core alignment techniques in post-training. Standard DPO requires loading a separate reference model alongside the policy model throughout training, substantially increasing GPU memory consumption and inference overhead. Furthermore, DPO without length normalization frequently suffers from length exploitation (favoring longer responses merely due to cumulative token log-probabilities).

SimPO (Simple Preference Optimization with a Reference-Free Objective, Meng et al., NeurIPS 2024, https://arxiv.org/abs/2405.14734) directly solves these issues:

1. Reference-Model Free: Eliminates the reference policy $\pi_{\text{ref}}$, slashing memory and forward pass compute.
2. Length-Normalized Implicit Reward: Normalizes sequence log-probabilities by completion length:
   $$r_{\text{SimPO}}(x, y) = \frac{\beta}{|y|} \log \pi_\theta(y \mid x)$$
3. Target Reward Margin ($\gamma$): Enforces a margin between winning and losing generations:
   $$\mathcal{L}{\text{SimPO}} = -\mathbb{E}{(x, y_w, y_l)} \left[ \log \sigma \left( \frac{\beta}{|y_w|} \log \pi_\theta(y_w \mid x) - \frac{\beta}{|y_l|} \log \pi_\theta(y_l \mid x) - \gamma \right) \right]$$

This PR introduces SimPO support into AReaL's preference learning engine:

1. Configuration:
   • Added "simpo" to loss_type in DPOEngineConfig (["sigmoid", "ipo", "simpo"]).
   • Added simpo_gamma: float = 0.5 with non-negative validation (--dpo.actor.simpo_gamma and --dpo.ref.simpo_gamma).
2. Loss Formulation:
   • Extended dpo_preference_loss in areal.utils.functional with loss_type="simpo" and simpo_gamma margin.
   • Updated compute_dpo_loss in areal.trainer.dpo.dpo_engine to compute length-normalized policy log-probabilities and margin-adjusted logits.
3. Implicit Reward & Training Metrics:
   • Computes length-normalized implicit rewards ($r_w, r_l$), reward accuracy ($\mathbb{I}(r_w > r_l)$), and reward margin ($r_w - r_l$) for stats_tracker.
4. Documentation & Tests:
   • Regenerated bilingual CLI reference docs (docs/en/cli_reference.md and docs/zh/cli_reference.md).
   • Added unit test suite in tests/test_simpo_loss.py covering mathematical correctness, margin sensitivity, length normalization, autograd backward pass, and configuration validation.

Related Issue

N/A

Type of Change

• 🐛 Bug fix
• ✨ New feature
• 💥 Breaking change
• 📝 Documentation update
• ♻️ Refactoring
• ⚡ Performance improvement
• ✅ Test coverage improvement

Checklist

• I have read the Contributing Guide
• Pre-commit hooks pass (pre-commit run --all-files)
• Relevant tests pass; new tests added for new functionality
• Documentation updated (if applicable; built with ./docs/build_all.sh)
• Branch is up to date with main
• Self-reviewed via /review-pr command
• This PR was created by a coding agent via /create-pr
• This PR is a breaking change

Additional Context

• Validated via uv run pytest -q tests/test_simpo_loss.py (5/5 passed).
• Verified existing DPO suite uv run pytest -q tests/test_dpo.py (26/26 passed).
• All 16 pre-commit hooks passed cleanly across the entire repository.

[Le8r0nJames]

Could we skip reference-model initialization and log-prob computation for SimPO, and add a test without ref_logprobs? The loss is reference-free, but the training and evaluation paths still require the reference model.

[hsusul]

Fixed in 906c59b. For loss_type=simpo, DPOTrainer no longer creates, initializes, runs or destroys the ref engine (train and eval), and compute_dpo_loss doesn't need ref_logprobs. New tests cover the loss without ref_logprobs and the trainer train/eval paths with no ref model. 6e2b011 merges main and keeps the guard in the new eval cleanup.

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity within the last 14 days.

Please add a comment or push new commits to keep it active.

Thank you for your contribution!

[Le8r0nJames]

LGTM