feat(trainer): add composable advantage computation

[fishcrap]

Description

Introduce a composable advantage computation pipeline through actor.advantage. Users can override outcome reward shaping, temporal estimation, process advantage shaping, or the complete computation with a built-in implementation or a dotted callable and serializable arguments.

The default pipeline reuses the existing GAE kernels and preserves PPO, PRM, and MOPD behavior, logical rollout normalization, truncation handling, and optimizer scheduling. Existing PRM configuration remains supported; explicit advantage-stage settings take precedence.

Custom outcome shaping can provide logical reward references in the same units as the shaped rewards. Plugin results are validated for shape, device, dtype, finite values, binary masks, and permitted field updates. English and Chinese documentation describe configuration, extension contracts, and compatibility behavior.

Related Issue

Follow-up to #1782. No dedicated issue is associated with this change.

Type of Change

• ✨ New feature

Validation

• Regression and contract tests: 470 tests passed, including 13 CUDA cases and multiprocess normalization checks.
• Default behavior parity: 32 CPU differential cases produced output tensors exactly equal to the main baseline.
• Distributed training parity: dense and MoE models produced identical outputs, gradients, and parameter updates across three PPO updates for the repeated baseline, default pipeline, and explicitly configured plugins.
• End-to-end training: the final PR revision completed two GRPO steps with SGLang rollouts and Megatron training. Checks confirmed successful optimizer updates, finite nonzero gradients, updated-weight rollouts, weight-version publication, and finite changed checkpoint parameters.
• Quality checks: pre-commit run --all-files and ./docs/build_all.sh passed. Existing documentation warnings are outside the changed pages.

Regression command:

python -m pytest tests/test_advantage_pipeline.py tests/test_advantage_pipeline_cuda.py tests/test_advantage_algorithms.py tests/test_ppo_gae.py tests/test_token_rewards_advantage.py tests/test_reward_norm_variable_group.py tests/test_logical_rollout_groups.py tests/test_ppo_actor_truncation.py tests/test_training_diagnostics.py tests/test_mopd_compatibility.py tests/test_prompt_mean_actor.py tests/test_prm_config.py tests/test_prompt_mean_loss.py tests/test_ppo_actor_controller.py tests/test_ppo_stats.py tests/test_mopd_config.py tests/test_mopd_loss.py tests/test_incomplete_rollout_groups.py tests/test_prm_config_validation.py tests/test_logical_rollout_distributed.py --basetemp="$(mktemp -d)/pytest" -q

The distributed parity run preceded the final configuration and input-validation guards. The final revision passed the complete targeted suite and a fresh end-to-end run. FSDP and Archon end-to-end training and MTP were not included in this validation; training-engine and loss implementations are unchanged.

Checklist

• I have read the Contributing Guide
• Pre-commit hooks pass (pre-commit run --all-files)
• Relevant tests pass; new tests added for new functionality
• Documentation updated (if applicable; built with ./docs/build_all.sh)
• Branch is up to date with main
• Self-reviewed via /review-pr command
• This PR was created by a coding agent via /create-pr
• This PR is a breaking change

Additional Context

The main review areas are collective ordering during logical reward normalization, preservation of GAE/PRM computation order, and configuration precedence. This change is limited to advantage computation; efficiency presets, loss plugins, data hooks, and scheduler changes are outside its scope.