WIP: Multi-agent support: Agent, Protocol, MultiAgentEnv

[nph4rd]

Description

Adds foundational abstractions for multi-agent turn-based environments with support for heterogeneous (per-agent) rewards:

Core abstractions

• Agent: Dataclass representing a participant (id, system_prompt, is_trainable)
• Protocol: ABC defining turn order (get_initial_agent, get_next_agent)
• RoundRobinProtocol: Concrete implementation for sequential turns
• MultiAgentEnv: Base class extending MultiTurnEnv with multi-agent support

Heterogeneous rewards:

• MultiAgentRewardFunc: New reward function type returning dict[str, float] (agent_id → reward)
• Per-agent reward aggregation in Rubric.score_group()
• Per-agent advantages stored in trajectory steps

Additional fix: get_prompt_ids() now correctly handles multi-agent tokenization by finding the previous turn for the current agent instead of using the global last trajectory step. This prevents empty message errors when agents interleave turns during training.

Design goals:

• Protocol is required to encourage reusable turn-order logic
• Minimal but extensible - can add simultaneous moves, state splitting, harness etc. later

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Test improvement

Testing

• All existing tests pass when running uv run pytest locally.
• New tests have been added to cover the changes

Checklist

• My code follows the style guidelines of this project as outlined in AGENTS.md
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• Any dependent changes have been merged and published

Additional Notes

Note

Add multi-agent support with Agent, Protocol, and MultiAgentEnv abstractions

• Introduces MultiAgentEnv extending MultiTurnEnv with protocol-driven turn-taking, per-agent prompt building, and per-step agent metadata tagging during rollout.
• Adds Agent dataclass (id, system_prompt, is_trainable) and a Protocol interface with a RoundRobinProtocol implementation for turn ordering.
• Extends Rubric to support reward functions returning dict[str, float] per-agent rewards, aggregating them into state['agent_rewards'] and computing per-agent advantages.
• Propagates an actor_models dict through the full stack: Environment, EnvGroup, EnvClient, EnvWorker, and serve request types, allowing per-agent model routing.
• Behavioral Change: score_rollout and score_group now check for multi-agent reward functions before group functions; non-multi-agent paths are unchanged.

^{Macroscope summarized f430b7f. (Automatic summaries will resume when PR exits draft mode or review begins).}