feat(qwen35): tp2 phase 2a

[Mrtroll486]

Description

Part of issue #446 and #694. Follows pr #627 and #450.

This PR completes Qwen3.5 Tensor Parallel Phase 2A (P2A): eager TP2 mixed prefill/decode execution with a strict cross-rank lifecycle and failure contract.

P2A builds on the existing Phase 1 dense TP implementation. It keeps linear-attention/GDR weights and request state replicated; sharding that surface remains P2B.

What Changed

• Added a canonical, ordered RunUnifiedStep command for eager TP mixed prefill and decode.
• Reused the shared scheduler planner instead of the TP-only eager planner branch.
• Added an all-rank start gate for state-mutating commands:
  • workers execute only after every rank receives the command;
  • partial dispatch cancels the delivered prefix and poisons the replica.
• Hardened worker response validation:
  • requires exactly one response from every rank;
  • validates rank range, uniqueness, reply variant, and the rank-0 result contract.
• Added lifecycle-aware DropAck { existed } handling:
  • MustBeAbsent requires all ranks to report false;
  • MustExist requires all ranks to report true;
  • mixed or uniformly unexpected results are replica-fatal.
• Changed cancellation ordering to:
  • drain -> prune -> publish load -> admission -> plan
• Added fail-closed scheduler recovery:
  • closes and drains submissions after fatal TP failure;
  • sends one terminal error to every unresolved request owner;
  • publishes an idle terminal load snapshot;
  • begins whole-executor teardown without retrying per-request cleanup.
• Added a completion commit boundary:
  • TP EOS/length and immediate-prefill success events are withheld until all-rank cleanup succeeds.
• Replaced positional TP result matching with strict RequestId alignment:
  • shuffled valid artifacts are accepted;
  • unknown, duplicate, non-final, or missing artifacts poison the replica.
• Added pre-load validation for duplicate CUDA ordinals and ordinals outside the fixed 16-device Triton AOT handle table.
• Added lifecycle snapshots and fault-injection coverage for rank-local state, partial dispatch, state divergence, and worker disconnects.

Failure Semantics

Controller-provable structural errors are rejected before dispatch and leave the executor healthy.

After execution is released, worker-local state mismatches, CUDA/NCCL failures, malformed response sets, artifact violations, and lifecycle failures poison the complete TP replica. The scheduler then fails every unresolved request and exits.

Collective or teardown timeout remains process-level fail-stop.

Scope and Non-Goals

This PR does not:

• shard linear-attention/GDR weights, conv state, recurrent state, or scratch buffers;
• add a post-GDR hidden all-reduce;
• enable TP CUDA Graph capture or replay;
• add vocabulary-parallel embedding or lm_head;
• add TP-aware prefix caching;
• claim a performance improvement.

Those state-sharding changes belong to P2B.

Validation

Validated in release mode on 2x RTX 3090 (SM86) with real Qwen3.5-4B weights.

• P2A/TP lifecycle ignored library gates:
  • 14 passed, 0 failed
  • includes 13 TP2 gates and the TP1 cancellation regression
  • covers mixed unified execution, drop/refill, clean re-admission, lifecycle divergence, partial dispatch, and worker disconnect
  • memory-derived TP2 max_batch=64
• TP2 scheduler E2E:
  • 1 passed, 0 failed
  • covers context rejection, greedy/logprobs, sequential and repeated requests, concurrent mixed sampling, consumer drop, and post-drop health
• TP2 HF short/long golden gates:
  • 2 passed, 0 failed
  • short sequential: mean 0.0260, p99 0.1081
  • short batched: mean 0.0267, p99 0.1167
  • long 4097/8192-token replay: mean 0.0228, p99 0.0689
• TP2 OpenAI-compatible HTTP serving smoke:
  • 1 passed, 0 failed
  • covers streaming, non-streaming, concurrent completions, finite logprobs, and TP + CUDA Graph rejection
• Release all-target clippy with -D warnings: passed
• cargo fmt --all -- --check: passed
• git diff --check: passed

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update

Checklist

• My code follows the style guidelines of this project (see docs/conventions/coding-style.md).
• I have performed a self-review of my own code.
• I have formatted my commits according to Commitizen conventions.
• I have run the local test suite and all executed tests pass (see CLAUDE.md).

[Mrtroll486]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 10116dc125

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".