[aaab8b]

Sequence packing (THD) for DeepSeek-V4: many real SFT samples in one window, delimited by
cu_seqlens, with attention strictly isolated so no query sees another sample.

Previously stacked on #1047. That PR has since merged (as 1f1a8a7), so this branch was
rebased onto main and the two duplicated commits dropped. All commits here are now this
PR's own.

Recipes

script	model	context	nodes
run_128k_thd_packed.sh	3 layers (dense / CSA / HCA)	128k	1
run_full_4k_thd_multinode.sh	43 layers + MTP, 256 experts	4k	3
run_full_128k_thd_multinode.sh	43 layers + MTP, 256 experts	128k	3

All three delegate to run_full_4k_multinode.sh for networking, parallelism and offload —
the 128k one delegates further to the 4k one — so the BSHD and THD recipes cannot drift
apart. Documentation is consolidated into a single README_thd_packed.md.

Results

Full model, three nodes, ten steps, natural-length Alpaca packed at runtime:

	4k	128k
parallelism	TP=1 PP=3 CP=1 EP=8, DP=8	TP=1 PP=3 CP=8 EP=8, DP=1
loss	11.889 → 10.896	11.879 → 10.949
grad norm	30.7 → 20.6	31.2 → 19.6
peak memory	149.84 GB (78%)	163.60 GB (85%)
nan iterations	0	0

The two curves have the same shape, which is why the long one was worth running: 128k
introduces no divergence that 4k does not have.

At 128k the parallelism is not a choice. PP=3 is forced by parameter memory; TP cannot
shard V4's dominant tensors because they have no head axis (the indexer's scores is
[B, S, P] with heads summed out, the KV is a single MQA latent); so CP takes the
remaining 8 and DP falls out as 1.

Memory was measured, not extrapolated

seq	4k	16k	32k	64k	128k
peak	149.84	150.41	156.03	166.06	186.32 GB

The ladder predicted 186 GB at 128k against 186.32 observed. That is 97% of a 192 GB card,
so the 128k recipe defaults optimizer offload to 0.90 instead — 163.60 GB, 28 GB of
headroom, no measurable cost with ~2.9 TB of host memory free. A device-wide reading at 97%
is not something a run can rely on: another process taking 6 GB kills it.

FUSED_LINEAR_CE is required at 128k rather than an optimisation — the LM head's logits are
31.6 GB before the loss upcasts — and forces PRIMUS_OVERLAP_PARAM_GATHER=false, which
DeepseekV4Model already asserts.

Fix: MTP ran dense-causal across the whole pack

DeepseekV4MTPLayer declared packed_seq_params and did not pass it on. The inner layer
then saw None, _thd_seq_starts returned None, and that also switched off the packing
guard — so the MTP depth attended across every sample boundary, with no error, no NaN and
a finite loss.

It only appears with mtp_num_layers=1, which the full-model recipe enables by default, and
a PP=2 probe cannot reach it because MTP lives on the last pipeline stage. Verified by
removing the fix and watching four ranks report a null descriptor again.

Also here

• cu_seqlens is keyword-only on Indexer.forward. Merging the packed entry point with
  main's rotating one left position_ids and cu_seqlens adjacent and both
  tensor-shaped, so indexer(hidden, cu) bound cu_seqlens to position_ids and was
  accepted. Here it happened to raise; the general case rotates queries by cu_seqlens
  values in silence.
• run_full_4k_multinode.sh no longer depends on ambient environment. It probes for the
  tokenizer directory (renamed at least once) and supplies DOCKER_IMAGE, which
  run_deepseek_v4.sh hard-requires via ${VAR:?} even though the direct launcher never
  starts a container. The BSHD recipe "worked" before only because that variable happened to
  be set in some shells.
• TURBO_USE_GROUPED_MLP and PRIMUS_OPT_MEGA_MOE default off. main's grad-accum
  fusion and MegaMoE both call primus_turbo APIs the build in rocm/primus:v26.5 does not
  have (one renamed upstream). Note the effective switch is TURBO_USE_GROUPED_MLP, not the
  yaml key — run_deepseek_v4.sh passes it on the command line and overrides the config.
  Set both back on an image with a newer primus_turbo; CI builds it from source.
• Pack cache and its filelock are node-local. Sharing the cache over NFS collides on the
  writer's fixed <final>.tmp name; sharing the lock blocks 23 ranks past Megatron's
  10-minute collective timeout.

What packing buys, on the same corpus

Both layouts consume identical total tokens, so step time is directly comparable:

	total	real	supervised	ms/step	spread
BSHD	163,840	2.8%	1.9%	30,445	4.8%
THD	163,840	100.0%	86.1%	30,930	3.2%

The 485 ms difference is smaller than either configuration's own run-to-run spread, so step
time is indistinguishable and useful throughput tracks the token budget: 35.2× real
tokens/s, 44.6× supervised tokens/s.

That is not a claim that the existing BSHD recipe wastes compute.
run_full_4k_multinode.sh defaults to sft_4096.jsonl, whose rows are deliberately built
longer than the window so truncation rather than padding sets the length — measured at 99.3%
supervised. The comparison above forces both layouts onto the same natural-length corpus,
which is the only way to isolate what packing does.

Verification

• V4 unit suite on a gfx942 host, with the MI355X gate bypassed via
  PRIMUS_V4_UT_ALLOW_NON_MI355X=1 (without it the whole suite skips here):
  this branch 602 passed / 38 failed, plain main 548 passed / 38 failed. The 38
  failures are identical by name on both sides — pre-existing gfx950 kernel numerics,
  not regressions. The branch adds 54 passing tests and introduces zero new failures.
• CP=1 vs CP=2 and CP=4 over ragged lengths: dense / CSA / HCA all match.
• pre-commit run over every changed file: clean.
• Merges cleanly into current main.

Scope

Random init — no DeepSeek-V4 Megatron checkpoint exists — so this is SFT-shaped training
and no number here says anything about model quality. At 4k, ~40% of packs contain no
complete ratio-128 window, so the HCA branch carries almost no signal; 128k is where it
becomes meaningful. Step time at 128k is not a performance figure: num_microbatches = GBS = 3 = PP leaves the pipeline idle roughly two thirds of the time by construction.