Repository navigation

feat(csd): add cross-tokenizer distillation and sub-token preparation - #4680

Open

avenkateshha wants to merge 41 commits into

avenkateshha/xtoken-csd-shared-collatorfrom

avenkateshha/csd

Open

feat(csd): add cross-tokenizer distillation and sub-token preparation#4680
avenkateshha wants to merge 41 commits into
avenkateshha/xtoken-csd-shared-collatorfrom
avenkateshha/csd

Commits on Oct 19, 2026
- feat(csd): add cross-tokenizer distillation and sub-token preparation
  avenkateshha
  committed
  dae6334
- refactor(csd): use the shared same-tokenizer KD helper
  
  avenkateshha
  and
  claude
  committed
  252eb43
- refactor(csd): build CSD batches with the shared cross-tokenizer collator
  
  avenkateshha
  and
  claude
  committed
  6f282b2
- fix(csd): repair CSD test, recipe and suite configuration
  
  avenkateshha
  and
  claude
  committed
  da6fd8b
- fix(csd): add the Automodel checkpoint block to CSD configs
  
  avenkateshha
  and
  claude
  committed
  6db2544
- fix(csd): pad CSD CP=2 recipe sequences to a multiple of 2
  
  avenkateshha
  and
  claude
  committed
  badb142
- fix(csd): update tests that run only with the Automodel and Megatron extras
  
  avenkateshha
  and
  claude
  committed
  297bb2e
- fix(csd): set refit offload keys on CSD example teachers
  
  avenkateshha
  and
  claude
  committed
  75a3393
- feat(csd): run the CSD example on Megatron
  
  avenkateshha
  and
  claude
  committed
  3aaa43b
- docs(csd): describe the Megatron backend of the CSD example
  
  avenkateshha
  and
  claude
  committed
  5689406
- fix(csd): shift chunk predictors by default
  
  avenkateshha
  and
  claude
  committed
  0c6ec39
- feat(csd): make JSD the default CSD divergence
  
  avenkateshha
  and
  claude
  committed
  414950a
- feat(csd): add realized-only scoring for mismatch chunks
  
  avenkateshha
  and
  claude
  committed
  f7f77c5
- fix(xtoken): mark alignment pairs over tokenizer padding invalid
  
  avenkateshha
  and
  claude
  committed
  11a5ba6
- test(csd): cover fixed and dynamic KD/CE weighting
  
  avenkateshha
  and
  claude
  committed
  99a6af0
- chore(csd): remove the Moonlight router-replay recipe
  
  avenkateshha
  and
  claude
  committed
  6bf7b8f
- refactor(csd): remove the position-0 mismatch term
  
  avenkateshha
  and
  claude
  committed
  c7694c4
- feat(csd): weight mismatch chunks by 5 by default
  
  avenkateshha
  and
  claude
  committed
  b9cf3ad
- feat(csd): clip overlength text rows and run CSD recipes nightly
  
  avenkateshha
  and
  claude
  committed
  8f5db56
- test(csd): add a Megatron TP2xCP2 recipe to the nightly suite
  
  avenkateshha
  and
  claude
  committed
  010dd19
- feat(distributed): add vocab-parallel selected log-probs and sparse top-k primitives
  
  avenkateshha
  and
  claude
  committed
  ce38f6b
- feat(megatron): stream sparse top-k teacher rows with two forced labels
  
  avenkateshha
  and
  claude
  committed
  3ef5ec1
- feat(csd): read requested teacher rows from sparse and dense IPC
  
  avenkateshha
  and
  claude
  committed
  3709118
- feat(csd): evaluate cross-tokenizer KD on context-parallel owner ranks
  
  avenkateshha
  and
  claude
  committed
  5e366a8
- feat(csd): route eligible Megatron runs to owner-local evaluation
  
  avenkateshha
  and
  claude
  committed
  ab14ab2
- fix(csd): reduce same-tokenizer KD across Megatron context parallelism
  
  avenkateshha
  and
  claude
  committed
  e247ca2
- feat(csd): send sparse teacher top-128 by default
  
  avenkateshha
  and
  claude
  committed
  bd984b0
- docs(csd): describe owner-local evaluation and teacher transports
  
  avenkateshha
  and
  claude
  committed
  6fd0dda
- fix(csd): apply the CP divisor check to Automodel entities only
  
  avenkateshha
  and
  claude
  committed
  a281128
- fix(csd): limit chat CE to assistant content and end-of-turn
  
  avenkateshha
  and
  claude
  committed
  aefe0ed
- docs(csd): add a CSD cross-tokenizer distillation guide
  
  avenkateshha
  and
  claude
  committed
  dd864cf
- docs(csd): merge the CSD guide into the cross-tokenizer guide
  
  avenkateshha
  and
  claude
  committed
  44889c4
- docs(xtoken): rename the guide to cross-tokenizer-distillation.md
  
  avenkateshha
  and
  claude
  committed
  c516f03
Commits on Oct 21, 2026
- test(xtoken): pin the strict exact-map rule without the -1 sentinel
  
  avenkateshha
  and
  claude
  committed
  9946ac4
- test(csd): add the copyright header to test_subtoks_tables.py
  
  avenkateshha
  and
  claude
  committed
  3f92c61
- test: fix test-side failures and gaps in the Megatron and Automodel tests
  
  avenkateshha
  and
  claude
  committed
  7e79fde
- test(csd): trim the CSD nightlies and raise the nightly GPU-hour cap
  
  avenkateshha
  and
  claude
  committed
  d1af5c2
- fix(megatron): shard the router padding mask in the unpacked-CP branch
  
  avenkateshha
  and
  claude
  committed
  dae117b
- fix(csd): give the multi-teacher Phi-4-mini teacher its own table paths
  
  avenkateshha
  and
  claude
  committed
  00b688a
- fix(megatron): skip the dense causal mask in the unpacked-CP branch
  
  avenkateshha
  and
  claude
  committed
  f9a2db5
- fix(megatron): apply freeze_parameter_patterns on the Megatron backend
  
  avenkateshha
  and
  claude
  committed
  c560131

Commits

Commits on Oct 19, 2026

Commits on Oct 21, 2026