Add Nemotron 3.5 Lightning DFlash serving (stacked on #248)

docs/muse-glimmer-optimization-ledger.md

@@ -0,0 +1,45 @@
+# Muse-Glimmer-30B (q4) text-model optimization ledger
+
+Measure-first, A/B each candidate against stock decode (same weights, same
+shapes, in-window back-to-back so thermal drift doesn't confound), keep wins,
+reject losers with evidence. GPU work under the serialized MLX window.
+
+## Profile (q4, M5 Max, B=1 decode)
+
+- decode **26 tok/s (38.5 ms/tok) = 82% of the 36.6 tok/s bandwidth roofline** (16.75 GB read/token).
+- per-component census: **MLP 77%**, o_proj 7%, lm_head 6%, attn gate_proj 3%, q+kv 5%, norms 1%.
+- isolated GEMM sum ≈ 28.3 ms; the remaining **~10 ms/tok (26%)** is SDPA + per-layer glue (cache/rope/gate-mul/softcap) + B=1 host dispatch — the schedulable headroom.
+
+## Verdicts
+
+| # | candidate | verdict | evidence |
+|---|-----------|---------|----------|
+| 1 | **async scheduling** | ✅ already in stock | `mx.async_eval` throughout `batched_decode.py`; MTPLX stock decode already overlaps dispatch (the Laguna S1 +4% lever is already captured) |
+| 2 | **QKVG fusion** | ✅ **WIN +4.8%, integrated** | fuse q/k/v/gate → one `quantized_matmul`; **bit-exact** (max\|Δ\|=0), 208→52 launches/token; in-window A/B **26.10 → 27.35 tok/s**. Now the default path in `vendored_muse_glimmer_text.Attention` (id-cached lazy fuse, off the param tree). NB: this *mlx-level concat* beats stock, unlike the Laguna hand-kernel qkvg which was ineligible on affine. |
+| 3a | **MLP gate/up fusion (naive mlx-concat)** | ❌ −0.6% | *Not a valid rejection* — mlx-level concat, not a shape-optimized kernel. Superseded by 3b. |
+| 3b | **MLP dense-SwiGLU fused KERNEL (shape-optimized)** | ✅ **+5.2%**, quality-parity | Laguna `dense_swiglu_qmv` (in-kernel affine dequant, gate+up+silu+mult fused, per-output-element row-owned) at Glimmer's exact gs32/4-bit/6656/19968. Decode 26.58 → 27.96. **Not** bit-exact (5.86e-3, FP accum-order), but HumanEval **28/40 vs 27/40 stock = parity**. Env-gated `MG_MLP_KERNEL`. Proves a shape-optimized kernel beats stock at M=1 where the naive concat lost. |
+| 4 | **qk-norm+rope kernel (row-owned)** | ✅ **+4.8%, bit-exact** | Laguna `fused_qk_rope_sliding` at Glimmer's `SlidingRopeSpec` (32q/2kv, hd128, θ=500000, param-free norm via q_weight=3.87·ones/k_weight=ones, eps1e-5), sliding layers only (globals NoPE). max\|diff\|=0 on q and k. Decode 26.59 → 27.86. Win is **dispatch** (4 kernels × 39 layers → 1 each; host-encode lag), not GPU-exec. Earlier "reject by analogy" was invalid. |
+| 5 | **gated-o_proj kernel (row-owned)** | ✅ **+5.1%** isolated | Custom kernel (MLP-`down` pattern + sigmoid-gate fused into the input read), Glimmer 4/5-bit gs32. Decode 26.51 → 27.85. Non-bit-exact (3.12e-2; in-kernel sigmoid) — would need a HumanEval gate, but ~0 stacked so not pursued. |
+
+## Stacking: the wins SATURATE (do not add)
+
+All four are **dispatch / host-encode** wins competing for one fixed budget. Stacked (QKVG + MLP-kernel + qk-rope), decode = **27.86 tok/s ≈ any single kernel** (MLP-alone was 27.96). So the bankable win is **~+7% over raw stock (~26.0 → ~27.9)** from **QKVG (bit-exact, default) + the MLP kernel (quality-parity, env-gated)**; qk-rope and gated-o_proj are genuine +4.8–5.1% *isolated* wins but ~0 marginal on top.
+
+**Method lesson (David's correction, 4/4 vindicated):** you cannot reject a fusion by a naive mlx-concat or by analogy — only by benchmarking a kernel **tiled+fused for the model's exact shapes**. Every candidate rejected the wrong way became a real +5%-class isolated win, because at M=1 decode the bottleneck is stock `qmm`'s small-T ramp + per-op host-encode lag, which an in-kernel-dequant row-owned kernel beats. The *new* structural lesson is that these wins saturate against a fixed dispatch budget rather than compounding.
+
+## Salvage benchmarks (2026-08-10, in-window guarded)
+
+| test | result | verdict |
+|------|--------|---------|
+| **gated-o_proj stacked on QKVG+MLP** | QKVG 26.72 → +MLP 28.02 → +MLP+gated-oproj **27.95 (−0.3%)** | ❌ no salvage — confirms the budget is fully saturated by QKVG+MLP; gated-o_proj's isolated +5.1% was pure dispatch, zero marginal here. Dead. |
+| **MLP kernel at prefill (L=512)** | stock qmm **932.4** → MLP kernel **896.3 (−3.9%)** | ⚠️ the MLP kernel is a **decode-only** win. At L=512 the regime is compute-bound (large-T qmm is optimal), the M=1-tuned row-owned kernel loses. **Gate `MG_MLP_KERNEL` to L==1** so prefill keeps stock qmm. |
+
+Net after salvage: the bankable win stands at **QKVG (bit-exact, always-on) + MLP kernel (decode-only, L==1-gated) ≈ +7% decode over raw stock**. gated-o_proj and qk-rope add nothing on top (saturated). The MLP-kernel gate is now **`MG_MLP_KERNEL` AND L==1**, not unconditional.
+| 5 | **lm_head 5→4-bit** | ⚠️ not worth | ~1% byte win; lm_head is deliberately 5-bit for output fidelity (quality-gated). |
+
+## Net
+
+One integrated win: **QKVG fusion, +4.8%, bit-exact.** MLP and SDPA are already
+stock-`qmm`/flash-optimal and are intentionally **not** hand-kerneled. The
+remaining decode gap is bandwidth (MLP, quant-gated) + B=1 dispatch (already
+async-overlapped in stock).

mtplx/artifacts.py

@@ -471,6 +471,7 @@ class ModelInspection:
     backend_status: str | None = None
     backend_artifact: dict[str, Any] | None = None
     gemma4_pair: dict[str, Any] | None = None
+    dflash_pair: dict[str, Any] | None = None
 
     @property
     def passes_primary_gate(self) -> bool:
@@ -517,6 +518,7 @@ def to_dict(self) -> dict[str, Any]:
             "backend_status": self.backend_status,
             "backend_artifact": self.backend_artifact,
             "gemma4_pair": self.gemma4_pair,
+            "dflash_pair": self.dflash_pair,
             "mtp_supported": self.compatibility.get("mtp_supported"),
             "mtp_arch": self.compatibility.get("arch_id"),
             "recommended_backend": self.compatibility.get("recommended_backend"),
@@ -1109,6 +1111,60 @@ def inspect_model(model_dir: Path | str) -> ModelInspection:
     if repo_id is not None:
         return _inspect_hf_model(repo_id)
     model_path = Path(model_dir)
+    try:
+        from .dflash_pair import dflash_pair_inspection, resolve_dflash_pair_paths
+    except Exception:
+        dflash_pair = None
+    else:
+        dflash_pair = resolve_dflash_pair_paths(model_path)
+    if dflash_pair is not None:
+        payload = dflash_pair_inspection(
+            model_ref=str(model_path),
+            bundle_root=dflash_pair["bundle_root"],
+            target_model=dflash_pair["target_model"],
+            drafter_model=dflash_pair["drafter_model"],
+            metadata=dflash_pair["metadata"],
+        )
+        target_config = load_config(dflash_pair["target_model"])
+        tcfg = text_config(target_config)
+        target_quant = (
+            target_config.get("quantization")
+            or target_config.get("quantization_config")
+            or tcfg.get("quantization")
+            or tcfg.get("quantization_config")
+            or {}
+        )
+        return ModelInspection(
+            model_dir=str(model_path),
+            source="local",
+            config_exists=True,
+            architecture=str(payload.get("architecture") or "DFlashDrafterPair"),
+            model_type=str(payload.get("model_type") or "dflash_pair"),
+            mtp_num_hidden_layers=1,
+            hidden_size=tcfg.get("hidden_size"),
+            num_hidden_layers=tcfg.get("num_hidden_layers"),
+            vocab_size=tcfg.get("vocab_size"),
+            num_experts=tcfg.get("n_routed_experts") or tcfg.get("num_experts"),
+            num_experts_per_tok=tcfg.get("num_experts_per_tok"),
+            mtp_pattern="dflash-pair",
+            quantization=target_quant,
+            sidecars={name: False for name in MULTIMODAL_SIDECARS},
+            model_files=tuple(
+                sorted(
+                    path.name
+                    for path in Path(dflash_pair["target_model"]).glob(
+                        "model*.safetensors"
+                    )
+                )
+            ),
+            runtime_model=payload.get("runtime_model"),
+            dflash_pair=payload.get("dflash_pair")
+            if isinstance(payload.get("dflash_pair"), dict)
+            else None,
+            compatibility=payload.get("compatibility")
+            if isinstance(payload.get("compatibility"), dict)
+            else {},
+        )
     try:
         from .gemma4_pair import gemma4_pair_inspection, resolve_gemma4_pair_paths
     except Exception:

mtplx/backends/descriptors.py

@@ -729,11 +729,63 @@ def supports(self, capability: str) -> bool:
 )
 
 
+DFLASH_DESCRIPTOR = BackendDescriptor(
+    backend_id="dflash",
+    architecture_id="dflash-drafter-pair",
+    model_family="dflash",
+    display_name="DFlash external drafter",
+    artifact_layout="dflash_pair_bundle",
+    runtime_capabilities=(
+        "target_logits",
+        "external_dflash_drafter",
+        "target_prefix_greedy_verification",
+        "requires_generation_thread_affinity",
+    ),
+    sampler_defaults=SamplerDefaults(temperature=0.0, top_p=1.0, top_k=0),
+    reasoning_codec=ReasoningCodec(
+        parser="none",
+        display_name="Tokenizer-native text",
+        default_mode="off",
+        supported=False,
+    ),
+    draft_semantics=DraftSemantics(
+        request_field="speculative_depth",
+        display_label="DFlash block",
+        default=8,
+        minimum=2,
+        maximum=8,
+        unit="block",
+    ),
+    uses_external_assistant=True,
+    uses_draft_lm_head=False,
+    hidden_variant="dflash_aux_taps",
+    mtp_history_policy="dflash_context_kv",
+    tune_policy=TunePolicy(
+        supported=False,
+        unsupported_reason="DFlash block tuning has not been wired to mtplx tune.",
+    ),
+    kv_quant_policy=KVQuantPolicy(
+        supported=False,
+        disabled_reason="KV quantization is not validated for DFlash hybrid caches.",
+    ),
+    context_window_policy=ContextWindowPolicy(
+        maximum=1_048_576,
+        default=131_072,
+        source="dflash_target_config",
+    ),
+    validation_status="runtime_runnable_qa_pending",
+    status="runtime_runnable_qa_pending",
+    profile_policy="backend-aware-sustained",
+    notes=("DFlash currently exposes exact greedy target-prefix verification.",),
+)
+
+
 DESCRIPTORS_BY_BACKEND_ID: dict[str, BackendDescriptor] = {
     QWEN3_NEXT_DESCRIPTOR.backend_id: QWEN3_NEXT_DESCRIPTOR,
     LAGUNA_AR_DESCRIPTOR.backend_id: LAGUNA_AR_DESCRIPTOR,
     NATIVE_CONTRACT_DESCRIPTOR.backend_id: NATIVE_CONTRACT_DESCRIPTOR,
     GEMMA4_ASSISTANT_DESCRIPTOR.backend_id: GEMMA4_ASSISTANT_DESCRIPTOR,
+    DFLASH_DESCRIPTOR.backend_id: DFLASH_DESCRIPTOR,
     STEP3P5_MTP_DESCRIPTOR.backend_id: STEP3P5_MTP_DESCRIPTOR,
     DEEPSEEK_MTP_DESCRIPTOR.backend_id: DEEPSEEK_MTP_DESCRIPTOR,
     GLM_MTP_DESCRIPTOR.backend_id: GLM_MTP_DESCRIPTOR,